AMD GPU passthrough to an Edera zone

6 min read · Intermediate

This guide walks through how to passthrough a GPU to an Edera zone using protect and run a workload with AMD’s ROCm.

Getting Started

Edera must be successfully installed. See our installation guides, which should also include making sure you have access to the Edera image registry.
Find the GPUs on your host.
Supported GPUs, like the AMD MI250, should show up in the output of lspci:
```
sudo lspci -Dknn -d ::03xx
```
Example output:
```
0000:03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran/MI200 [Instinct MI250X/MI250] [1002:740c] (rev 01)
   ...
```
Make note of the PCI ids at the front (like 0000:03:00.0) for the next step.
Add your GPUs to the Protect demon’s config file.
This tells Edera which GPUs are available for passthrough and names them:
```
#cat /var/lib/edera/protect/daemon.toml

[pci.devices]
[pci.devices.gpu0]
locations = [
  "0000:03:00.0",
]
permissive = true
msi_translate = false
power_management = true
rdm_reserve_policy = "relaxed"
```
The key gpu0 is the primary identifier for the GPU in Edera. It links the physical device to the zone and workload later via --device gpu0. Although we only show one GPU here, if you have multiple GPUs you will need a block for each GPU you want to pass through and each will need a unique name. We recommend numbering them gpu0, gpu1, gpu2 and so on, but you can choose any name here (for example testcard, mi250), but it must be unique and must match the name used later in your launch commands.
Optionally check the amd kernel variant defined in the daemon config file.
By default each Edera release ships an amd kernel variant, which is pre-defined in the daemon config file. This lets you launch a zone with --kernel-variant amd and Edera will resolve the correct images for you. Typically you shouldn’t need to modify this, but if you do see Selecting a zone kernel with variants` for more details.
```
# /var/lib/edera/protect/daemon.toml

[zone.kernel-variants]
amd = "ghcr.io/edera-dev/zone-amdgpu-kernel:6.14.10"
```
Restart the Protect daemon and verify.
If you made any changes to the daemon.toml file, restart the daemon (sudo systemctl restart protect-daemon).
If you changed the kernel variant configs, confirm the variant is resolvable with sudo protect image list-kernel-variants.
Additional Steps if using KVM

ℹ️

GPU passthrough for KVM is supported starting with Edera v1.11.0.

When using GPU passthrough with KVM as the hypervisor, the GPU must first be bound to the VFIO driver before Edera can attach it to a zone.

Ensure that no other driver has grabbed the GPU:
Similar to above, run sudo lspci -Dknn -d ::03xx and locate the GPUs. They should still not display a Kernel driver in use: field. Make note of the PCI Vendor and Device ids that look like [1002:7408] or [1002:740c] for use in the next step.
Configure module parameters for the VFIO kernel modules:
Use the PCI vendor and device IDs from the previous step. If you need to support multiple PCI IDs, separate them by commas.
```
# /etc/modprobe.d/gpu-vfio.conf (new file)

options vfio_iommu_type1 allow_unsafe_interrupts
options vfio_pci ids=1002:740c
```

Enable automatic loading of the VFIO kernel modules on boot:

# /etc/modules-load.d/gpu-vfio.conf (new file)

vfio_iommu_type1
vfio_pci

Reboot the system or (re-)load the VFIO kernel modules:

sudo modprobe -r vfio_pci
sudo modprobe -r vfio_iommu_type1
sudo modprobe vfio_iommu_type1
sudo modprobe vfio_pci

Confirm that the GPUs are successfully bound to the vfio-pci driver:

sudo lspci -Dknn -d ::03xx

Example output:

...
0000:03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran/MI200 [Instinct MI250X/MI250] [1002:740c] (rev 01)
...
Kernel driver in use: vfio-pci
...

Launch a zone with GPU passthrough

Launch a new zone and attach a GPU:

sudo protect zone launch -n zone-gpu0 \
  --device gpu0 \
  --kernel-cmdline-append 'drm.debug=0xff pic=debug' \
  --kernel-verbose \
  -m 8000 \
  -R static \
  --kernel-variant amd

Explanation:

-n zone-gpu0: name of the zone
--device gpu0: This is the name you gave the GPU in the daemon.toml file as pci.devices.gpu0.
--kernel-cmdline-append: adds kernel parameters to the zone
-m 8000: allocates 8 GB memory to the zone
-R static: resource adjustment policy
--kernel-variant amd: use the AMD GPU kernel variant defined in daemon.toml

Check that the zone launched successfully:

sudo protect zone list

Expected outcome:

| name | uuid                                 | state | ipv4           | ipv6                          |
|------|--------------------------------------|--------|----------------|-------------------------------|
| zone-gpu0 | a42b9d46-9069-4cf0-a00c-56738aee2932 | ready  | 10.75.0.2/16   | fdd4:1476:6c7e::2/48          |

Load GPU driver in the zone

sudo protect zone exec -t zone-gpu0 -- /bin/busybox modprobe amdgpu

Confirm the driver is loaded:

sudo protect zone logs zone-gpu0
sudo protect zone exec -t zone-gpu0 -- /bin/busybox lsmod

Expected outcome:

| Module                       | Size    | Used by                                                                 |
|-----------------------------|---------|-------------------------------------------------------------------------|
| amdgpu                      | 14086144| 0                                                                       |
| amdxcp                      | 12288   | 1 amdgpu                                                                |
| i2c_algo_bit                | 12288   | 1 amdgpu                                                                |
| drm_client_lib              | 12288   | 1 amdgpu                                                                |
| drm_ttm_helper              | 16384   | 1 amdgpu                                                                |
| ttm                         | 106496  | 2 amdgpu,drm_ttm_helper                                                 |
| drm_exec                    | 12288   | 1 amdgpu                                                                |
| gpu_sched                   | 61440   | 1 amdgpu                                                                |
| drm_suballoc_helper         | 12288   | 1 amdgpu                                                                |
| video                       | 81920   | 1 amdgpu                                                                |
| drm_panel_backlight_quirks | 12288   | 1 amdgpu                                                                |
| cec                         | 65536   | 1 amdgpu                                                                |
| drm_buddy                   | 24576   | 1 amdgpu                                                                |
| drm_display_helper          | 221184  | 1 amdgpu                                                                |
| drm_kms_helper              | 237568  | 4 amdgpu,drm_client_lib,drm_ttm_helper,drm_display_helper               |
| syscopyarea                 | 12288   | 1 drm_ttm_helper                                                        |
| sysfillrect                 | 12288   | 1 drm_ttm_helper                                                        |
| sysimgblt                   | 12288   | 1 drm_ttm_helper                                                        |
| fb_sys_fops                 | 12288   | 1 drm_ttm_helper                                                        |
| backlight                   | 12288   | 3 amdgpu,video,drm_display_helper                                      |
| fb                          | 77824   | 1 drm_ttm_helper,drm_kms_helper,backlight                              |

Run a GPU workload

Launch a privileged workload using ROCm:

sudo protect workload launch -n workload_gpu0 \
  --privileged \
  --zone zone-gpu0 \
  rocm/rocm-terminal

Check the workload is running:

sudo protect workload list

Expected outcome:

| name           | uuid                                   | zone                                   | state   |
|----------------|----------------------------------------|----------------------------------------|---------|
| workload_gpu0  | 91deb366-fa91-4b6d-8a99-b1ad012f25c6   | 827c4315-07dc-464c-b137-3fa6cff195a9   | running |

Execute the ROCm command:

sudo protect workload exec workload_gpu0 -- sudo rocm-smi

Expected outcome:

$ sudo protect workload exec workload_gpu0 -- sudo rocm-smi
[2025-07-16T10:17:37Z WARN styrolite::wrap] unable to set target GID: Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }

========================= ROCm System Management Interface =========================
===================================================================================
GPU  Temp   Power  SCLK     MCLK     Fan  Perf  PwrCap  VRAM%  GPU%
ID   (Edge) (Avg)  (MHz)    (MHz)    (%)        (Watt)
===================================================================================
0    48.0°C 0.0W   1700Mhz  1600Mhz  0%   auto  560.0W   0%     0%
===================================================================================
============================= End of ROCm SMI Log ================================

Success. We’ve configured the GPU and have launched a workload in an isolated zone.

Cleanup

Destroy the workload:

sudo protect workload destroy workload_gpu0

Unload the driver:

sudo protect zone exec -t zone-gpu0 -- /bin/busybox modprobe -r amdgpu

Destroy the zone:

sudo protect zone destroy zone-gpu0

Next steps

Now that you’ve tested GPU passthrough, you can integrate this into your AI pipeline or run additional ROCm-based workloads within an isolated Edera zone.

AMD GPU passthrough to an Edera zone

Getting Started

Launch a zone with GPU passthrough

Load GPU driver in the zone

Run a GPU workload

Cleanup

Next steps

Further reading