Verify IOMMU Isolation
For working PCI passthrough, you need a dedicated IOMMU group for all PCI devices you want to assign to a VM.
You should have something like:
# find /sys/kernel/iommu_groups/ -type l /sys/kernel/iommu_groups/0/devices/0000:00:00.0 /sys/kernel/iommu_groups/1/devices/0000:00:01.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.0 /sys/kernel/iommu_groups/1/devices/0000:01:00.1 /sys/kernel/iommu_groups/2/devices/0000:00:02.0 /sys/kernel/iommu_groups/3/devices/0000:00:16.0 /sys/kernel/iommu_groups/4/devices/0000:00:1a.0 /sys/kernel/iommu_groups/5/devices/0000:00:1b.0 /sys/kernel/iommu_groups/6/devices/0000:00:1c.0 /sys/kernel/iommu_groups/7/devices/0000:00:1c.5 /sys/kernel/iommu_groups/8/devices/0000:00:1c.6 /sys/kernel/iommu_groups/9/devices/0000:00:1c.7 /sys/kernel/iommu_groups/9/devices/0000:05:00.0 /sys/kernel/iommu_groups/10/devices/0000:00:1d.0 /sys/kernel/iommu_groups/11/devices/0000:00:1f.0 /sys/kernel/iommu_groups/11/devices/0000:00:1f.2 /sys/kernel/iommu_groups/11/devices/0000:00:1f.3 /sys/kernel/iommu_groups/12/devices/0000:02:00.0 /sys/kernel/iommu_groups/12/devices/0000:02:00.1 /sys/kernel/iommu_groups/13/devices/0000:03:00.0 /sys/kernel/iommu_groups/14/devices/0000:04:00.0
To have separate IOMMU groups, your processor needs to have support for a feature called ACS (Access Control Services).
All Xeon processor support them (E3,E5) excluding Xeon E3-1200.
Haswell-E (LGA2011-v3) i7-5960X (8-core, 3/3.5GHz) i7-5930K (6-core, 3.2/3.8GHz) i7-5820K (6-core, 3.3/3.6GHz) Ivy Bridge-E (LGA2011) i7-4960X (6-core, 3.6/4GHz) i7-4930K (6-core, 3.4/3.6GHz) i7-4820K (4-core, 3.7/3.9GHz) Sandy Bridge-E (LGA2011) i7-3960X (6-core, 3.3/3.9GHz) i7-3970X (6-core, 3.5/4GHz) i7-3930K (6-core, 3.2/3.8GHz) i7-3820 (4-core, 3.6/3.8GHz)
AMD chips from Ryzen 1st generation and newer are fine too.
If you don’t have dedicated IOMMU groups, you can try:
1) moving the card to another pci slot
2) adding «pcie_acs_override=downstream» to kernel boot commandline (grub or systemd-boot) options, which can help on some setup with bad ACS implementation.
More infos:
