Troubleshooting
BAR 3: can’t reserve error
If you have this error when you try to use the card for a VM:
vfio-pci 0000:04:00.0: BAR 3: can't reserve
you can try to add the following kernel commandline option:
video=efifb:off
SPICE
Spice may give trouble when passing through a GPU as it presents a «virtual» PCI graphic card to the guest and some drivers have problems with that, even when both cards show up.
It’s always worth a try to disable SPICE and check again if something fails.
HDMI Audio crackling/broken
Some digital audio devices (usually added via GPU functions) may require MSI (Message Signaled Interrupts) to be enabled to function correctly. If you experience any issues, try changing MSI settings in the guest and rebooting the guest.
Linux guests usually enable MSI by themselves. To force use of MSI for GPU audio devices, use the following command and reboot:
echo "options snd-hda-intel enable_msi=1" >> /etc/modprobe.d/snd-hda-intel.conf
Use ‘lspci -vv’ and check for the following line on your device to see if MSI is enabled:
Capabilities: MSI: Enable+ Count=1/1 Maskable- 64bit+
If it says ‘Enable+’, MSI is working, ‘Enable-‘ means it is supported but disabled, and if the line is missing, MSI is not supported by the PCIe hardware.
This can potentially also improve performance for other passthrough devices, including GPUs, but that depends on the hardware being used.
BIOS options
Make sure you are using the most recent BIOS version for you mainboard. Often IOMMU groupings or passthrough support in general is improved in later versions.
Some general BIOS options that might need changing to allow passthrough to work:
- IOMMU or VT-d: Set to ‘Enabled’ or equivalent, often ‘Auto’ is not the same
- ‘Legacy boot’ or CSM: For GPU passthrough it can help to disable this, but keep in mind that PVE has to be installed in UEFI mode, as it will not boot in BIOS mode without this enabled. The reason for disabling this is that it avoids legacy VGA initialization of installed GPUs, making them able to be re-initialized later, as required for passthrough. Most useful when trying to use passthrough in single GPU systems.
Assigning Devices to VMs
Note that only «real» devices can be assigned, i.e. assigning of USB hubs is not possible.
In Configuration
General Remark
Using the below mentioned «qm set» command has the same effect as writing directly into the conf file. Moreover a preconfigured usb configuration file /usr/share/qemu-server/pve-usb.cfg could be adapted. But in order to avoid confusion this article recommends to use only the «qm set» command.
USB1.x, USB2.0
qm set <VM-ID> -usb host={ <bus>-<port>(.<port>)* | <vendor-id>:<product-id> }
According to the above example let´s say the Alcor stick and the Cruzer Edge connected to the external hub (which is device (4) from the example) should be assigned to VM 804:
For the Alcor stick (as already shown):
qm set 804 -usb0 host=058f:6387
For the Cruzer Edge this form will not be sufficient — there are two devices from the same type. Therefore the form specifying bus and port is used:
qm set 804 -usb1 host=2-1.2.2
When the usb devices have to be removed from configuration the conf file has to be changed. In our example remove in /etc/pve/qemu-server/804.conf the following two lines:
usb0: host=058f:6387 usb1: host=host=2-1.2.2
USB3.0
since qemu-server package version 4.0-55 you also can set USB3 devices directly via the command line
qm set 804 -usb0 host=058f:6387,usb3=yes qm set 804 -usb1 host=2-1.2.2,usb3=yes
it will automatically add the appropriate xhci controller at an available pci address
for any version <4.0-55 you can use the following instructions
1 Device
The external Toshiba drive should be assigned VM 804 and work with USB 3.0 protocol on a USB 3.0 port.
qm set 804 -args "-device nec-usb-xhci,id=xhci,addr=0x1b,bus=pci.1 -device usb-host,bus=xhci.0,vendorid=0x0480,productid=0xa007"
Note that «addr=0x5» works only if there is no other device assigned to this address. If a failure occurs because of double use of this address choose another one.
When the usb device has to be removed from configuration the conf file has to be changed. In our example remove in /etc/pve/qemu-server/804.conf the following line:
args: -device nec-usb-xhci,id=xhci,addr=0x1b,bus=pci.1 -device usb-host,bus=xhci.0,vendorid=0x0480,productid=0xa007
More Devices
For one machine only one «-args» argument is possible. If already an «-args» advice was given giving another one overwrites the first. Moreover, the usb3 driver needs only to be defined once. In the following example 2 USB3 devices are assigned to the VM, the second one has the ID 1058:0820
qm set 804 -args "-device nec-usb-xhci,id=xhci,addr=0x1b,bus=pci.1 -device usb-host,bus=xhci.0,vendorid=0x0480,productid=0xa007 \ -device usb-host,bus=xhci.0,vendorid=0x1058,productid=0x0820"
Assign Devices to an already Running VM
How to connect devices to a running machine without shutting it down is demonstrated by the examples from above. If device is assigned in that way the assignment is valid until the machine stops. The assignment is independent of the actual status of the device (whether it´s currently plugged in or not), i.e. it can be plugged in and out multiple times and will be always assigned to the VM; regardless if the «Productid» or «busaddress» method is used.
USB1.x, USB2.0
qm monitor 804
device_add usb-host,vendorid=0x058f,productid=0x6387,id=someid device_add usb-host,hostbus=2,hostport=1.2.2,id=someotherid
Note that the «usb_add» command offered in «qm monitor» is NOT FUNCTIONING for this case!
someid and someotherid are ids of your choice, those can later be used to remove the usb device from the guest
USB3.0
qm monitor 804
device_add nec-usb-xhci,id=xhci,addr=0x1B,bus=pci.1 device_add usb-host,bus=xhci.0,vendorid=0x0480,productid=0xa007,id=anotherid
anotherid is chosen by you, so the device can later be removed again
Alternate way: pass the USB port to the guest
Example
In order to demonstrate how devices, physically connected to the host can be identified it´s assumed the following 7 USB devices are currently connected:
- External HD Toshiba USB 3.0
- memory stick Cruzer Blade USB 2.0
- memory stick SanDisk USB 2.0
- memory stick SanDisk USB 2.0 behind a USB hub
- memory stick Alcor behind the same USB hub as above
- Ethernet Network adapter USB 1.1 behind the same USB hub as above
- External HD Western Digital USB 3.0
Show Device Types
This is the simplest form and sufficient in most cases.
lsusb
The result is
Bus 002 Device 004: ID 0781:5567 SanDisk Corp. Cruzer Blade # <-------- (2) Bus 002 Device 008: ID 0fe6:9700 Kontron (Industrial Computer Source / ICS Advent) DM9601 Fast Ethernet Adapter # <-------- (6) Bus 002 Device 007: ID 0781:556b SanDisk Corp. # <-------- (4) Bus 002 Device 013: ID 058f:6387 Alcor Micro Corp. Flash Drive # <-------- (5) Bus 002 Device 005: ID 05e3:0605 Genesys Logic, Inc. USB 2.0 Hub Bus 002 Device 010: ID 0781:556b SanDisk Corp. # <-------- (3) Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 004 Device 003: ID 1058:0820 Western Digital Technologies, Inc. # <-------- (7) Bus 004 Device 002: ID 0480:a007 Toshiba America Info. Systems, Inc. External Disk USB 3.0 # <-------- (1) Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
The numbers on the far right refer to the previously defined example. The other entries are the USB hubs of the system.
Note that the «Device» number is just a number assigned automatically by the host, independent where the device is connected to. If the device is removed and plugged in again it gets a new number.
Show Device Map and Types
Sometimes it is necessary to specify where a device is physically connected. This information can be obtained by
qm monitor 804
where «804» is the ID of a running VM — regardless which VM is specified, the following command inside of the monitor reports always the same
info usbhost
in case of our example it is
Bus 2, Addr 4, Port 1.8, Speed 480 Mb/s Class 00: USB device 0781:5567, Cruzer Blade # <---- (2) Bus 2, Addr 8, Port 1.2.3, Speed 12 Mb/s Class 00: USB device 0fe6:9700, USB 2.0 10/100M Ethernet Adaptor # <---- (6) Bus 2, Addr 7, Port 1.2.2, Speed 480 Mb/s Class 00: USB device 0781:556b, Cruzer Edge # <---- (4) Bus 2, Addr 13, Port 1.2.1, Speed 480 Mb/s Class 00: USB device 058f:6387, Mass Storage # <---- (5) Bus 2, Addr 10, Port 1.1, Speed 480 Mb/s Class 00: USB device 0781:556b, Cruzer Edge # <---- (3) Bus 4, Addr 3, Port 1, Speed 5000 Mb/s Class 00: USB device 1058:0820, My Passport 0820 # <---- (7) Bus 4, Addr 2, Port 1, Speed 5000 Mb/s Class 00: USB device 0480:a007, External USB 3.0 # <---- (1)
The devices behind the external hub have port addresses 1.2.x
This form shows also the currently used USB protocol (btw: even the device (6) calls itself «USB 2.0 10/100M Ethernet Adaptor» it works only as USB 1.1 device)
- 12 MB/s = USB 1.1
- 480 Mb/s = USB 2.0
- 5000 Mb/s = USB 3.0
Just the map can be displayed by
lsusb -t
but it´s sometimes not complete and not recommended.
Complete Information
usb-devices
which contains everything but i´s a little bit difficult to read — for the current subject it´s sufficient to use the qm monitor as described above. The only one thing is a running VM must exist when processing the command.
