Forwarding NVIDIA Quadro 4000 to a Virtual Machine Using Xen Hypervisor

Published on December 07, 2012

Forwarding NVIDIA Quadro 4000 to a Virtual Machine Using Xen Hypervisor

Having once read a post [ 1 ] about the successful forwarding of a video card to a virtual machine, I thought it would be nice for me to get such a workstation.

When developing cross-platform software, there are often problems with testing it. I carry out all my work exclusively under Linux, while the end user works exclusively in the Windows operating system (OS). One could use VirtualBox, for example, but when you need to check the operation of modules using OpenGL or CUDA, serious problems arise. I do not even consider Dual Boot as an option. It turns out that, one way or another, I have to use a second computer, which is simply not where to put it. However, most of the time he is idle idle. It turns out that the scheme is extremely inefficient from the point of view of resource use.

One day my dream turned into an emergency. It was necessary to assemble a graphic station with the following characteristics:
  1. Windows 7 operating system (hereinafter Windows);
  2. A set of software using DirectX, OpenGL and CUDA;
  3. High-speed, local, fault-tolerant storage of about 10 TB in size;
  4. The mechanism of backup and recovery of the entire system;
  5. Periodic automatic backup of user data.

Many may think: “And what about virtualization?” The problem is that Windows, in my personal opinion, is not very reliable. Often the end users of the system are not very qualified people, as a result of which malicious software that can destroy all data on all drives gets onto the computer. In this case, it is necessary that the backup is stored locally, but it was not possible to destroy it. Organizing a large and fast data warehouse is also not a trivial task. One way or another, it was decided to run Windows in a virtual machine (VM) environment.

I read a lot of articles and documentation (including on the hub [ 1 - 3 ]) about forwarding video cards to the VM environment. The conclusion I came to was not comforting for me. Most successful forwarding was carried out with ATI graphics cards. Throwing an NVIDIA graphics card is possible, but by no means every. In this case, it is required to patch the hypervisor [ 4 , 5 ].

Due to the need to use CUDA technology, I needed to forward the NVIDIA graphics card. There is NVIDIA Multi-OS technology [ 6 ], which allows you to forward video cards to VMs. Only Quadro series accelerators [ 7 ] support this technology .

Another drawback of most video cards is their minimum two-slot width. This is often unacceptable and that is why the single-slot NVIDIA Quadro 4000 graphics card was chosen. It is the most powerful representative of the Quadro family among single-slot options.

Key elements of the system:
  1. Motherboard ASUS P9X79 WS;
  2. Core i7-3820 processor;
  3. RAM 32 GB (4 strips of 8 GB);
  4. Video card for dom0 (no matter which);
  5. Video card for domU NVIDIA Quadro 4000.

Initially, I wanted to implement my plan using the Xen hypervisor. As the OS for dom0, I chose Debian GNU / Linux wheezy (hereinafter referred to as Debian). The Linux kernel has an excellent implementation of software RAID arrays.

During the experiments, I tried KVM. He did a great job with any device except a video card. Unfortunately, with Xen 4.1 , which is part of common distributions, I also failed. In the virtual machine environment, the video card stubbornly refused to start.

Fortunately, not long before I began to deal with this problem, Xen 4.2.0 was released . It is available, of course, only in the form of source codes.

In order to avoid problems with disconnecting the devices necessary for forwarding from dom0, the xen-pciback driver must be included in the kernel. On Debian distributions, it comes as a module.

Thus, it was necessary to assemble the hypervisor and core.

Install Debian


When installing Debian, I did not do anything unusual. Installed only the base system. The graphical environment should not be installed, since a lot of software is installed with it, some of which will conflict with the assembled Xen.

At the end of the basic installation, I installed the minimum set of necessary software.

$ sudo apt-get install gnome-core gvncviewer mc

Hypervisor and kernel assembly


In order not to clog the combat system, I performed the assembly in a clean OS specially installed under VirtualBox.

To build Xen, you need to run a series of straightforward commands [ 8 ].

$ sudo apt-get build-dep xen
$ sudo apt-get install libglib2.0-dev libyajl-dev fakeroot bison flex libbz2-dev liblzo2-dev
$ wget http://bits.xensource.com/oss-xen/release/4.2.0/xen-4.2.0.tar.gz
$ tar xf xen-4.2.0.tar.gz
$ cd xen-4.2.0
$ ./configure --enable-githttp
$ echo "PYTHON\_PREFIX\_ARG=--install-layout=deb" > .config
$ make deb

At the output I received the xen-upstream-4.2.0.deb package.

The kernel was built due to the need to include the xen-pciback driver in the kernel. The build process is well described in the official Debian documentation [ 9 ].

$ sudo apt-get install linux-source
$ sudo apt-get build-dep linux-latest
$ sudo apt-get install kernel-package fakeroot
$ tar xf /usr/src/linux-source-3.2.tar.bz2
$ cd linux-source-3.2
$ cp /boot/config-`uname -r` .config
$ sed -i 's/CONFIG_XEN_PCIDEV_BACKEND=.*$/CONFIG_XEN_PCIDEV_BACKEND=y/g' .config
$ make-kpkg clean
$ fakeroot make-kpkg --initrd --revision=1.0.0 kernel_image

The output received linux-image-3.2.32_1.0.0_amd64.deb.

Installing the kernel and hypervisor


I installed the received packages on the target system. When you install Xen in the / boot directory, unnecessary symbolic links are created that need to be removed. Since dependencies are not registered in the compiled package xen-upstream-4.2.0.deb , the packages required for Xen to work must be installed manually.

$ sudo dpkg -i linux-image-3.2.32_1.0.0_amd64.deb
$ sudo dpkg -i xen-upstream-4.2.0.deb
$ sudo su -c "cd /boot; rm xen.gz xen-4.gz xen-4.2.gz xen-syms-4.2.0"
$ sudo apt-get install libyajl2 liblzo2-2 libaio1

Making Xen bootable by default.
$ sudo dpkg-divert --divert /etc/grub.d/08_linux_xen --rename /etc/grub.d/20_linux_xen

Register Xen services in the system.
$ sudo update-rc.d xend start 99 2 3 5 stop 99 0 1 6
$ sudo update-rc.d xencommons start 99 2 3 5 stop 99 0 1 6
$ sudo update-rc.d xendomains start 99 2 3 5 stop 99 0 1 6
$ sudo update-rc.d xen-watchdog start 99 2 3 5 stop 99 0 1 6

To hide the devices necessary for forwarding from dom0, the xen-pciback driver needs to submit a list of them. To disable Memory Ballooning, you must first specify the amount of memory required for dom0. To do this , add 2 lines to / etc / default / grub .

GRUB_CMDLINE_LINUX_XEN_REPLACE_DEFAULT = "xen-pciback.hide = (02: 00.0) (02: 00.1) <br> (00: 1a.0) (00: 1d.0) (09: 00.0) (0a: 00.0) (0b: 00.0 ) (03: 00.0) (00: 1b.0) "
GRUB_CMDLINE_XEN_DEFAULT = "dom0_mem = 15360M"

In addition to the video card, 4 USB controllers, a network and sound card, and a number of peripherals were forwarded.

After that, you need to update the GRUB bootloader configuration file.

$ sudo update-grub

Since the motherboard ASUS P9X79 WS has two network interfaces, it was decided not to make any network bridges. dom0 and domU will have independent network connections.

The following sequence of commands sets the minimum amount of memory for dom0 (thereby finally disabling Memory Ballooning) and disables unnecessary network scripts. I also made VNC available at any IP address.

$ sudo sed -i 's/^XENDOMAINS_SAVE=.*$/XENDOMAINS_SAVE=""/g' /etc/default/xendomains
$ cd /etc/xen
$ sudo sed -i 's/^.*.(dom0-min-mem.*)$/(dom0-min-mem 15360)/g' xend-config.sxp
$ sudo sed -i 's/^.*.(enable-dom0-ballooning.*)$/(enable-dom0-ballooning no)/g' xend-config.sxp
$ sudo sed -i 's/^(network-script.*)$/#&/g' xend-config.sxp
$ sudo sed -i 's/^(vif-script.*)$/#&/g' xend-config.sxp
$ sudo sed -i "s/^.*.(vnc-listen.*)$/(vnc-listen '0.0.0.0')/g" xend-config.sxp

Windows installation


I will not dwell on the issue of installing Windows in a VM environment. I can only say that the installation must be done remotely using VNC. NVIDIA Quadro starts to work only after installing the drivers. It starts only at the final boot of the OS. Paravirtual drivers have also been successfully installed [ 10 ].



Among network devices, there is a VirtualBox Host-Only Ethernet Adapter. The fact is that I installed VirtualBox in a VM environment (inside domU). Inside, Windows XP was installed. Even with double nesting inside the VM, Windows XP worked with acceptable performance.

Domain configuration file
builder = "hvm"
memory = 16384
vcpus = 8
name = "windows-7"
uuid = "830460b8-3541-11e2-8560-5404a63ce590"
disk = ['phy: / dev / storage / windows-7, hda, w']
boot = 'c'
pci = ['02: 00.0 ',' 02: 00.1 ',' 00: 1a.0 ',' 00: 1d.0 ', <br> '09: 00.0', '0a: 00.0', '0b: 00.0 ', '03: 00.0', '00: 1b.0']
gfx_passthru = 0
pae = 1
nx = 1
videoram = 16
stdvga = 1
vnc = 1
usb = 1
usbdevice = "tablet"
localtime = 1
xen_platform_pci = 1

Test results


I’ll make a reservation right away that I didn’t even try to install Windows on bare metal, so there’s nothing to objectively compare with me. The whole assessment came down to viewing the Windows Experience Index (WEI) and installing 3DMark 2011. That is, to be honest, I did not conduct performance tests as such.



WEI is determined by the test result with the worst result. In my case, the bottleneck was the “hard disk”. The fact is that I carried out all the experiments using the only one old hard drive. Otherwise, the results are not very bad.



The NVIDIA Quadro 4000 graphics card cannot be called gaming, but Crysis worked smartly.

Xen Notes


Starting with version 4.1 , the xl toolkit came to Xen. It does not need xend and in the future it should replace xm.

When working manually xl did not cause any complaints. However, the xendomains script is clearly not designed to work with it. When /etc/init.d/xendomains start was started , domU domains started without problems. However, when /etc/init.d/xendomains stop was called , nothing happened. DomU did not automatically shut down, as a result of which the whole system crashed. During a small investigation, it turned out that some of the functions inside xendomains were simply not adapted for working with xl. In principle, I was ready to handle these problems.

The worst thing was that when launching, domU xl gave an error saying that it could not reset a PCI device, namely a video card, through sysfs. For a video card, the reset file in sysfs is simply missing. I'm not completely sure, but it seems to me that precisely because of this, after rebooting domU, the video card often refused to start. Only restarting the entire system helped.

libxl: error: libxl_pci.c: 1001: libxl__device_pci_reset: <br> The kernel doesn't support reset from sysfs for PCI device 0000: 02: 00.0
libxl: error: libxl_pci.c: 1001: libxl__device_pci_reset: <br> The kernel doesn't support reset from sysfs for PCI device 0000: 02: 00.1

In xm, the reset of PCI devices seems to be implemented differently. At least when working with him there were no problems with the video card. That is why I have stopped so far on xm.

Conclusion


It took me 5 days to implement everything described. In general, the system worked very stably. No BSODs and freezes were observed. I hope that in the near future, without exception, all video cards will be suitable for forwarding to VMs, since NVIDIA Quadro can not be called a budget option.

List of sources used


  1. Forwarding a video card to a virtual machine
  2. Forwarding a video card to a guest OS from the Xen hypervisor
  3. Forwarding a video card in Xen, from under Ubuntu
  4. Xen VGA Passthrough
  5. VGA device assignment - KVM
  6. NVIDIA Multi-OS
  7. Graphics and Virtualization - NVIDIA
  8. Compiling Xen From Source
  9. Compiling a new kernel
  10. Xen Windows GplPv / Installing