A journey through virtualization

This weekend I had a look on enabling the OpenGL compositor when running in a virtual machine. My assumption was that given that the next version of Ubuntu is going to require OpenGL, there should be no problem with getting OpenGL running in a virtual machine.

I normally use KVM when I have need for a virtual machine running a Linux guest. As far as I know you cannot get OpenGL with this technology, so I hadn’t looked into it for quite some time.

My first attempt was to just use the project neon weekly build. Unfortunately I failed to convert it to a VirtualBox image, nevertheless I decided to run it and verify that KVM does not support OpenGL. Well I was rather surprised once the machine was up to see that OpenGL was functional, but well it was slow. Restarting KWin showed me that there is a missing slot (which I am aware of and is already fixed in one of the pending reviews) in master which makes the fallback to XRender not work, but KWin just uses llvmpipe. OK, result incorrect, but at least validated that KVM is not working.

I then decided to try VirtualBox with an install of Kubuntu Natty. Install went well but no OpenGL available. The Xorg log file told me that VirtualBox does not know the XServer version. Installed the latest version of VirtualBox, installed the latest version of guest additions in the Kubuntu guest, but no success: driver does not work.

So I decided to test with an outdated system, aka Debian Testing as I’m also running this on my workstation. Here I was more successful. After installing the system in the virtual machine OpenGL worked out-of-the-box. And what surprised me even more was that OpenGL in KWin worked when using KWIN_DIRECT_GL=1 and KWIN_COMPOSE=O. This means OpenGL works in general and it’s only blocked by KWin not knowing the driver.

The main reason is that KWin does not use direct rendering if it finds an unknown driver and VirtualBox only provides OpenGL in direct rendering mode. If LIBGL_ALWAYS_INDIRECT is set, it falls back to Mesa’s software rasterization and KWin sets this environment variable if it doesn’t know the driver.

This meant I had to perform some coding to recognize the driver and had to test it in the virtual machine. Thanks to host and guest running the same version of Debian I was able to setup a NFS share and just use my normal development version of KWin.

After a little bit of hacking and fighting with NFS I got the driver reported correctly and KWin running with OpenGL without using environment variable hacks. But VirtualBox’s driver is not complete. Some methods we (optionally) use from GLX 1.3 are not implemented and trying to use it resulted in warnings. That unfortunately needed a hack inside KWin (it’s small and well contained, so it’s not much of an issue).

With VirtualBox working I decided to also look into VMware and have installed VMware Player for the first time in years. Converted my Debian image into a VMware compatible format, enabled 3D acceleration and started the system, just to be presented with warnings that 3D is not working.

The Internet told me to add

mks.gl.allowBlacklistedDrivers = TRUE

to the vmx file. But that did not resolve the issue. The two warnings went down to one, but still a warning. Apparently one has to install libtxc-dxtn-s2tc0 in order to get 3D support.

Installed and no warnings: yeah. Started the system, but no OpenGL. Installed the guest additions, but it told me that the system already has X11 integration. Looked into the Xorg log and yes it complained that something is wrong with the module. So no OpenGL inside Debian guest.

Resolved to my previous plan of trying Kubuntu 12.10. Started it in a live session and KWin is not composited. But glxinfo reported a driver, so I restarted KWin with KWIN_COMPOSE=O and there it was: OpenGL running also in VMware.

Looking at KWin’s output I was quite happy to see that it is a Gallium3D driver, which means we don’t need any adjustments at all. KWin uses OpenGL out-of-the-box, it was only not running, because of the live session of Kubuntu. So all that was needed was adding some detection code, that KWin doesn’t print out “unknown” in the debug output. Unfortunately using my KWin from Debian did not work, so I could not yet verify the output, but that can be done once it hit project neon.

Once the patches are merged into master KWin will use OpenGL in virtual machines – if supported. But given the experience it looks like at least on Linux hosts OpenGL is basically not supported in default setups. I would like to also support Parallels and Windows Virtual PC, but am of course lacking both required host operating system and licenses.