Running KWin’s auto test suite

KDE’s sysadmins set up a new CI system and over the last weeks I have been helping in getting the yellow dots blue again. As I think it’s quite useful I decided to write a blog post on how to setup the test environment for KWin.

For setting up a test suite of KWin one also needs to consider the most important dependencies. In case of KWin these are KWindowSystem, which provides a library for building an X11 window manager, and KWayland, which provides a library for building a Wayland compositor. Given that, let’s start with those.

Testing KWindowSystem

KWindowSystem is a platform independent library to provide access to windowing system functionality. For the X11 part it has an implementation for the NetWM specification and that’s what is important for KWin. The NetWM implementation has a window manager mode which KWin uses. So all the “what’s the window title”, “is the window maximized”, “minimize the window” is implemented in KWindowSystem and KWin only calls into this functionality.

For frameworks 5 this code got reimplemented using xcb and at the same time gained a nice test suite. According to the latest build the X11 implementation of KWindowSystem has a line coverage of 68 % with the NetWM implementation even having 83 %. Testing this piece of code is not trivial, classic unit testing with mocking is not really an option (I don’t care whether my code is correct with the mock object if the one and only X server behaves differently) so the tests are from a definition point of view integration tests.

We have in KWindowSystem two kind of tests: some which need a running window server and X server and some which don’t. The recommendation is to run the tests with Xvfb and openbox as the window manager on Xvfb. This might be surprising given that our aim is to test KWin, so why openbox? KWindowSystem is a dependency of KWin. When you test KWindowSystem you normally don’t have KWin yet, so another window manager needs to be used. Btw. our test suite found bugs in openbox 😉

The tests which don’t need a window manager as they test the window manager functionality start their own Xvfb from within the test to have a clean setup.

The framework has also a small Wayland part and we have a test which starts weston. So overall to test KWindowSystem you need:

• Xvfb
• openbox
• Weston

Testing KWayland

KWayland is our young framework which implements a Qt style wrapper around the Wayland server and client API. It has been implemented in a test-driven manner from the start. The line coverage is around 90 % and to a large degree the client side only exists to be able to properly test the server side.

Testing KWayland thus does not require lots of effort, most tests don’t need any environment. There is one test which needs Weston (fullscreen shell, the only interface we only have in the client, but not in the server library) and one which starts a test with a QtWayland powered Qt application. So one needs

• Weston
• QtWayland QPA

Testing KWin

Now the real challenge: KWin. We still don’t have KWin blue, one test is waiting for newer software in the base image. KWin uses a combination of unit tests, integration tests and full system tests with deep introspection. The last part is where most code gets tested and your bugs normally get fixed by writing a new test, see for example latest bug fix.

KWin uses for the unit and integration tests the Xvfb+Openbox variant just like KWindowSystem. If you run the test locally: remember to start Xvfb and don’t run them on your actual X server, they will fail. We have tests which assume the default DPI of Xvfb and on your high resolution display they will fail. You can spend quite some time investigating why they fail if you forget that. Also remember to set QT_QPA_PLATFORM=xcb when running the test suite if you are on Wayland. Your tests will fail if not and you will spend quite some time investigating why they fail.

There is one test which needs yet another Xserver: Xephyr. It’s a test to verify the screen setup.

The most requirements have the system tests. The system tests (in KWin code called “integration” tests) start a full KWin on KWin’s virtual framebuffer platform. As it’s a full KWin it will behave like KWin to the outside world. It registers on DBus, it tries to get kglobalaccel, it locks the screen, etc. It is important to separate the execution. If you use ctest each test is run in an own dedicated dbus session. If you invoke the tests manually, please remember to first run export $(dbus-launch). If not your global shortcuts of the KWin session are gone.

Of course the tests bring in yet another X server: Xwayland. Like KWin itself it needs Xwayland. So the list of X servers to test KWin is now: Xvfb, Xephyr and Xwayland.

The next requirement is breeze. Please compile KWin with breeze support. Otherwise KWin will have to use Aurorae and that uses QtQuick and might fail on the virtual framebuffer. That brings us to OpenGL. Our old CI system couldn’t do OpenGL and now our new system is finally able to execute the OpenGL tests. For this you need to compile KWin with gbm support and the system you are executing the test should either have a render node or the drm/vgem device. If the system does not have a /dev/dri/card0 the test is skipped. If you use a vgem device please make sure to give the user running the test the right to read/write the dev node. Unlike “normal” drm/gbm plugin KWin does not use logind to open the drm device in the virtual environment.

KWin also starts some external applications during the tests. If we find an application which e.g. crashes KWin it’s best to integrate it in the test suite. Due to that the test suite runtime depends on glxgears (yes we had a crash when closing glxgears). Like KWayland KWin also tests Qt and thus needs QtWayland and the xcb platform (one test starts an application once under Wayland and once under X11).

And the most interesting dependency is the DMZ-White cursor theme. We have tests verifying that the cursor handling works correctly and needed a good theme which has the sizes we want to test and the cursors we want to test.

So overall we have:

• Run tests in own dedicated Dbus session
• export QT_QPA_PLATFORM=xcb
• run Xvfb with openbox
• Xephyr
• Xwayland
• Weston
• glxgears
• Qt Wayland
• Qt XCB platform
• gbm as compile time dependency
• breeze as compile time dependency
• vgem drm device
• DMZ-White cursor theme

And I’m sure the list is going to grow as we create more tests. I’m especially looking forward to test our OpenGL renderer. Now that we have support on the CI system for it, this becomes really interesting.