Running KWin’s auto test suite

KDE’s sysadmins set up a new CI system and over the last weeks I have been helping in getting the yellow dots blue again. As I think it’s quite useful I decided to write a blog post on how to setup the test environment for KWin.

For setting up a test suite of KWin one also needs to consider the most important dependencies. In case of KWin these are KWindowSystem, which provides a library for building an X11 window manager, and KWayland, which provides a library for building a Wayland compositor. Given that, let’s start with those.

Testing KWindowSystem

KWindowSystem is a platform independent library to provide access to windowing system functionality. For the X11 part it has an implementation for the NetWM specification and that’s what is important for KWin. The NetWM implementation has a window manager mode which KWin uses. So all the “what’s the window title”, “is the window maximized”, “minimize the window” is implemented in KWindowSystem and KWin only calls into this functionality.

For frameworks 5 this code got reimplemented using xcb and at the same time gained a nice test suite. According to the latest build the X11 implementation of KWindowSystem has a line coverage of 68 % with the NetWM implementation even having 83 %. Testing this piece of code is not trivial, classic unit testing with mocking is not really an option (I don’t care whether my code is correct with the mock object if the one and only X server behaves differently) so the tests are from a definition point of view integration tests.

We have in KWindowSystem two kind of tests: some which need a running window server and X server and some which don’t. The recommendation is to run the tests with Xvfb and openbox as the window manager on Xvfb. This might be surprising given that our aim is to test KWin, so why openbox? KWindowSystem is a dependency of KWin. When you test KWindowSystem you normally don’t have KWin yet, so another window manager needs to be used. Btw. our test suite found bugs in openbox 😉

The tests which don’t need a window manager as they test the window manager functionality start their own Xvfb from within the test to have a clean setup.

The framework has also a small Wayland part and we have a test which starts weston. So overall to test KWindowSystem you need:

  • Xvfb
  • openbox
  • Weston

Testing KWayland

KWayland is our young framework which implements a Qt style wrapper around the Wayland server and client API. It has been implemented in a test-driven manner from the start. The line coverage is around 90 % and to a large degree the client side only exists to be able to properly test the server side.

Testing KWayland thus does not require lots of effort, most tests don’t need any environment. There is one test which needs Weston (fullscreen shell, the only interface we only have in the client, but not in the server library) and one which starts a test with a QtWayland powered Qt application. So one needs

  • Weston
  • QtWayland QPA

Testing KWin

Now the real challenge: KWin. We still don’t have KWin blue, one test is waiting for newer software in the base image. KWin uses a combination of unit tests, integration tests and full system tests with deep introspection. The last part is where most code gets tested and your bugs normally get fixed by writing a new test, see for example latest bug fix.

KWin uses for the unit and integration tests the Xvfb+Openbox variant just like KWindowSystem. If you run the test locally: remember to start Xvfb and don’t run them on your actual X server, they will fail. We have tests which assume the default DPI of Xvfb and on your high resolution display they will fail. You can spend quite some time investigating why they fail if you forget that. Also remember to set QT_QPA_PLATFORM=xcb when running the test suite if you are on Wayland. Your tests will fail if not and you will spend quite some time investigating why they fail.

There is one test which needs yet another Xserver: Xephyr. It’s a test to verify the screen setup.

The most requirements have the system tests. The system tests (in KWin code called “integration” tests) start a full KWin on KWin’s virtual framebuffer platform. As it’s a full KWin it will behave like KWin to the outside world. It registers on DBus, it tries to get kglobalaccel, it locks the screen, etc. It is important to separate the execution. If you use ctest each test is run in an own dedicated dbus session. If you invoke the tests manually, please remember to first run export $(dbus-launch). If not your global shortcuts of the KWin session are gone.

Of course the tests bring in yet another X server: Xwayland. Like KWin itself it needs Xwayland. So the list of X servers to test KWin is now: Xvfb, Xephyr and Xwayland.

The next requirement is breeze. Please compile KWin with breeze support. Otherwise KWin will have to use Aurorae and that uses QtQuick and might fail on the virtual framebuffer. That brings us to OpenGL. Our old CI system couldn’t do OpenGL and now our new system is finally able to execute the OpenGL tests. For this you need to compile KWin with gbm support and the system you are executing the test should either have a render node or the drm/vgem device. If the system does not have a /dev/dri/card0 the test is skipped. If you use a vgem device please make sure to give the user running the test the right to read/write the dev node. Unlike “normal” drm/gbm plugin KWin does not use logind to open the drm device in the virtual environment.

KWin also starts some external applications during the tests. If we find an application which e.g. crashes KWin it’s best to integrate it in the test suite. Due to that the test suite runtime depends on glxgears (yes we had a crash when closing glxgears). Like KWayland KWin also tests Qt and thus needs QtWayland and the xcb platform (one test starts an application once under Wayland and once under X11).

And the most interesting dependency is the DMZ-White cursor theme. We have tests verifying that the cursor handling works correctly and needed a good theme which has the sizes we want to test and the cursors we want to test.

So overall we have:

  • Run tests in own dedicated Dbus session
  • export QT_QPA_PLATFORM=xcb
  • run Xvfb with openbox
  • Xephyr
  • Xwayland
  • Weston
  • glxgears
  • Qt Wayland
  • Qt XCB platform
  • gbm as compile time dependency
  • breeze as compile time dependency
  • vgem drm device
  • DMZ-White cursor theme

And I’m sure the list is going to grow as we create more tests. I’m especially looking forward to test our OpenGL renderer. Now that we have support on the CI system for it, this becomes really interesting.

5 Replies to “Running KWin’s auto test suite”

  1. Have you guys looked into extending the coverage on openQA(I expect that KDE neon will start using it too, sooner or later)?

    1. Answering for the case of KWin: there openQA doesn’t make that much sense. OpenQA is more a solution of testing a whole system or a whole distribution. What openQA can provide to KWin we can test with our own test suite in a more suited way. Let’s say it like that: KWin doesn’t care how a panel looks like as long as it behaves like a panel. Thus we test the functionality of the panel, we don’t very that it is visually at the correct position. On the other hand we verify in our tests that rendering is done correctly, but for that we again don’t need a panel and use blue and red surfaces.

      If on the other hand you want to ensure that all icons are correctly loaded and the panel is truly at the bottom and populated in the correct way: then openQA is the right tool.

  2. Dear Martin Gräßlin

    I am grateful for every decision made to offer us one of the most advanced Desktop Environments in the world, I am a user and soon a sponsor of KDE and I have great admiration for your work, a strong hug and live the KDE community. 🙂

  3. KWin’s excellence is unsurpassed. Thank you for all your hard work.

    I am certain this test suite will be useful in my porting adventures. I am also sure this will help driver developers. This is great. It’s going to save a lot of people a lot of time.

    I hope I have good stuff to send upstream to you one day. Your work is inspiring.

Comments are closed.