Evolving 3D desktop effects in Plasma

The latest Plasma release dropped a few desktop effects: the cube family, CoverSwitch and FlipSwitch. All of those effects were written back in 2008, the early days of KDE 4.x and the early days of desktop effects in KWin. The effects were implemented by me and when Vlad asked about removing them I saw the need for this and supported this step for technical reasons. With this blog post I want to share a little bit of why it was needed to remove them and why this means that they can come back in better ways than ever before.

State of CPU in 2008

To really understand this we need to time travel back to 2008 and the years before when desktop effects were introduced. This can help to understand how the hardware architecture changed and how that influenced design decisions in the effects API which are nowadays problematic. First of all CPUs. The Intel Core 2 Duo architecture was launched in 2006 as the brand new thing which had multiple (namely 2) cores which slowly replaced the NetBurst architecture which dominated Desktop computing for the beginning of that decade.

Also for developers these multi-core systems were a new thing. And that meant lots of software written before was single threaded. Especially KWin at that time was single threaded as it also depended on libraries which were not really thread safe such as the Xlib library and back at that time even OpenGL. Even years later when Qt introduced the threaded rendering in Qt 5 on many Mesa drivers this was disabled due to thread safety issues. Nobody would have thought of having any benefit of a threaded compositing approach back in 2007 with the state of hardware and the available libraries. Thread libraries were of course already available such as QtConcurrent or ThreadWeaver, but not of a useful use in KWin. This means the API written back then did not support ideas like doing rendering on a second thread or even rendering for each screen in a thread.

State of OpenGL in 2008

KWin’s compositing pipeline was written for OpenGL 1.4 which was the only supported version in the Mesa stack. KWin supported shaders written in OpenGL Shading Language which replaced parts of the fixed functionality pipeline in OpenGL 1.4 and emulated the fixed functionality pipeline. This was only supported on the proprietary NVIDIA driver. OpenGL 3 was released in 2008, but it was years till it became available on the Linux desktop (according to Wikipedia in 2012).

Thus the design of the OpenGL compositing scene and desktop effect system was written for the fixed functionality and only with this in mind. While KWin gained support for the programmable pipeline and is the only supported way nowadays the design is still in place and problematic.

State of multi screen support

Multiple screens were not that common back in 2008 and there existed multiple competing technology approaches. There was the old “one X screen per screen” approach with a different DISPLAY variable for every screen. There was Xinerama, with the NVIDIA implementation called TwinView and there was the “new kid on the block” XRandR. Using multiple screens still required hacking the xorg.conf file and then with luck it was working. Especially if you had an NVIDIA card which one needed for good desktop effect performance.

 

From https://xkcd.com/963/

From an X11 perspective there was (and even today) there is not such a thing as multi screen. For the compositor everything is one screen and we have to present all screens at the same time. So much for variable refresh rates (AMD FreeSync introduced in 2015), buffer age extension (implemented in KWin in 2013) and so on. From rendering point of view there was not much difference between rendering one screen or multiple screens. All we had in KWin was an integer variable telling us the count of known screens and the geometries.

State of Alt+Tab

Alt+Tab, which CoverSwitch and FlipSwitch provide, was still a hard coded implementation in QWidget. With the effect system an API was added to suppress the QWidget and use a desktop effect as replacement. This allowed to have an effect (BoxSwitch) which showed thumbnails for the windows. Overall this was rather a break with the other parts of the effects API as the effects are mostly used to influence position and drawing of the windows. With the Alt+Tab API an internal part of KWin was exposed and suppressed. It was it’s own API inside the API.

CoverSwitch and FlipSwitch even took this a step further by introducing 3D elements in the so far 2D world of desktop effects and completely intercepted the rendering. Most effects do not change the order how windows are rendered, but with those effects it was important to render the windows exactly in the order Alt+Tab wants them. So the effect intercepts the normal render calls and renders the windows in another order. To make this worse the CoverSwitch included reflections which meant the windows needed to be rendered twice. And the effects had to combine windows from multiple screens. The cube effect family did even more horrible things.

How Alt+Tab evolved

The biggest change to Alt+Tab happened thanks to QtQuick. While it was reworked before already and was no longer QWidget based, QtQuick allowed to easily define new layouts. The internal Alt+Tab API was one of the first parts to be exposed to QtQuick in KWin. Furthermore we introduced an interesting concept to make it possible to render window thumbnails in the QtQuick scene. This was done by telling the compositor to draw a thumbnail at a specific region of the QtQuick Alt+Tab window. As this was perfectly synced we finally had an easy to use API to make Alt+Tab layouts with all the fancy things only desktop effects could give us. Though we could not put other elements on top of the thumbnails (e.g. a close button) or transform them from within QtQuick. This was the end to the already mentioned BoxSwitch effect and the Alt+Tab mode in PresentWindows. Thus the only remaining effects for Alt+Tab are FlipSwitch and CoverSwitch.

With Qt 5 there was the hope to further improve that. Now also QtQuick used OpenGL for rendering which meant that in theory it would be possible to make our window textures available to QtQuick. Thanks to the work on the compositing engine lately this is now possible and gives even more flexibility to render Alt+Tab and makes it possible to implement FlipSwitch and CoverSwitch with QtQuick. This is really awesome as it means we have much better tools at hand to implement such fancy effects, we don’t have to develop our own toolkit and implement our own transition handling. Instead we can use all the great things like PathView in QtQuick. The C++ desktop effect implementation of CoverSwitch was 1000 lines of code, while the new emerging QtQuick based implementation is just about 200 lines of code.

How the effects evolved

Also the KWin effect system had a transition. We noticed that most effects are actually animations and added a dedicated implementation for it. This implementation is exposed to JavaScript and most of those effects are nowadays written in JavaScript. The effects in C++ are often the odd ones which do too much and use the wrong toolkit for the wrong thing. Such as I said years ago that PresentWindows and DesktopGrid need to be rewritten in QtQuick. The effect system should be for animating, not for developing a user interface. With QtQuick we have a way better toolkit.

Overall huge effects such as the cube family, CoverSwitch and FlipSwitch are standing in the way of evolving the effect system. While we have better tools to implement and maintain them thanks to QtQuick.

On forking

This blog post expresses my personal opinion. Currently I’m not an active KWin developer, thus I think I have an outside view. I’m not contributing due to personal reasons, mostly lack of time. I would love to start contributing again, but in the current pandemic situation I won’t find the time for it. Given that I’m just an armchair developer with some background view of KWin.

As the former maintainer of KWin I’m rather shocked to read that a fellow KWin developer announced a fork. Personally I find this very disappointing and very sad. In my opinion a fork should be the last option if any other option failed. There are very few legitimate reasons and I think a fork always harms both projects due to the split of development efforts and the bad publicity such a fork creates. The announcement renders KWin in a light it doesn’t deserve. There are of course successful forks such as X.org and LibreOffice.org, but those forks were created due to serious issues with the project and had most of the developers on board. I do not see any reason in the current KWin development! For the same reason I also do not like the lowlatency fork. Of course it is an important area where KWin needs improvements, but I would love to see that happen in the KWin repository and not in an outside repository. Working together, bringing the experience together is much better than working on your own.

Personally I am very happy how KWin evolved since I stepped down as the main developer and maintainer. I was afraid that my loss in activity could not be compensated and I am very, very happy to see that development activity in KWin is much higher than it was most of the years before. Looking at the mailing list I see new names and old ones. KWin looks very healthy to me from a developer perspective.

Having read the announcement and the reasoning for the fork, I was left puzzled. What went wrong? When I looked at the mailing lists I never noticed any conflicts. In fact there is even strong agreement on the areas which need work. Such as a reworked compositor pipeline, the KWayland situation, etc. Of course we need to be careful when rewriting, reworking central parts of KWin. One of the main areas in the work for preparing KWin for Wayland was to move the code base in ways allowing to rewrite parts without risking the stability of the whole project. Personally I think this served KWin well. And even if the KWin team does not want a quick rewrite of central parts it’s no reason for a fork. This still can be handled upstream, through branches. One could even release an “experimental” KWin release with central parts reworked. Overall I just don’t get it and hope that what matters is a good KWin.

Also I must say that if I wanted to write a new Wayland compositor I would not fork KWin. KWin started as an X11 window manager and those roots will always be there. In our decision on how to handle the transition to Wayland this was a central aspect as we still have to provide an X11 window manager for those who cannot and do not want to switch. Thus we needed to evolve KWin instead of starting something new. But for a Wayland first compositor I would not start with KWin.

This blog post has comments disabled as I won’t have the time to moderate or answer.

KWin/X11 is feature frozen

Yesterday the KDE Community released the Beta for Plasma 5.12 LTS. With that release the feature freeze for 5.12 is in place and also an eternal feature freeze for KWin/X11. To quote the release announcement: “5.12 is the last release which sees feature development in KWin on X11. With 5.13 onwards only new features relevant to Wayland are going to be added.” This raised quite some questions, concerns and misunderstandings in the social networks. With this blog post I try to address those question and explain why this change in policy is done.

Is KWin/X11 still maintained?

Yes! We are just in the process of releasing an LTS. Of course KWin is fully maintained in the LTS life time. While in 5.8 only X11 was maintained, now we are able to offer maintenance for both X11 and Wayland. For the maintenance we do not differentiate between windowing systems.

Will X11 bugs still be fixed?

As X11 is under maintenance, I expect bugs to still get fixed.

Does this mean that in 5.13 X11 will be unmaintained?

We are going to forward port bug fixes from the 5.12 branch to master. Thus any release after 5.12 will get all bug fixes from 5.12. Given that I would say 5.13 will also be maintained on X11.

Does this mean that in the next LTS X11 will be unmaintained?

We will decide when the time comes. Currently I do not expect that we would drop maintenance.

Does this mean Plasma 5.13 will default to Wayland?

This is about feature freeze for X11. Whether Wayland will be the default or not is completely unrelated to this.

Will X11 users not get any new features in KWin?

Of course there will be new features! Most functionality in KWin is independent of the windowing system. Any improvement to those areas benefit Wayland and X11 users. Currently we have a few improvements in the pipeline, for example tooltips on decoration buttons, improved blur effect, a rework of slide desktop effect, improvements to the cube effect and a few more. All of them will be available to both X11 and Wayland users.

How do you decide whether it’s an X11 only feature?

In the case of KWin this is very simple. There are areas in our code structure which are only get pulled in if run on X11, other areas are marked in an if (x11) section. If the new feature touches this code, it’s probably not going to be added.

Does this feature freeze also apply to other KDE software?

No, but personally I would recommend any maintainer to apply a similar feature freeze. I personally will not help developing any X11 specific feature any more and resigned as maintainer of various X11 specific frameworks this week.

What are you going to do if someone present a feature for X11?

It won’t be merged.

But why?

This requires a little bit more explanation. I had a look at the most prominent issues we had over the last years. Where are our stability problems, where are our quality problems, what costs most development time? I observed that it was always in new features specific to X11. Very often even for features we added to Wayland without causing problems.

So I started to look into why that’s so. The obvious answer that we don’t get bugs for Wayland because nobody uses is, is not the case. We get many bug reports for Wayland and many users are nowadays running Wayland. So the reason must be somewhere else.

On Wayland we are able to test the code. Let’s take an example: we get input events through libinput. For this we have unit tests through mocking. Thus we can automate the testing of the lowest layer. From libinput events are sent into KWin core through an internal API and that we can also use in the integration tests. So we can simulate everything from the point where libinput events would hit KWin core. We can test this properly, we know that we get all events (because KWin is the display manager) and we can test every aspect. We can lock the screen and verify how it works, we can make applications grab the pointer or keyboard and test this. We can invoke global shortcuts, etc. etc. It’s all just as if we get the events through libinput. The quality of these new areas in KWin feels really good.

A few weeks ago some commits I did hit the Linux media about KWin/Wayland without X11 starting too fast and due to that causing bugs. These issues were found through our test suite, before any user would ever be exposed to them.

What we did in the past was taking these new features and bring them to X11. But there we cannot test. There is no way on X11 to e.g. fake a touch screen. On X11 we cannot test how this behaves if we lock the screen or used Alt+Tab. We can write the code and manually test it. Hey it works, awesome! But that was mostly not the case. There were corner cases which caused trouble. And to this comes that the main devs run Wayland instead of X11. If features break they are not exposed to the bugs.

Could you give some examples of things that broke?

Sure! The first feature we backported to X11 was “modifier only shortcut”. We spent months fixing the fallout because of X11 weirdness. Another feature backported was panels on shared screen edges. It worked great for Wayland, but on X11 some corner cases were overseen and caused issues, which affected our users and required time to fix. We backported the touch screen swipe gesture on X11 which was a mess. On X11 touch screens are also mouse events, that made it difficult and created many corner cases.

The problem is not adding the feature. The problem is fixing the bugs created by the new features.

But if a new foo requires adjustments?

KWin won’t be adjusted to any new requirements in the XServer, Mesa, input stack, proprietary drivers, etc. If something breaks it’s the fault of those components which broke it.

Plasma/Wayland and NVIDIA – 2017 edition

More than a year ago I elaborated whether KWin should or should not add support for NVIDIA’s proprietary Wayland solution. I think it is time to look at the situation of Plasma/Wayland and NVIDIA again. In case you haven’t read my previous blog post on that topic I recommend to read it as I use it as the base for this blog post.

Compared to a year ago not much has changed: NVIDIA still does not support the standard Linux solution gbm, which is supported by all vendors and nowadays even going to enter the mobile space. E.g. the purism phone is going to have a standard graphics stack with gbm. So no additional code required. But NVIDIA doesn’t support gbm. Instead it has a proprietary implementation called EGLStreams, which no other vendor implements. Due to that Plasma/Wayland cannot support OpenGL for NVIDIA users.

But current KWin master branch (what will become 5.12) supports automatical fallback to Qpainter compositing in case OpenGL fails. With that it should be possible to run Plasma/Wayland on NVIDIA hardware. Granted it won’t be accelerated, but as explained above that is only NVIDIA’S fault. Also I have not tested this yet. It might be that NVIDIA doesn’t support dumb buffers on DRM.

In my blog post from last year I elaborated whether KWin should add support for NVIDIA’S proprietary solution. I put out a requirement that the patches for Weston need to be merged first:

For KWin such patches do not exist and we have no plans to develop such an adaption as long as the patches are not merged into Weston. Even if there would be patches, we would not merge them as long as they are not merged into Weston.

A year passed and nothing happened. I don’t expect anything further will happen: the patches won’t be merged into Weston. Given that it doesn’t make any sense to make this a requirement for KWin. If we hold up the requirement it would be like saying we would never accept it.

Last year I hoped that XDC would find a solution for the problem:

We do hope that at XDC the discussions will have a positive end and that there will be only one implementation.

And it looked positive: a new universal buffer allocation library got announced. I hoped that this would solve the situation pretty quickly. The new library would get implemented, Mesa and NVIDIA add support for it, we slightly adjust our code and drop gbm. Everybody happy.

But now another XDC passed and from what I understand not much process was made. The library is not ready and no driver uses it. We waited one year for it, I doubt waiting another year will change the situation. Realistically I think it will take another two to three years for the library to get implemented if at all. Then it will take about a year for Mesa gaining support and shipping a release. Yet another year for distributions shipping with that version so that we can depend on it without people killing us. So maybe in five years there will be a replacement. I don’t think we can wait that long.

So does this mean KWin will gain support for EGLStreams? Certainly not! I do not think that the KDE community should spend any time to support NVIDIA’s proprietary solution! We are a free software community and we should not implement code which only benefits proprietary non-free solutions. There are way more free things to do to improve Wayland without having to write code for proprietary solutions.

Also there is another aspect which I did not consider in my blog post last year: XWayland. XWayland also uses gbm and does not have support for NVIDIA’s proprietary solution. Due to that one does not have OpenGL for any legacy X application. This probably includes most games, but currently also browsers such as Firefox and Chromium. I don’t think that this would give a satisfying experience to NVIDIA users. And it is also hardly better than rendering everything through the QPainter compositor. This is another reason for the KDE Community to not spend any time on implementing support for EGLStreams.

But I think there is lots NVIDIA could do. Today I would accept a patch for EGLStreams in KWin if NVIDIA provides it. I would not be happy about it, but I would not veto it. If it is well implemented and doesn’t introduce problems for the gbm implementation I would not really have an argument against it. But I expect NVIDIA to do it. I don’t want a contribution from a non-NVIDIA developer. This mess was created by NVIDIA, NVIDIA needs to fix it.

Similar I think that NVIDIA should adjust XWayland. I understand that NVIDIA is not happy with the design of XWayland, but nevertheless they should make it work. Their users pay quite some money for the hardware. I think they have a right to demand from NVIDIA to fix this situation. Ideal would of course be NVIDIA adopting gbm. But as that seems unlikely, I think it is the duty of NVIDIA to provide patches for their users.

KWin/Wayland goes real time

Today I landed a change in KWin master branch to enable real time scheduling policy for KWin/Wayland. The idea behind this change is to keep the graphical system responsive at all times, no matter what other processes are doing.

So far KWin was in the same scheduling group as most other processes. While Linux’s scheduler is completely fair and should provide a fair amount of time to KWin, it could render the system hard to use. All processes are equal, but some processes are more equal than others. KWin – as the windowing system – is to a certain degree more equal. All input and all rendering events need to go through KWin.

We can now imagine situations where a system can become unusable due to KWin not getting sufficient time slots. E.g. if the user moves the mouse we expect a timely cursor motion. But if KWin is not scheduled the system is quickly starting to lagging. Basically what we expect is that when the mouse moves with the next page flip the cursor must be at the updated position. A page flip happens normally every 16 msec (60 Hz), so if we miss one or two we are in the area where a user visually notices this. For a gamer with a high precision mouse this could be quite problematic already.

Another situation is a few processes running amok and one wants to switch to a virtual terminal to kill the processes. The key combination (e.g. Ctrl+Alt+F1) needs to go through KWin. Again the user wants to have a responsive system in order to be able to switch the session.

While thinking about these general problems I came to the conclusion that we should try making KWin a real time process. This would ensure that KWin gets the CPU whenever it wants to have the CPU. This is actually quite important: KWin is only taking CPU if there is a reason for it, e.g. an input event or a window requesting a repaint, etc. So by adding a real time scheduling policy to KWin we can ensure that all input events and all rendering events are handled in a timely manner. My hope is that this in general results in slightly smoother experience and especially in a smoother experience if the system is under heavy load.

Of course I considered the disadvantages. If KWin is a real time process, it always wins. It’s on the scheduler’s fast lane. This could of course be a problem for other processes actually demanding a higher scheduling policy or which are real time itself. E.g. a game or a video player might already be real time. But here we also see that there is no point in having the game be real time for fast input if the windowing system itself is not real time. So this is actually not a disadvantage, but rather another reason to make KWin a real time process.

There could be other real time processes which are more important. Of course those should win against KWin. To support this KWin requests only the minimum priority, so any other real time process in the system is considered more important than KWin.

The only real “problem” I see would be KWin running amok, because then KWin would get too much CPU time. But in this case the system would be broken anyway. If e.g. KWin would be stuck in an endless loop one would not be able to switch to another VT as KWin is stuck in the endless loop. This is obviously only a theoretical example.

So how is it done? KWin gained a new optional dependency on libcap and on installation sets the capability CAP_SYS_NICE on the kwin_wayland binary. @Distributions: please update your build deps. On startup kwin_wayland adjusts the scheduler policy to request SCHED_FIFO and drops the capability again before doing anything else.

As this change is in master it won’t be part of the Plasma 5.11 release which also means we have now the maximum time span till it ends up in a release. I would be happy about users testing it and especially report any negative influences of the change.

Announcing the XFree KWin project

Over the last weeks I concentrated my work on KWin on what I call the XFree KWin project. The idea is to be able to start KWin/Wayland without XWayland support. While most of the changes required for it are already in Plasma 5.11, not everything got ready in time, but now everything is under review on phabricator, so it’s a good point in time to talk about this project.

Why an XFree KWin?

You might wonder why we spend time on getting KWin to run without X11 support. After all we need to provide support for XWayland anyway to be able to support legacy applications which are not yet on Wayland. So what’s the advantage if in a normal session one needs XWayland anyway?

One aspect is that it shows that our porting efforts to Wayland are finished. Over the last years I tried to start KWin without XWayland support for a few times just to find areas which are not yet ported. By being able to run KWin without X11 support we know that everything is ported or at least does not hard depend on X11 any more.

Another aspect is obviously Plasma Mobile which does not really require XWayland and also XWayland not making much sense on e.g. the libhybris enabled systems as Xwayland doesn’t have OpenGL there. By not requiring XWayland we can reduce our runtime and memory costs.

Speaking of runtime costs: not requiring X11 means that we don’t have to wait for XWayland during KWin startup. Instead XWayland can be started in parallel. This means KWin and the complete Plasma session starts a little bit faster.

And most important this is an important prerequisite to be able to handle a crashing XWayland. So far when XWayland crashed KWin terminated gracefully as KWin depends on X11. The hope is that when XWayland crashes we can just restart it and keep the session running.

How it was done

The general idea behind getting KWin X11 free is “code that isn’t loaded, cannot interfere”. KWin uses platform plugins (not Qt QPA plugins) for the various platforms KWin can run on. There is also a platform plugin for KWin/X11, so code which is only required in the KWin/X11 case can be moved into the platform plugin. As KWin/Wayland does not load this plugin we are certain that the code will not be loaded and thus cannot interfere.

But how to find code which is only required on KWin/X11? After all KWin’s code base is about 150 kSloc (according to cloc) and that makes it rather difficult. A good help here was our CI system which performs code coverage. KWin’s tests mostly are mostly based on KWin/Wayland so an area which does not have any test coverage is a good candidate for being X11 specific. By looking at these areas it was possible to find patterns which also helped to find more usages. A good help is KWin’s internal X11 API such as displayWidth, displayHeight, rootWindow and connection. The usage of these functions is partially so few that one could just evaluate each usage. As a result of this work the functions displayWidth and displayHeight are not used at all any more.

Plugin based compositors

Another idea was to get our compositors into plugins. Especially the XRender based compositor is not of much use in a Wayland world and thus should not be loaded into the binary. Unfortunately various parts of KWin called directly into the concrete compositor implementations, so to solve this we had to extend the internal API. In Plasma 5.11 the XRender and QPainter compositor are now loaded as plugins, so on Wayland the not-compatible XRender compositor is no longer loaded into memory and on X11 the not-compatible QPainter compositor is no longer loaded into memory. But also on Wayland the QPainter compositor is only loaded into memory if it is going to be used.

The OpenGL compositor is still always loaded in Plasma 5.11, but the change to have it as a plugin is already prepared and will be merged into master soonish. This will bring great advantages to the stability of the system: currently we are not able to define which platform supports which compositor as the initialization code just didn’t support being more flexible. But now with the plugin based approach I’m very confident that we can make this work in a better way.

Outlook

Being able to start and run KWin/Wayland without X11 support is of course only the start. More changes will be required. For example to delay loading XWayland until an application tries to connect to it (c.f. Weston). This would not make much sense in the start of Plasma yet as we still have applications in our startup which require X11 (e.g. ksmserver).

Another area is to try to get KWin compile without X11 support and to move everything required for Xwayland into a plugin. This will be a larger project as it will require to move much code around and to add further abstractions in some areas of KWin. Hint: this could be a nice Google Summer of Code project. As a fast step for especially Plasma Mobile and the Purism Librem phone an ifdef around every X11 code area could be a solution.

But my personal main goal is to be able to handle a crashing XWayland. This will also be an interesting task as the X11 code in KWin also depends on other libraries (e.g. KWindowSystem) which do not expect to outlive the X server. So that will be a challenging task to find all areas where KWin holds X11 data structures and to clean them up properly without crashing due to some cleanup code calling into xcb.

KWin requires C++14

This is a short public service announcement: KWin master as of today requires a compiler which supports C++14. This means at least gcc 5 or clang 3.4. All major distributions support at least one of the two.

Plasma Wayland and Qt 5.9 and beyond

As you might know Qt 5.8 created challenging problems for our Wayland session and threw our efforts back quite a bit. In this post I want to discuss the actual problems it created, how we are addressing them and looking into the future.

How our integration used to work

Our integration uses additional Wayland protocols. We have a protocol for server side window decorations which we use in our Plasma integration plugin to inform KWin whether the window should have a decoration or not. We have a protocol for client provided shadows which is e.g. used by our widget style Breeze to add shadows to the context menus. We have a protocol for the desktop shell, so that it can mark windows as desktop, panel, auto hiding panel, position the window, etc. Also we have a few protocols for interacting with our effect system, e.g. sliding popups, blur behind.

To use these protocols we need to interact with Qt in a low level way. We use the native interface in the Qt Platform Abstraction to get a wl_surface pointer for the QWindow. In order to not have to keep this simple for our applications our KWayland::Client API provides an API point for it: Surface::fromWindow(QWindow*) -> Surface*.

But when exactly to inject our own integration? We found a very handy way which worked much better than what we had used in the past for X11 (and based on that also transitioned X11 code to use it). Qt emits an event once it has created the native platform surface (in case of Wayland the wl_surface*) for a QWindow. Verbatim quote of the documentation:

The QPlatformSurfaceEvent class is used to notify about native platform surface events.

Platform window events are synchronously sent to windows and offscreen surfaces when their underlying native surfaces are created or are about to be destroyed.

Applications can respond to these events to know when the underlying platform surface exists.

Awesome! We get an event when the surface is created and when it gets destroyed. This made it very simple to create the integration and what is really important for us is that we get this event before the window is shown. So we can prepare everything so that KWin gets a good state.

And the KWin side was to a large part implemented on assumptions on how the sequence will work. We first get the surface, then the (xdg) shell surface, then the integration bits. Sure it would be nice if KWin handled also other sequences, but as the only implementation of this is Qt it doesn’t make sense to really care about it. We know that it was not perfect, we even had the test cases for it, which expect failed.

What broke with Qt 5.8?

In Qt 5.8 our complete integration broke. When the platform surface created event was emitted the wl_surface was not created, see QTBUG 58423. This is in my humble opinion a clear violation of the documented behavior and thus a breakage of the stability guarantees Qt provides, but others might disagree. After some discussion, trying out patches by those who had a Qt 5.8 build we had a patch for Qt which made things mostly work together with a patch to KWin. But at the time we had the patch ready Qt 5.8 was already declared end of life with no prospect of a Qt 5.8.1 bug fix release. For our Wayland session it was just impossible to get Qt 5.8 compatible again. All we could do was to advise distributions to not combine Qt 5.8 with the Wayland session. From our side it was fine for distros to ship Qt 5.8, but if they do they should make it impossible to install the Wayland session. The state was just too broken.

Qt 5.9

With Qt 5.9 the situation looks better. The required patch is merged, it’s an LTS release and we already had the first bug fix release. Qt wants to create more bug fix releases for it and this allows us to use it as a new target for integration. But still the situation is not as good as it used to be. If you currently use our Wayland session with Qt 5.9 you will still see quite some rough corners compared to where we were with Qt 5.7.

The main problem for us is that the platform surface regression was not the only change affecting us. Pre Qt 5.7 a wl_surface lived as long as a QWindow. Now the wl_surface gets destroyed whenever the window gets hidden and a new one created on every show. Unfortunately without a platform surface created event. This means our integration breaks as soon as a window gets hidden. E.g. after closing KRunner the integration for KRunner is broken.

We tried to address this problem in various places, but it is challenging. On the show event we don’t have the wl_surface yet (too early), on the expose event the window is already mapped (too late). This creates problems for KWin which is not prepared for the integration bits to hit when the window is mapped. Our protocols were designed with the platform surface created event semantics. For example in KRunner we face an issue with the integration. KRunner is a panel, which accepts focus and allows windows to go below, also it positions manually. Now when it gets re-shown KWin doesn’t know that this window is supposed to position itself and positions it. Now we get the request to put it as panel, KWin adjusts and moves maximized windows around. And then we get the request that the panel allows windows to go below. KWin shuffles the windows again as the maximized area changed again.

This is a rather tricky situation as we cannot really do something about it. If the window is mapped it’s too late. Even if we improve our API to handle the situation better it will be too late.

There are two possibilities to handle this: Qt stops to destroy the surface or sends a platform surface created event when it recreates the surface. The latter would be my personally preferred solution as this would match the documentation again and allow us to just use the one event handler.

Other regression

The situation around the changed behavior in Qt 5.8 caused a few steps back. Our code needed to be adjusted and that sometimes caused issues. We had a few regressions which also affected the compositor, so the stability of the whole system suffered. These issues are luckily investigated and fixed. But there are still bugs lurking in the system.

For me personally the most annoying bug is a crash in Qt which affects the auto completion of kdevelop. This makes hacking in a Wayland session rather difficult. I’m running currently a patched KWin which disables the virtual keyboard integration to not hit this issue.

A huge problem is that context menus are not marked as transient windows. This means that the Wayland compositor does not know that it is a menu and positions the menu anywhere on the screen. It gives the system a very unfinished touch.

If KRunner is closed through the escape key, the key starts to repeat on the window constantly and due to that it is not possible to open KRunner again. Similar if you start an application in Kickoff through the enter key, when opening Kickoff again it automatically launches the currently selected item. This again makes it very difficult to use the session and gives the whole system an unfinished look. We are working on a workaround for this issue in the server.

Towards the future

Qt 5.9 is here to stay and that’s what we have to use as integration target. Given that Qt 5.9 and Qt 5.7 behave very differently it will become difficult for us to maintain support for both. My suggestion is that we drop support for Qt 5.7 and require Qt 5.9 for the Wayland session. In addition there is hope that we can improve the integration. Marco and David have been working on adding support for XDG Shell unstable v6 which is already supported by Qt and makes it easier to integrate with. Once this landed in KWin Qt will be switched from wl_shell to xdg shell. This will improve the situation for us quite a bit as we then have one code path for both Qt and GTK applications.

How to prevent such regressions in the future

The change of behavior in Qt 5.8 threw our Wayland efforts back a few months. This is something we communicated to Qt quite early and it’s something which worries me a lot. We cannot spend time on changing our integration every time Qt releases a new version. Given that we need to look into how to prevent that such a situation happens again.

I hope that we can improve the integration on the testing front between Qt and KDE. We have a huge test suite which can find regressions in Qt. If Qt would run KDE’s tests during the integration phase Qt would notice regressions before they hit the code base. Given that all our tests are free software it should be possible for Qt to integrate them.

But also the other side would be interesting: if we could get the latest Qt into our CI system we could also discover breakages early. We have now a new docker based CI system which allows running multiple builds of the same change (e.g. Plasma gets build on openSUSE tumbleweed and on FreeBSD) – an image with a daily or weekly Qt snapshot could help us and Qt a lot to detect breakages early.

I also hope in openQA which allows to test the full operating system. This would spot regressions like the misplaced context menus even if KWin’s own auto tests would not spot them (KWin doesn’t care about Qt there, only about the Wayland and X11 protocols). There we might need to invest some work to make sure that KWin/Wayland can be properly run in the openQA tests.

I hope that our Plasma devs can discuss this in more detail with Qt devs during Akademy in person. Unfortunately I cannot be at Akademy this year 🙁 so I cannot discuss in person.

Last but not least it is important that developers test. It would help a lot if the developers working on QtWayland test their changes in a running Plasma Wayland session. We are now overall in a state that the session is suited for hacking on. I do all my Wayland hacking in a Wayland session, experiencing all the glitches like kdevelop crashing.

Of course you might wonder what about us? Shouldn’t we KDE devs also test against the latest Qt? For me personally that is not the case. I’m working on the server side and not on the client side. I’m also not testing the latest GTK for example. Nevertheless I tried to use Qt 5.9 before it got released. Used the installer, spent a day to compile everything on top of it it, just to notice that it doesn’t have QtWayland and won’t get it. I didn’t give up that easily. So I tried to compile QtWayland myself. But when I tried to use it, it turned out to not have any keyboard support, because qtbase was compiled without support for xbkcommon. At that point I gave up. Not having QtWayland is one thing, but not being able to use keyboard is another, it’s rather pointless. The only other option is to compile Qt, but that is hardly an option as it’s really difficult to compile an actually working Qt with all components. The last times I tried, I failed, wasting days compiling. If there were usable weekly images for Qt I would be happy to try it, but of course only with a properly compiled and included QtWayland.

Running KWin’s auto test suite

KDE’s sysadmins set up a new CI system and over the last weeks I have been helping in getting the yellow dots blue again. As I think it’s quite useful I decided to write a blog post on how to setup the test environment for KWin.

For setting up a test suite of KWin one also needs to consider the most important dependencies. In case of KWin these are KWindowSystem, which provides a library for building an X11 window manager, and KWayland, which provides a library for building a Wayland compositor. Given that, let’s start with those.

Testing KWindowSystem

KWindowSystem is a platform independent library to provide access to windowing system functionality. For the X11 part it has an implementation for the NetWM specification and that’s what is important for KWin. The NetWM implementation has a window manager mode which KWin uses. So all the “what’s the window title”, “is the window maximized”, “minimize the window” is implemented in KWindowSystem and KWin only calls into this functionality.

For frameworks 5 this code got reimplemented using xcb and at the same time gained a nice test suite. According to the latest build the X11 implementation of KWindowSystem has a line coverage of 68 % with the NetWM implementation even having 83 %. Testing this piece of code is not trivial, classic unit testing with mocking is not really an option (I don’t care whether my code is correct with the mock object if the one and only X server behaves differently) so the tests are from a definition point of view integration tests.

We have in KWindowSystem two kind of tests: some which need a running window server and X server and some which don’t. The recommendation is to run the tests with Xvfb and openbox as the window manager on Xvfb. This might be surprising given that our aim is to test KWin, so why openbox? KWindowSystem is a dependency of KWin. When you test KWindowSystem you normally don’t have KWin yet, so another window manager needs to be used. Btw. our test suite found bugs in openbox 😉

The tests which don’t need a window manager as they test the window manager functionality start their own Xvfb from within the test to have a clean setup.

The framework has also a small Wayland part and we have a test which starts weston. So overall to test KWindowSystem you need:

  • Xvfb
  • openbox
  • Weston

Testing KWayland

KWayland is our young framework which implements a Qt style wrapper around the Wayland server and client API. It has been implemented in a test-driven manner from the start. The line coverage is around 90 % and to a large degree the client side only exists to be able to properly test the server side.

Testing KWayland thus does not require lots of effort, most tests don’t need any environment. There is one test which needs Weston (fullscreen shell, the only interface we only have in the client, but not in the server library) and one which starts a test with a QtWayland powered Qt application. So one needs

  • Weston
  • QtWayland QPA

Testing KWin

Now the real challenge: KWin. We still don’t have KWin blue, one test is waiting for newer software in the base image. KWin uses a combination of unit tests, integration tests and full system tests with deep introspection. The last part is where most code gets tested and your bugs normally get fixed by writing a new test, see for example latest bug fix.

KWin uses for the unit and integration tests the Xvfb+Openbox variant just like KWindowSystem. If you run the test locally: remember to start Xvfb and don’t run them on your actual X server, they will fail. We have tests which assume the default DPI of Xvfb and on your high resolution display they will fail. You can spend quite some time investigating why they fail if you forget that. Also remember to set QT_QPA_PLATFORM=xcb when running the test suite if you are on Wayland. Your tests will fail if not and you will spend quite some time investigating why they fail.

There is one test which needs yet another Xserver: Xephyr. It’s a test to verify the screen setup.

The most requirements have the system tests. The system tests (in KWin code called “integration” tests) start a full KWin on KWin’s virtual framebuffer platform. As it’s a full KWin it will behave like KWin to the outside world. It registers on DBus, it tries to get kglobalaccel, it locks the screen, etc. It is important to separate the execution. If you use ctest each test is run in an own dedicated dbus session. If you invoke the tests manually, please remember to first run export $(dbus-launch). If not your global shortcuts of the KWin session are gone.

Of course the tests bring in yet another X server: Xwayland. Like KWin itself it needs Xwayland. So the list of X servers to test KWin is now: Xvfb, Xephyr and Xwayland.

The next requirement is breeze. Please compile KWin with breeze support. Otherwise KWin will have to use Aurorae and that uses QtQuick and might fail on the virtual framebuffer. That brings us to OpenGL. Our old CI system couldn’t do OpenGL and now our new system is finally able to execute the OpenGL tests. For this you need to compile KWin with gbm support and the system you are executing the test should either have a render node or the drm/vgem device. If the system does not have a /dev/dri/card0 the test is skipped. If you use a vgem device please make sure to give the user running the test the right to read/write the dev node. Unlike “normal” drm/gbm plugin KWin does not use logind to open the drm device in the virtual environment.

KWin also starts some external applications during the tests. If we find an application which e.g. crashes KWin it’s best to integrate it in the test suite. Due to that the test suite runtime depends on glxgears (yes we had a crash when closing glxgears). Like KWayland KWin also tests Qt and thus needs QtWayland and the xcb platform (one test starts an application once under Wayland and once under X11).

And the most interesting dependency is the DMZ-White cursor theme. We have tests verifying that the cursor handling works correctly and needed a good theme which has the sizes we want to test and the cursors we want to test.

So overall we have:

  • Run tests in own dedicated Dbus session
  • export QT_QPA_PLATFORM=xcb
  • run Xvfb with openbox
  • Xephyr
  • Xwayland
  • Weston
  • glxgears
  • Qt Wayland
  • Qt XCB platform
  • gbm as compile time dependency
  • breeze as compile time dependency
  • vgem drm device
  • DMZ-White cursor theme

And I’m sure the list is going to grow as we create more tests. I’m especially looking forward to test our OpenGL renderer. Now that we have support on the CI system for it, this becomes really interesting.

How input works – touch screen edge swipe gestures

Continuing my series about how input works in KWin/Wayland I want to discuss a brand new feature we implemented for Plasma 5.10. This year we had a developer sprint in Stuttgart and discussed what kind of touchpad and touch screen gestures we want to support and how to implement it. Now the result of this discussion got merged into our master branch and we are currently discussing which actions to use by default.

Gesture Recognizer

Touchpad and touch screen gestures are kind of similar so the approach we took is able to handle both of them. We introduced a GestureRecognizer which is able to recognize gestures (surprise) in a very abstract way. It doesn’t know how the input events look like, whether a touch screen, touchpad, mouse or whatever input device generated the event.

To use the GestureRecognizer a Gesture needs to be registered. The Gesture describes the actual Gesture which needs to be recognized. E.g. how many fingers need to participate in the gesture (for our touch screen gestures it is one, for our touchpad gestures it is four), the direction of the gesture (leftwards, rightwards, downwards, upwards), the minimum distance, the trigger position, etc. etc.

Now input events can be fed into the GestureRecognizer and the GestureRecognizer decides whether a Gesture is active or becomes non-active. As said this is absolutely generic, it doesn’t care how the events are triggered.

This alone does not yet allow to do anything with it, we don’t have any way to use the GestureRecognizer yet.
At this point our implementations for touchpad and touch screen divide. We have different existing implementations which are more suited than trying to have something shared for both.

Touchpad gestures

For touchpad gestures our global shortcuts handling is used. GlobalShortcutsManager is a KWin internal (!) mechanism to register some internal actions to tigger in a global way through input events. The GlobalShortcutsManager gained a GestureRecognizer and various areas in KWin can now register a QAction as a global touchpad gesture.

So far we still haven’t reached the elements we discussed in the previous posts like InputEventFilter. There is of course an InputEventFilter which feeds events into the GlobalShortcutsManager. This filter got extended to support touchpad gestures and now we have the full stack together:

  • libinput reports a touchpad gesture event
  • InputEventFilter passes the touchpad gesture event to the GlobalShortcutsManager
  • The GlobalShortcutsManager passes the information to the GestureRecognizer
  • The GestureRecognizer triggers the QAction
  • something happens

By default the following gestures are supported:

  • 4 finger swipe down: Present Windows
  • 4 finger swipe up: Desktop Grid
  • 4 finger swipe left/right: desktop next/previous

Screen edge support for touch

For touch screen gestures we used a different area in KWin which already provides a fairly similar functionality: Screen edge activation for mouse events.

It’s similar in the way that it activates an action when an input event happens at a screen edge. The main difference is the direction of the input event: for mouse it’s towards the edge, for touch it is from the edge.

The ScreenEdges gained a GestureRecognizer (you see there are two different, independently acting GestureRecognizers) and every Edge defines a Gesture. The Gesture is only passed to the GestureRecognizer if the Edge is reserved for a touch action. Each Edge can have a different action configured and of course you can configure different (or same) action for touch and mouse on the same Edge. When a Gesture for an Edge gets started we provide the same visual feedback as for the edges started through mouse events.

For ScreenEdges there is also a dedicated InputEventFilter which now gained support for touch events and feeds the touch events into the GestureRecognizer.

But there is more to it. This feature got backported to X11. Our X11-standalone platform plugin gained support for XInput 2.2 and touch events. KWin now listens for touch events on X11 and passes these events into the GestureRecognizer of the ScreenEdges. Thus the feature which we implemented on Wayland for Wayland is also available on X11 and directly usable for all users – be it on X11 or on Wayland.

Touchpad gestures are unfortunately only available on Wayland.