Server-Side decorations coming to KWin/Wayland

As a kind of Christmas present to our Wayland users I’m happy to announce that over the last two weeks I worked on adding support for server-side decorations.

The main motivations for working on it was the fact that I want to switch to Wayland as primary driver for my system and the nested KWin running on top of another Wayland server, which I need for development, doesn’t have any decorations. Of course I could have implemented client-side decorations for it. But as my readers might know, I consider client-side decorations as an inferior solution. And KWin of course has support for server-side decorations anyway for X11 and thus it’s less work to go for server-side decoration than to go for client-side.

The second reason is that Qt’s default client-side decorations are comparable ugly and lack important features like a difference between active and inactive windows which makes using a Wayland session really hard.

In this case a possibility could have been to develop a plugin so that KDecoration based themes could be used for client-side decoration. But to get it really useable this would have required a complex protocol to get in on par with what KWin internally has.

So here’s the solution:

Server Side decoration support in KWin/Wayland
Server Side decoration support in KWin/Wayland

A core element is a protocol to negotiate whether a window should have server-, client-side or no decoration which got added to KWayland. KWin got an implementation for that both as server and client. I plan to submit the protocol for inclusion in Wayland next year. I do think that this can be a general solution: KWin won’t stay the only Wayland compositor preferring to not have client-side decorations. If we think about tiling and use cases like phones we see that client-side cannot be the ultimate solution. Thus I think it’s a useful extension. Of course it doesn’t forbid client-side decoration, that’s still possible with the protocol. So GTK+ applications build upon client-side decoration are still able to use it, but of course I highly recommend to use server-side decorations on a system that prefers server-side decorations (the protocol is also able to tell that).

The last part to get this working got implemented in our Qt Platform Theme plugin for Plasma. This plugin will move from frameworksintegration to Plasma with 5.6, so we can easily extend it and depend on KWayland. The plugin checks whether the server supports the protocol and if it does it disables Qt’s client side decorations. For each new created Wayland window it tells KWin to either use server-side decoration or no decoration (popup windows). As all of that is implemented in our platform theme plugin it means that it doesn’t affect other Wayland compositors. There the plugin does not get loaded and Qt’s client-side decorations will be used. So no fear: this won’t affect GNOME Shell at all. As the plugin is currently in the process of being moved, it’s only in a scratch repository and won’t make it to main this year. Our code deserves a Christmas break as well :-)

Happy holidays and a successful Wayland year 2016!

Gaming on Linux: Move to next generation?

In this blog post I want to outline some thoughts on how the gaming experience can be improved on Linux.

Situation on X11

On X11 the big problem for gaming is the Compositor. Games need access to the GPU and need to pretty much exclusively use the GPU. Compare to a true console like a PlayStation: while the game is running you can be sure it’s the exclusive user of the GPU. The way how compositing works on X11 this cannot be provided. There is the compositor which needs to render the scene. The setup looks simplified like:

  1. Game renders through OpenGL/GLX
  2. X-Server notifies Compositor through XDamage extension
  3. Compositor schedules a repaint for changed area
  4. Compositor uses the XComposite extension to get a pixmap for the Game window
  5. Compositor binds pixmap to an OpenGL texture
  6. Compositor renders the texture using OpenGL/GLX to the composite overlay window
  7. X server presents the rendered image from the Compositor through kernel mode settings

In this setup we have a few things which are not optimal for a fullscreen game: the roundtrips through Xserver-compositor and the possible delay added by the compositor. In such a setup the compositor will always
slow down the game as it performs vsync, etc.

Workarounds on X11

There is an existing solution to improve this which is known as “unredirection of full-screen windows”. The idea is that if there is a full screen window the compositor won’t run for it and the “normal” non-composited functionality of the X server is used.

In my opinion this is the wrong solution to the problem, because the Compositor is still running, it still gets damage events (from possibly other windows) and might start to composite the scene at any time again (e.g. a notification pops up as a override-redirect window).

In KWin/Plasma we have a better solution: blocking compositing. We can do that as we do not require compositing, other environments which require compositing can only use unredirection as best measurement. In fact we even have high-level API for games to tell us that they want to block compositing. It’s just one X atom, please use it.

This also explains why in KWin/Plasma the option to fullscreen unredirection is not enabled by default. It has drawbacks for a non-gaming usage, while we offer a better solution for games. It also explains why I don’t care at all about gaming benchmarks through the PTS as that in my opinion tests an invalid setup for us. If we cared about it, it would be easy to just ensure that the games used by PTS disable compositing.

Situation on Wayland

On Wayland the setup is better as we don’t have X11 in between. So a setup looks like the following:

  1. Game renders through OpenGL/EGL
  2. Compositor gets notified through wl_surface damage
  3. Compositor directly presents the wl_buffer through KMS as it knows there is nothing else to see

So the situation is much improved in an ideal case. I need to point out that KWin doesn’t support this ideal case yet and still renders through OpenGL, but that’s where we are going to.

But I think there are still problems. Our Compositor still gets damage events from other windows, it gets woken up, etc. Running a game in a desktop environment means that there are other session processes which the game needs to share resources with. We want to have a PlayStation like setup: game everything, everyone else nothing. I don’t want KWin to take away important CPU/GPU time from the game.

Kernel Mode Settings in games

So what can we do? I thought about this and propose that we change gaming completely on Linux: remove the windowing system! Games should talk to kernel mode settings directly, games should interact with libinput directly. Let’s remove everything in between, we don’t need it, it only can worsen the gaming experience.

When a fullscreen game starts, it can create a “sub-session” on a new virtual terminal and become the logind- session controller for that session. This would allow the game to open the device files for rendering and for input just like a Wayland compositor does. Rendering could be done through EGL on top of DRM/GBM just like a Wayland compositor. The game would have full control over rendering, there is no desktop environment to slow it down any more. And it would have control over mode setting. Need a different resolution? No problem, just set it. On a desktop environment that’s always problematic (terrible on X11, better on Wayland). For games in windowed mode nothing should be changed, those should stay on the desktop environment.

Of course this would remove all interaction with the desktop environment. This is something which needs to be considered, like how to get Mumble to work in such a setup? Maybe the game could launch its own Wayland server?

That breaks Alt+Tab! Well not really. For one on X11 at least games often grab the keyboard, so Alt+Tab won’t work anyway. And of course one can still switch with ctrl+alt+f1 to the running session. Games should also have a common way to achieve this in my opinion.

There are certainly a few things which still need to be solved to get there, like how to start such a sub-session, how to return to the running session without needing to unlock the screen. The experience should be as good as on dedicated gaming console. So it’s still some work, but I think this is work well invested and better than trying to somehow make games work in Desktop Environments as that just cannot work as good as a dedicated gaming console.

Looking at the security of Plasma/Wayland

Our beta release of Plasma 5.5 includes a Wayland session file which allows to start in a Plasma on Wayland session directly from a login manager supporting Wayland session. This is a good point in time to look at the security of our Plasma on Wayland session.

Situation on X11

X11 is not secure and has severe conceptual issues like

  • any client connected to the X server (either remote or local) can read all input events
  • any client can get information about when another window rendered and get the content of the window
  • any client can change any X attribute of any other window
  • any window can position itself
  • many more issues

This can be used to create very interesting attacks. It’s one of the reasons why I for example think it’s a very bad idea to start the file manager as root on the same X server. I’m quite certain that if I wanted to I could exploit this relatively easily just through what X provides.

The insecurity of X11 also influenced the security design of applications running on X11. It’s pointless to think about preventing potential attacks if you could get the same by just using core X11 functionality. For example KWin’s scripting functionality allows to interact with the X11 windows. In general one could say that’s dangerous as it allows untrusted code to change aspects of the managed windows, but it’s nothing you could not get with plain X11.

Improvements on Wayland

Wayland removes the threats from the X11 world. The protocols are designed in a secure way. A client cannot in any way interact with windows from other clients. This implies:

  • reading of key events for other windows is not possible
  • getting window content of other windows is not possible

In addition the protocols do not allow to e.g. position windows themselves, or raise themselves, change stacking order and so on.

Security removed in Plasma

But lots of these security restrictions do not work for us in Plasma. Our desktop shell need to be able to get information of other windows (e.g. Tasks applet), be able to mark a panel as a panel (above other windows) and need to be able to position windows itself.

Given that we removed some of the security aspects again and introduced a few custom protocols to provide window management facilities and also to allow Plasma windows to position themselves. At the moment we have no security restrictions on that yet which gives this functionality to all clients.

We will address this in a future release. There are multiple ways how this could be addressed, e.g. using the Wayland security module library or use systemd in some way to restrict access. Overall I think it will require rethinking security on a Linux user session in general, more on that later on.

Security added in Plasma compared to X11

The most important change on the security front of the desktop session is a secure screen locker. With Plasma 5.5 on Wayland we are able to provide that and address some long standing issues from X11. The screen locks even if a context menu is open or anything else grabbing input. The compositor knows that the screen is locked and knows which window is the screen locker. This is a huge change compared to X11: the XServer has no concept of a screen locker. Our compositor can now do the right thing when the screen is locked:

  • don’t render other windows
  • ensure input events are only handled in the lock screen
  • prevent access to screen grabbing functionality while screen is locked

As a matter of fact the Wayland protocol itself doesn’t know anything about screen locking either. This is now something we added directly to KWin and doesn’t need any additional custom Wayland interfaces.

How to break the security?

Now imagine you want to write a key logger in a Plasma/Wayland world. How would you do it? I asked myself this question recently, thought about it, found a possible solution and had a key logger in less than 10 minutes: ouch.

Of course there is no way to get a client to act as a key logger. The Wayland protocol is designed in a secure way and also our protocol additions do not weaken that. So the key to get a key logger is to attack KWin.

So what can an attacker do with KWin if he owns it? Well pretty much anything. KWin internally has a very straight forward trust model: everything is trusted and everything can access anything. There is not much to do about that, this is kind of how binaries work.

For example as a Qt application each loaded plugin has access to the QCoreApplication::instance. From there one could just use Qt’s meta object inspection to e.g. get to the InputRedirection model and connect to the internal signal on each key press:

<code>void ExamplePlugin::installKeyLogger()
{
    const auto children = QCoreApplication::instance()-&gt;children();
    for (auto it = children.begin(); it != children.end(); ++it) {
        const QMetaObject *meta = (*it)-&gt;metaObject();
        if (qstrcmp(meta-&gt;className(), "KWin::InputRedirection") != 0) {
            continue;
        }
        connect(*it, SIGNAL(keyStateChanged(quint32,InputRedirection::KeyboardKeyState)), this, SLOT(keyPressed(quint32)), Qt::DirectConnection);
    }
}

void ExamplePlugin::keyPressed(quint32 key)
{
    qDebug() &lt;&lt; "!!!! Key: " &lt;&lt; key;
}
</code>

But Martin, why don’t you just remove the signal, why should any other aspect of KWin see the key events? Because this is just the example of the most trivial exploit. Of course it’s not the only one. If you have enough time and money you could write more sophisticated ones. For example look at this scenario:

KWin uses logind to open restricted files like the input event files or the DRM node. For this KWin registers as the session controller in logind. Now a binary plugin could just send a DBus call to logind to also open the input event files and read all events. Or open the DRM node and take over rendering from KWin. There is nothing logind could do about it: how should it be able to distinguish a valid from an invalid request coming from KWin?

How to secure again?

As we can see the threat is in loading plugins. So all we need to do is ensure that KWin doesn’t load any plugins from not trusted locations (that is not from any user owned locations). This is easy enough for QML plugins where we have the complete control. In fact it’s easy to ensure for any of KWin’s own plugins. We can restrict the location of all of them.

And even more: by default a system is setup in a way that no binary plugins are loaded from user’s home. So yeah, no problem after all? Well, unfortunately not. During session startup various scripts are sourced which can override the environment variables to influence the loading of plugins. And this allows to also use the well known LD_PRELOAD hack. My naive approach to circumvent this issue didn’t work out at all as I had to learn that already the session startup and the PAM interaction source scripts. So your session might be owned very early.

An approach to black list (unset) env variables is futile. There are too many libraries KWin relies on which in turn load plugins through custom env variables. Most obvious examples are Qt and Mesa. But there are probably many more. If we forget to unset one variable the protection is broken.

A different approach would be to white list some known secure env variables to be passed to KWin. But this also requires that at the point where we want to do the restriction the session is not already completely broken. This in turn means that neither PAM nor the session manager may load any variables into the session before starting the session startup. And that’s unfortunately outside what we can do in our session startup.

So for Plasma 5.5 I think there is nothing we can do to get this secure, which is fine given that the Wayland session is still under development. For Plasma 5.6 we need to rethink the approach completely and that might involve changing the startup overall. We need to have a secure and controlled startup process. Only once KWin is started we can think about sourcing env variables from user locations.

So how big is the threat? By default it’s of course secure. Only if there is already a malicious program running in the system there is a chance of installing a key logger in this way. If one is able to exploit e.g. a browser in a way that it can store an env variable script in one of the locations, you are owned. Or if someone is able to get physical access to your unencrypted hard disk, there is a threat. There are easy workarounds for a user: make all locations from where scripts are sourced during session startup non-writable and non-executable, best change ownership to root and encrypt your home location.

Overall it means that Plasma 5.5 on Wayland is not yet able to provide the security I would have liked to have, but it’s still a huge improvement over X11. And I’m quite certain that we will be able to solve this.

October Plasma on Wayland Update: all about geometry

Last month our Wayland efforts made a huge step forward. In KWin we are now at a state where I think the big underlying work is finished, we entered the finishing line of the KWin Wayland porting. The whole system though still needs a little bit more work.

The big remaining task which I worked on last month was geometry handling. That is simplified: moving and resizing windows. Sounds relatively easy, but isn’t. Moving and resizing windows or in general the geometry handling is one of the core aspects of a window manager. It’s where our expertise is, the code which makes KWin such a good window manager. Naturally we don’t want to throw that code out and want to reuse it in a Wayland world.

And that meant: refactoring. The big problem here was that the code in question is also highly X11 specific. Examples are: the sync protocol for synced resizing, window gravity handling, various X11 specific quirks for moving X11 window moving smooth (e.g. only update at end of move if compositing is active, direct updates if not active).

The task was now to separate the X11 specific geometry handling from generic geometry handling so that it can support both Wayland and X11 windows. With this now in place all the geometry handling works also for Wayland windows. This includes move/resize through the alt+f3 menu, but also triggered from the window itself. During move/resize the quick tiling areas are of course triggered correctly, show the outline and snap to the area if released. Resizing is automatically synced to the speed the client supports and that also in a much better way than on X11 (yay for Wayland and double-buffered state!). During move/resize the windows snap to screen borders and other windows.

Unfortunately our checks to ensure that the window titlebar is not moved outside the visible area doesn’t work, because of client side decorations. Yay, everything is awesome! Have I mentioned that client side decorations are a stupid idea because it breaks useful stuff? No, cannot be. Related to that: QtWayland enters move mode if you click the title bar to activate the window. Of course there should be a delay which KWin implements for it’s decorations. After all we have more than 15 years of experience in doing window decorations. Yay, client side decos! Given control to the application! Let’s leave them in charge of window management, what could go wrong? Wohoo! Totally awesome! Broken window management! That’s the way to go! Of course that could be fixed in Qt, but well it doesn’t fix the problem. We have seen this in Chromium years ago, we even had to adjust our quick tiling/maximization behavior because of it. Another fun fact: QtWayland deco has a minimize button which does nothing because it’s not supported in wl_shell protocol. Have I mentioned that client side decos are awesome?

All right, all right, I’ll end the sarcasm mode now. Our geometry handling has a few more very handy features like window packing: you can configure a shortcut to move the window left (or other direction) and it will move to the next window or the screen border. Similar we support growing or shrinking the window to the next window. I was hardly aware of that feature before doing the refactoring, so I thought I should point it out. It can be used from scripting, so should be useful for ideas like poor man tiling or resizing multiple quick tiled windows at the same time. As a nice addition this code is now covered by auto tests.

So with geometry handling in place it’s possible to do real testing and one of my systems (my notebook) migrated from X11 to Wayland. Actually I have been using Wayland on that system already for watching videos since April as it gives tear-free rendering. A clear plus when watching videos.

And this gives us already the outlook for what we will see in the November update: I’ll focus on stabilizing the current state of Plasma/Wayland and fix bugs, fix bugs and fix bugs. My aim is to have a useable early-adopters version ready for the Plasma 5.5 release. It’s looking good, so I’m confident that we will reach that state.

Upgrading libhybris

One of the most important dependencies for our phone project is libhybris. Libhybris is a neat technology to allow interfacing with Android drivers allowing for example to bring Wayland to a device where all we have are Android drivers.

Given that KWin provides a hwcomposer backend which uses libhybris to create an OpenGL context. All other applications need libhybris indirectly to have the Wayland OpenGL buffer exchange work automatically.

When we started the work on the hwcomposer backend we based it on the libhybris version used in Ubuntu (0.1.0+git20131207) as we used Ubuntu as the reference platform. Soon enough we noticed that this version diverged a lot from the upstream version. Lots of recent changes are missing and there are API incompatible changes.

This made working with it difficult. How much time should we invest in investigating issues? Should we write code which we know might break once Ubuntu decides to upgrade libhybris? How well is Wayland integrated in the Ubuntu version given that they don’t need it? If we need help, who to talk to? Ubuntu who will tell us that they don’t know anything about Wayland, or the libhybris devs who might just tell us: use later version?

Furthermore we want other distributions to provide Plasma for the phone. This means they need to provide libhybris. Of course this is difficult if we need to tell them that we need exactly the version used by Ubuntu. And even more it might conflict with other uses. Considering distributions like Mer would have to chose between a libhybris for Plasma and a libhybris for lipstick.

With that in mind we wanted to invest some time on upgrading libhybris in our stack in this release cycle and then fix the issues we were seeing in the stack. Our awesome packagers did the job of creating packages so that I can port KWin against it. And in deed after some hacking I had KWin rendering again. A more difficult task was to get other applications to work as we run into a problem that libwayland-egl does not use the alternatives system. Thus our packagers needed to do some ld tricks to get this worked around. But with that we had a nice rendering system.

A surprise in this exercise was that our input handling code in the hwcomposer backend didn’t compile any more. The code was gone. While that was in the first moment an unpleasant surprise, it soon turned into something wonderful. If that code is not needed at all on an Android powered device it means that we must be able to get libinput to work with it. 400 lines of code deleted and it’s using the shared input stack through libinput. I’m very happy about that!

With that all in place we finally were able to investigate the rendering issues we were seeing. My hope was that just upgrading libhybris would fix the visual tearing, but unfortunately not. While I’m still surprised that it’s possible to get tearing in the first place on Android devices (hey ever heard of things like Atomic Mode Settings, Android?), it at least gives us a vsync event. Unfortunately the only tear-free solution I could find invokes blocking till we get the event. I don’t like that and I think that’s a bad architecture. One can have blocking free and tear free rendering. Our DRM (kernel mode setting) backend can do so with an easy to use API. Really disappointing that the Android stack is in that regard not better than the glx backend. But well at least it’s tear free :-)

As we now use upstream libhybris I hope to see distributions to pick up the work and provide a Plasma phone spin. I’d love to see an openSUSE phone or a Fedora phone (or any other distribution). Distributions: you can of course ask us on how to integrate :-)

September update for Plasma’s Wayland porting

September was a busy months in the KDE Wayland world. We have worked hard to bring Plasma closer to a workable system and could cross off some very important milestones.

Transient window positioning

One of the biggest oddities when trying out Plasma on Wayland in the 5.4 release is the fact that menus open at random positions. The reason for this is that KWin applied it’s placing strategy on it and ignored the hints provided by the window. We have now implemented support for the so-called transient windows in KWayland and use the provided placement hint in KWayland. So now all menus open at the correct position. A useable Wayland session is much closer now.

Plasma/KWin specific extensions

Marco did quite some work for the integration of Plasma. KWin provides some Plasma specific extensions like the sliding popups effect, blur and background contrast effect, etc. We have an abstraction through KWindowSystem so that applications do not have to use low level X11 calls directly. Now we extended this abstraction to also Wayland: if the application uses the API it will work on both X11 and Wayland. Granted it will only work with compositors providing the specific protocols, but that is no difference to X11. Also there the compositor needs to implement the custom protocol.

On the client side the protocol is implemented in KWayland and the integration for KWindowSystem is provided through
a plugin provided by the kwayland-integration repository. On KWin side the protocol is also provided in KWayland allowing a very easy to use API. Of course also KWin needed small adjustments in the effects to announce support for the protocol and read the information provided by the windows. Thanks to nice wrapping in KWayland the code is cleaner and simpler than in the X11 case.

Support for multiple X Servers

A change not directly relevant for KWin went into KWindowSystem and will be released with the upcoming 5.15 KDE Frameworks release. KWindowSystem provides an X11 API which looks like it supports multiple X Servers (e.g. one application talking to multiple servers), but that has never worked as it fetched required atoms only on first connection. We refactored the relevant code to no longer have this limitation.

Current Plasma in a nestedKWin/Wayland session with grabbed input
Current Plasma in a nested KWin/Wayland session with grabbed input

Granted normally applications do not talk to multiple X servers, but there is a mode in KWin which uses just that: a nested kwin_wayland on X11. To explain: it needs to talk to the host X server for rendering two (one server) and it starts it’s own Xwayland server (second server). In case you have ever wondered why the nested kwin_wayland window released with Plasma 5.4 neither has a window icon nor a window title: that’s the reason. We couldn’t use Qt’s abstraction (wrong QPA plugin) and also not KWindowSystem as we needed to make sure the atoms get resolved for the Xwayland server. Now with this restriction removed the window has an icon and a title. Even more I added a “grab input mode” as one might know from virtual machines. While it’s easy to implement I didn’t want to implement it without having a way to tell the user what happened and what the current grab state is.

Preparing KWin for the cloud

The most exiting new feature in my opinion was born last Friday based on frustrations about testing KWin Wayland. Last week Marco and I spent quite some time investigating a few regressions (as it turned out due to adding transient window support). The way to test it was just uncomfortable. We had a test case but it meant starting KWin, waiting till it’s started, start the test application, perform some clicks and interpret whether it worked. Once we fixed that issue I started to look at a crash and the process was similar annoying. What I wanted was a way to automate the test condition, that is an autotest which we could even run in our CI system.

So on Friday I decided to dedicate my development time on a virtual framebuffer backend.This backend (to start use kwin_wayland --xwayland --virtual) doesn’t render to any device, but only “simulates” rendering by using a QImage which then isn’t used at all. Well not completely true: there is an environment variable to force the backend to store each rendered frame into a temporary directory.

Why is such a virtual backend so exiting? Well it means we can run KWin anywhere. We are not bound to any hardware restrictions like screen attached or screen resolution. With other words we can run it on servers – in the cloud. The first such instance runs on our CI servers in the form of an automated integration test. And in future there will be much more such tests.

But that is not only interesting for KWin to have it’s autotests, it’s also interesting for all other projects of the workspace as we have now a virtual Wayland server which is functional identical to the one we use. We also have a better virtual X server now as we have Xwayland instead of Xvfb (e.g. support for XRandR extension).

Once I had it implemented ideas came to me for improving our CI system: we could use it for something like OpenQA (or integrate the existing tool) and start a complete Plasma session and screenshot various points (if that sounds like an exiting project: please contact me, also if you want to do that as a Season of KDE project ).

Or integrate a remote rendering solution (e.g. VNC, rdesktop, spice, html5) and run a complete session through the web. That could be a very interesting feature for designers, translators, supporters and many other non-developers. Get the latest state of the code directly tested. We have things like Kubuntu CI which make it easy to test in a virtual machine, but wouldn’t it be even more awesome to just run the latest build of the CI system in the browser?

KWin Tests

With the help of this virtual backend we are now able to start a “complete” KWin in the autotests. This allows us to very precisely test whether a specif feature works as expected. Unit tests are great, but sometimes one wants to test the complete integration and that’s now possible.

Working auto completion tooltip in Kate on KWin/Wayland
Working auto completion tooltip in Kate on KWin/Wayland

The first problem addressed with this new possibility was a bug noticed while writing this blog post. I used Kate on Wayland and the tooltips got keyboard focus. So now we have an autotest which ensures this case works.

Comparing DPMS on X11 and Wayland

On the Plasma workspaces Display Power Management Signaling (DPMS) is handled by the power management daemon (powerdevil). After a configurable idle time it signals the X-Server through the X11 DPMS Extension. The X-Server handles the timeout, switching to a different DPMS level and restoring to enabled after an input event all by itself.

The X11 extension is a one-way channel. One can tell the X-Server to go to DPMS, but it neither notifies when it has done so nor when it restored. This means powerdevil doesn’t know when the screens are off. Granted, there is a call to query the current state, but we cannot perform active polling to figure out whether the state changed. That would be rather bad from a power management perspective.

So overall it’s kind of a black box. Our power management doesn’t know when the screens turn off, Neither does any other application including KWin. For KWin it would be rather important to know about it, because it’s the compositor and there is no point in running the compositor to render for something which will never be visible. The same is obviously true for any other application: it will happily continue to render updated states although it won’t be visible. For a power management feature this is rather disappointing.

Now let’s switch to Wayland – in this case kwin_wayland. Last week I started to implement support for DPMS in KWin. Given that KWin is responsible for DRM, we need to add support for it in KWin. Of course we want to leverage the existing code in powerdevil, so we don’t add the actual power management functionality, but rather an easy to use interface to tell KWin to enter a DPMS state on a given output.

Of course we tried to not repeat the mistake from X11 and thus the interface is bi-directional: it notifies when a given state is entered. Although in a Wayland world that’s not so important any more: KWin enables DPMS and knows that DPMS is enabled and can make internal use of this. So whenever the outputs are put into standby the compositor loop is stopped and doesn’t try to repaint any more. If the compositor is stopped windows are not being repainted and aren’t signaled that the last frame got painted. So all applications properly supporting frame callbacks stop rendering automatically, too. This means on Wayland we are able to properly put the rendering of all applications into an off state when the connected outputs are in standby mode. A huge improvement for power management.

Layered compositing

On X11 the (OpenGL) compositor renders into a single buffer through the overlay window. This is needed to get features like translucency, shadows, wobbled windows or a desktop cube as Xorg itself doesn’t have any support for such features. The disadvantage of this approach is that we basically always have to perform a “copy” of what needs to be rendered. Consider VLC is playing a fullscreen video the compositor needs to take VLC’s video pixmap and render it onto the overlay window. The compositor needs to run, evaluate the scene and then render the one window.

Of course the compositors optimize for this case by only repainting what is needed and repainting from top to bottom in the window stack so that only the one fully opaque window gets rendered. Even more they have features like unredirecting fullscreen windows or (in the case of KWin) blocking compositing while specific windows are active. Unredirecting is a nice idea but it doesn’t really cover the use case. If one has just a small window (a tool tip, menu, on-screen volume change display) on top of the VLC video window, we need to start compositing again. Although we still would like to just forward the one VLC window.

On Wayland the situation looks better. Instead of rendering to an overlay window the compositor can directly interact with the hardware through e.g. DRM. This allows to for example take the buffer provided by VLC and pass it directly to DRM and bypass the compositing all together. But modern hardware allows us to do more than Xorg ever allowed us through “layers”. Be it the raspberry pi, Android’s hwcomposer or Linux’s KMS Planes, the idea is always similar: let the hardware compose the final image together.

If we think of Plasma phone we realize that in most cases we just need three layers: top panel, bottom panel and a maximized window in between. There is no need to do complex composition of the scene at all as all that is needed is putting the three visible (and opaque) windows in the three layers.

On a desktop system it’s a little bit more complicated. Our windows are not all maximized, they have translucent window decorations, shadows, there is a panel with blurred background. Especially the latter one needs a composition through OpenGL. Nevertheless there should be situations where we can make use of layers. Especially in the case of a fullscreen or maximized window this should help making the compositor faster.

Using layers will mean a large change on how KWin’s compositor works. The infrastructure for unredirecting of fullscreen windows cannot be reused in a sensible way as it is evaluated only for fullscreen windows and before the compositor runs. That is before the effects are executed. The decision whether to put a window into a layer needs to come at a later point: once we have evaluated whether we need to compose the complete screen (you might be wobbling a window) or can short pass through layers.

What I have in mind is to give this architecture a try with the VC4 hardware of the raspberry pi which supports layers in a decent way. And to first adjust the QPainter based compositor (as it’s the simpler one) to make use of this architecture. The idea is to experiment with it to get a useful generic KWin-internal backend API, so that we can also use it for OpenGL compositor and even more for our DRM and hwcomposer backend.

For the DRM backend I want to use the new atomic modesetting feature which will be introduced in Linux 4.2. Given that we cannot depend on it anytime soon, this will still need some time till it will be implemented. Overall all of this is not going to happen tomorrow – there are still more important things to work on. But if you, dear reader, are interested on working on this, please contact me. I’m happy to walk you through the code and share my ideas.

As this blog post is mostly about optimizing the compositor by making use of new hardware features which we didn’t have on X11, I can already point out that I will do a follow up blog post on Vulkan.

Now to something else!

KDE is currently running a fundraiser for the Randa Meetings with the topic “Bring Touch to KDE!”. I personally will not participate in the Randa Meetings, but many other KDE developers will go there and we need your support for it. Please help and make the Randa Meetings possible.

Fundraiser

A Qt Platform Abstraction plugin for KWin

In Qt we have the Platform Abstraction (QPA) which allows to better interact with the used windowing system through a plugin. In case of KWin we use the “xcb” plugin on X11 and on Wayland we used to use the “wayland” plugin provided by QtWayland. For quite some time I had been thinking about migrating away from those and use an own KWin-specific plugin at least for Wayland.

What’s wrong with QtWayland?

QtWayland provides a plugin for Wayland clients, but KWin is not a Wayland client: it is the Wayland server. This created some interesting “problems” which we had to workaround in KWin. KWin just doesn’t fit the usecase of QtWayland QPA plugin and it would not be a good idea to change QtWayland in a way that it supports KWin as a first-class citizen – it’s just a too special use case. So let’s look at some of the issues which motivated me to write an own QPA plugin.

Connecting to the Wayland server

The wayland plugin is meant for Wayland clients and tries to connect to the Wayland server during the creation of the QGuiApplication. As KWin is the Wayland server the plugin wants to connect to, we needed to make sure that the Wayland server and the event loop are up and running before constructing the QGuiApplication. This added a restriction that we have to monitor the server socket from the main thread.

Blocking roundtrips

The wayland plugin performs a blocking roundtrip to the Wayland server during startup to wait for all outputs to be announced. But this happens from the main gui thread, in which also the server lives. You can guess what happens: dead-lock. We fixed this one in QtWayland, but the problem was still there: any blocking call from the main gui thread to the Wayland server would freeze KWin. We experienced this with e.g. eglInitialize, startup of dbus services (e.g. kded, kamd), rendering of Qt internal windows in case the compositor didn’t send the frame rendered yet. Overall I found this aspect way to fragile and consider especially the fact that the server is in the main thread as a problem.

BypassWindowManagerHint

KWin uses the Qt::BypassWindowManagerHint on all it’s windows because this is needed on X11. The hint is X11 specific and doesn’t make much sense on other platforms, so the Wayland plugin doesn’t show such windows at all. That is a valid approach – there is nothing I could say against it. So in KWin we needed to remove the hint for all windows on Wayland.

OpenGL Context for QtQuick

QtQuick requires an OpenGL context and if it cannot create one it will abort. That is normally not a problem but can easily become one in the case of KWin: KWin creates a “low-level” OpenGL context using the specific platform (e.g. DRM/GBM, hwcomposer, whatever). Qt(Wayland) doesn’t know which one KWin uses and might not even have code for it. QtWayland tries to create an OpenGL context using the EGL platform Wayland. In the case of Mesa everything works fine, but we noticed that on libhybris the creation of a Wayland context fails if there is already one created for hwcomposer. One could consider this as a libhybris specific issue, but I wouldn’t say so – interacting with multiple EGL platforms from one process is kind of out of the specification.

Window decorations

This is a small issue: QtWayland creates window decorations around all Qt windows. But in the context of KWin we do not want any decorations. So we had to disable them by setting the QT_WAYLAND_DISABLE_WINDOWDECORATION environment variable.

And here is the plugin

So after Akademy I decided to work on it and see whether we can get a plugin working. Now it’s in a state where it works good enough to blog about it. The plugin does not support everything that might be used by Qt, but rather reduces to the feature set used by KWin. For example we don’t need to forward any input events, because KWin (mostly) handles them internally anyway and sends them directly to the respective internal Qt window.

The plugin brings the advantage that it no longer depends on the Wayland server being started before creating the QApplication. In fact it does not even try to connect to it, but can use KWin’s internal connection. Similar we can use this to shortcut e.g. creation of low level windows, etc.

On the OpenGL side the plugin approach shows it’s true strength. As it can access KWin internals we can use the EGLDisplay used by the compositing scene and create a sharing OpenGL context. So we can bypass the windowing system completely to render QtQuick windows and this will also allow us to make our window textures available to QtQuick. So a nice improvement.

In case our compositor doesn’t use OpenGL the plugin creates an OpenGL context for the platform Wayland. This is done in a way that it doesn’t hit the blocking issue I mentioned above.

So is the plugin something you should use in other applications?

The plugin interacts with KWin internals and thus is not in any way useable for non-KWin applications. In fact it even checks whether the running binary is KWin and does nothing if that’s not the case. It will also never be turned into a generic plugin as there is no need for that (just use QtWayland).

When will this plugin get merged?

The plugin can be found in branch “qpa” on my personal kwin clone on git.kde.org. It’s not yet in a state that I could push it to master. I still need to test more and make sure that everything works as intended.

Now to something else!

KDE is currently running a fundraiser for the Randa Meetings with the topic “Bring Touch to KDE!”. I personally will not participate in the Randa Meetings, but many other KDE developers will go there and we need your support for it. Please help and make the Randa Meetings possible.

Fundraiser

Porting Qt applications to Wayland

During Akademy I hold a session about porting applications to Wayland. I collected some of the general problems I saw in various KDE projects and want to highlight them in this blog post, too.

Starting Wayland

First the basics: how to start a Wayland compositor? Assuming one is using Plasma 5.4 (aka master) the way to start a nested KWin is:

kwin_wayland --xwayland

Important: ensure that you have Xwayland installed!

If you run nested this will open a window, if it turns black, everything works. If it doesn’t turn black: contact me (please check whether you use free drivers first).

As an alternative you can also start Weston:

weston

If you run into issues with your application running under Wayland, please always test whether the problem is also visible on Weston before blaming me 😛

Starting applications on Wayland

Now we have our black KWin window and we want to get a window to it. There are several ways to start it. As an example I’ll use kate.

Environment variable

Maybe the easiest way is to just use an environment variable to switch the QPA platform to Wayland:

export QT_QPA_PLATFORM=wayland
kate

And a Kate window will open on the nested KWin.

The environment variable is set to wayland in our startplasmacompositor script to start a full Plasma on Wayland session on tty.

Command line switch

Every Qt application also has a command line switch to select the windowing system:

kate --platform wayland

Starting applications together with KWin

KWin supports starting applications once the Wayland and Xwayland servers are started. You can add the programs to start to the command which starts KWin:

kwin_wayland --xwayland "kate --platform wayland"

Or to simplify with environment variables:

export QT_QPA_PLATFORM=wayland
kwin_wayland --xwayland kate

Of course it’s also possible to just start a Konsole and then launch applications from inside the Konsole. Please note that any key combination with ctrl-key is unfortunately broken when running Konsole as a Wayland application.

X11 Specific code

Many applications have X11 specific code. Being it implicitly or explicitly. So we need to move away from it and in most cases it means also improving the situation on X11 because the solution is wrong.

Opening X connection

A common mistake I saw in Qt applications is to open a connection to the X server. E.g. like that:

Display *dpy = XOpenDisplay(NULL);

Now this code is already wrong on X11 as it opens another connection which is not the Qt display connection. It might be on the wrong head, it might be on the wrong X server. From an integration point of view it’s broken.

So the solution is to always use QX11Info from QtX11Extras to integrate with the Display:

Display *dpy = QX11Info::display();

Runtime checks in addition to compile time checks

Before Qt5 the way to support multiple windowing systems was to use compile time switches, like:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    XFlush(QX11Info::display());
}
#endif

Now this will crash on Wayland. It will be compiled in, but crashes when running on Wayland. So you need to improve. Let’s try:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    Display *dpy = QX11Info::display();
    if (!dpy) {
        return;
    }
    XFlush(dpy);
}
#endif

Now this code will also crash! The problem is that QX11Info::display() goes through Qt’s native interface and uses the key “display”. QtWayland also provides the key “display” and returns a wl_display for it, which gets casted into a void in the native interface and then to Display* in QX11Info and boom at interesting places.

The proper way is to check the platform name:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    if (!QX11Info::isPlatformX11()) {
        return;
    }
    XFlush(QX11Info::display());
}
#endif

My recommendation is to keep the check in a local variable if the check is needed multiple times as in the end it’s a string comparison each time. Ideally you should restructure your code to have platform specific subclasses or similar refactored solutions.

General Changes with Wayland

Now here it becomes interesting: runtime changes. The most important information for application developers is probably that there are no more global window ids.

No global window ids

This has quite some implications for your application. E.g. QWindow::winId() returns an internal incremented number. Neither the compositor nor any other application knows anything about your internal window id counter. Similarly QWindow::fromWinId is not able to map a window id to a QWindow any more. This of course affects use cases like sending a window id through DBus to another process to use it for setting the transient hint. Also it means that KWindowSystem API has a hard time to support the calls on Wayland and in most cases it will just print a warning (this might improve, though).

Setting window icon

On X11 the applications exported the window icon as pixmap data. This is no longer possible on Wayland. Instead the compositor maps the window icon from the desktop file and the application announces the desktop file for a window. In Qt this is done by using the reverse domain name combined with the binary name. The domain name is set with KAboutData and defaults to “kde.org”. So for example in the case of kate the desktop file should be called “org.kde.kate.desktop”. So please fix the naming of your desktop files. If you want to test please be aware that this requires Qt 5.5 to work properly.

No longer working API

Quite some of the API calls of QWindow cannot be supported by Wayland on default (some of them might work on KWin, but please don’t implement against KWin!).
A short list of those API calls I know that they cannot work:

  • QWindow::setPosition
  • QWindow::setMouseGrabEnabled
  • QWindow::setKeyboardGrabEnabled
  • QWindow::alert
  • QWindow::lower
  • QWindow::raise
  • QWindow::requestActivate

In addition it’s no longer possible to grab the image of the screen using e.g. QScreen::grabWindow (most obvious for the case of “root” window). Also warping the pointer using QCursor::setPos is no longer supported. Please note that warping the pointer is also a bad idea on X11 and you shouldn’t do that.

In case your application set the Qt::BypassWindowManagerHint on your QWindow/QWidget you need to do some porting as QtWayland doesn’t show such windows. Unfortunately that needs a case by case evaluation of the use case and the solution I presented during my Akademy talk should not be applied for applications.

Porting to Wayland

Don’t port

The solution to port to Wayland is: DON’T PORT! Mostly your X11 specific code or behavior is wrong and could be solved in better ways. Instead of porting you might want to rethink your problem or change to existing abstracted API in e.g. Qt or KWindowSystem. I have seen many cases where X11 API was used as a workaround by e.g. raising windows, force activating them and so on. If one things about it one realizes that there seems to be an obvious problem. So don’t try to port that but rethink your problem and improve not just on Wayland, but also on X11.

But I really need platform specific code

Ok, ok, I get it! Now let’s start with the obvious: your code needs to be compile time and runtime checked. Let’s start with how to not do it:

#if HAVE_X11
    if (!QX11Info::isPlatformX11()) {
         return;
    }
    // here goes the X11 specific code
#else
    // here goes the Wayland specific code
#endif

The problem with this code snippet is that X11 support is probably also available when you use Wayland. Also the else part is not just Wayland, but also any other platform.

As I said the proper solution is to make it compile and runtime checked:

#if HAVE_X11
    if (QX11Info::isPlatformX11()) {
        callX11SpecificImplementation();
    }
#endif
#if HAVE_WAYLAND
    if (QGuiApplication::platformName().startsWith(
            QLatin1String("wayland"), Qt::CaseInsensitive)) {
        callWaylandSpecificImplementation();
    }
#endif

Be aware that your code might also run on other platforms on Wayland, e.g. eglfs or other custom windowing systems. So don’t crash if you don’t hit any of the code paths 😉