A task manager for the Plasma Wayland session

Plasma 5.7 will ship with a new taskmanager library. One of the reasons to implement a new taskmanager library is the port to Wayland. Our old code base was heavily relying on X11 and didn’t support the concept of multiple windowing systems. You can read more on that in Eike’s blog post about the new task manager. In this blog post I want to focus a little bit on the Wayland side of a task manager.

Architectural difference to X11

On X11 every application is able to read every aspect of every other window. This allows to implement a task manager. The task manager can get notified when a window is added, when a window is removed and install event filters for the changes of the window properties. That’s the basics for a task manager.

On Wayland the world is different. An application is not able to see the windows of other applications. This means by default Plasma is not able to see what windows exists, the taskmanager would be empty. The only application which knows about the existing windows is the Wayland compositor.

Merging Desktop Shell and Wayland compositor?

A possible solution to this problem could be to merge the desktop shell and the compositor into one process. This is the architecture used in e.g. Unity and GNOME Shell on X11.

But such an architecture has also disadvantages. It violates the concept of separating concerns and of “doing one thing and do it right”. We have an excellent task manager, an excellent window manager and an excellent desktop shell.

Also it would make code sharing between platforms more difficult. A task manager for Wayland would be very different from a task manager on X11. All the business logic and presentation should be the same. No matter which windowing system is used.

By merging the code bases together we would also make the code less reusable. One couldn’t use KWin/Wayland in another desktop environment (e.g. LXQt) nor could one use Plasma/Wayland with another compositor (e.g. sway). While we of course recommend to use KWin as the Wayland (and X11) compositor we don’t enforce it and also don’t want to enforce it.

Protocols, protocols

So a different solution is needed and that is done through a custom Wayland protocol which is called org_kde_plasma_windowmanagement. We have a few interfaces prefixed with Plasma. This does not mean that it can only be used by Plasma. It just means that the interface was first designed and developed for Plasma’s needs. We are happy to share the interface with other desktop environments.

The protocol is provided by KWin and it announces whenever a new window gets created. Once a new window gets created a bound client can bind an org_kde_plasma_window interface and get notified about all the states of the window.

The protocol exposes pretty much all the information KWin has about the window and it gets updated automatically whenever the state in KWin changes. In addition the protocol has requests the task manager needs, like “close the window”, “minimize it”, “maximize it”, etc. etc. KWin also listens to these requests and honors them.

Although the protocol is added to Wayland, it is windowing system agnostic. The created Plasma Window does not expose the actual windowing system resource (after all on Wayland another client cannot get access to it). KWin exposes a Plasma Window for both X11 managed windows and Wayland windows. By that the task manager is able to manage tasks from multiple windowing systems without knowing that it does so.

Evolution of the protocol

We added the Plasma window protocol initially for the work on the Plasma phone about a year ago. Also on the Plasma phone we had the use case of switching windows and using the X11 based task manager was just no option on a Wayland only device 😉

While the protocol worked fine for our initial needs, it was not yet sufficient for the use in the desktop task manager. So over the last weeks we introduced further support and improved the API to make it work more reliable. Also quite some changes went into KWin to ensure that only “interesting” windows are exposed. E.g. a context menu should not be shown in the task manager.

Our KWayland library of course has support for this protocol through the classes PlasmaWindowManagement and PlasmaWindow on client side. In addition there is a model exposed as PlasmaWindowModel.

Future work

The work on the interface is not done yet. There are still changes to the API needed and the upcoming KDE Frameworks 5.23 release will include new API calls in PlasmaWindow. As well we have already the first change scheduled for KDE Frameworks 5.24. And we also know that we need to expose a few more data elements in the protocol to get the task manager to the same level as on X11.

There is also an interesting idea floating around to use the – windowing system agnostic – Wayland implementation on both X11 and Wayland. For this KWin (on X11) would need to create a dummy Wayland server for the task manager to connect to. It’s an idea which we might or might not implement.

Security considerations

Currently every Wayland client is able to bind this interface, which means that some of the security improvements of Wayland are not available in Plasma/Wayland. We are fully aware of that and were also fully aware of the consequence when we added the interface. I do have ideas on how to address this and this will be implemented before we will recommend the Plasma/Wayland session for daily usage. The design allows to add security checks in these areas. Unfortunately my priority list did not allow me to implement this for Plasma 5.7, next chance 5.8.

Introducing a virtual framebuffer Wayland server in KWayland

The next KDE framework release will include a small addition in KWayland which will make it much easier to run auto tests for Wayland based applications. KWayland now installs a small virtual framebuffer test server into libexec directory.

This means it’s an application inspired by Xvfb which allows to run a graphical application with a virtual framebuffer. The new tool is the same for Wayland. KWayland makes testing already quite easy, but we noticed that we don’t have a good setup for the case that we want to test an application which uses QtWayland QPA plugin as that is not possible with the test support so far in KWayland.

The idea behind the server is that every test gets it’s own dedicated test Wayland server. The test Wayland server will start the test and terminate itself once the test finished (either successful, failure or crash). All output from the test – including exit code – is forwarded, which makes integration into a ctest based system really easy.

In fact if you use CMake as a build system to define your tests, this becomes super easy:

find_package(KF5Wayland CONFIG)
add_executable(myTest myTest.cpp)
target_link_libraries(myTest Qt5::Gui Qt5::Test)
kwaylandtest(myTest)

This also allows to run the same test on X11 and on Wayland. Assuming the test is meant for build.kde.org which has an Xvfb running during the test phase, one can add the test the normal way (X11) and just add the kwaylandtest in addition. So the test will run on both windowing systems.

Now this is a really new project and the test server is still rather limited. All it does is faking rendering at 60 Hz. Otherwise it’s still limited and does not provide the functionality of e.g. a window manager on X11. In that regard it’s also inspired by Xvfb which doesn’t support that. I plan to extend this as I have need in the tests I’m going to add for it.

Speaking of that: the first test case making use of this new infrastructure is already pushed.

Virtual keyboard support in KWin/Wayland 5.7

Over the last weeks I worked on improved input device support in KWin/Wayland and support for virtual keyboard. KWin 5.7 will integrate the new QtVirtualKeyboard module which is now available under GPLv3. For us this means that we have access to a high quality QML based keyboard. For Qt it means that the virtual keyboard is exposed to more users and thanks to the open source nature it means that we can upstream fixes.

The virtual keyboard is integrated into KWin/Wayland making it part of our platform. To support it we implemented the text input interface in KWayland. If an application supports it, then the virtual keyboard can input text from the compositor side.

Video on youtube:

As you can see in the video the virtual keyboard is only enabled by default if no hardware keyboard is available. If a hardware keyboard gets plugged in the virtual keyboard gets disabled automatically. In addition there is a Status Notifier Item registered which allows to enable/disable the virtual keyboard independently of the hardware keyboard state. So on a notebook with a touch screen it’s possible to get the virtual keyboard even if there is (of course) a hardware keyboard.

Implementing support for the virtual keyboard was an interesting journey which required many unexpected changes and improvements all over the place. First of all wl_text_input interface needed to be implemented as that’s what is used in QtWayland 5.6. At one point I hit a wall and saw that the implementation in QtWayland is incomplete. So I checked how it looks like in QtWayland 5.7 and alas the complete implementation changed, it’s using zwp_text_input_unstable_v2, which is not even in wayland-protocols. I’m not happy about QtWayland using not standardized interfaces as I think that’s very bad for compatibility. Nevertheless I also added support for it in KWayland as otherwise we would not be able to communicate with applications using Qt 5.7. As the QtVirtualKeyboard will only be available with Qt 5.7, we need to support Qt 5.7’s interface as well. In KWayland this is implemented in a transparent way and KWin announces support for both interfaces.

Another area which needed lots of work is the support for input devices in KWin. We need to properly detect whether a real keyboard is available. We needed to add further support for touch events in KWin. So thanks to the integration of virtual keyboard KWin now supports touch much better on Wayland.

The improvements for input devices are also useful in other areas. E.g. today I landed a change to not show the cursor image if no pointer device (mouse or touchpad) is connected.

KWayland joined KDE Frameworks

The current release 5.22 of KDE Frameworks gained a new framework: KWayland. So far KWayland got released together with Plasma. KWayland entered as tier 1/integration and is only available on Linux (and other Linux-like systems).

For us working on the Wayland stack in Plasma and KDE this is a very important step. Now we can use KWayland also in other frameworks. Also with KWayland in frameworks we expose it to a larger audience. We hope that it is a useful framework for anyone using Wayland with Qt. It’s not a replacement for QtWayland, rather an addition and way more flexible by being closer to the Wayland API.

This change is also important for distributions. Please note that this is only a logical move in the project structure. KWayland 5.22 is fully ABI compatible to the last release 5.6. You can just exchange the KWayland from Plasma with the one in frameworks. Plasma 5.6 of course works perfectly fine with KWayland 5.22.

Improved input device support in KWin/Wayland

One of the areas I’m currently working on is improving our support for input devices in KWin. While general usage is already quite good in Plasma 5.6, we are not yet able to configure the input devices. This is something I’m working on to improve for Plasma 5.7.

Input devices are provided by libinput and KWin integrates with that for quite some time. What it didn’t do yet is to configure them and keep track of them.

So as a first step we created a Device wrapper class which encapsulates a libinput device in a way that we can easily use it from inside KWin. The Device allows us to gain knowledge about what it supports and also to configure it.

The used Devices and their states can be queried through KWin’s new Debug Console:
Input Devices in Debug console

The Debug Console is also new in Plasma 5.7 and allows us to have a deeper introspection into KWin. It can be opened through KRunner (Alt+Space) and then with the command “KWin”. The functionality set is inspired by tools like xprop (one tab allows to read all window properties), xwininfo -tree (one tab shows a (sub)surface tree), xev (input events).

Back to input devices. Once we had the Device wrapper class in place we could start making use of it to configure devices. And today I landed a series of commits in KWin master which bring us back many features from X11. From the configuration point of view it’s really easy for us as we can just reuse our existing configuration tools. For example the mouse configuration module writes to a config file which KWin can read. When the settings change the configuration module sends out a DBus signal which KWin uses to trigger a reconfigure.

So KWin now supports setting pointer devices (mouse) to left-handed usage and modifying the pointer acceleration.
Toggle Touchpad

In addition KWin started to integrate changes for touchpads. KWin listens to the global shortcut for touchpad toggle and uses this to enable/disable the touchpad. Of course this comes with triggering the new on-screen-display notification for enabling the touchpad. Next step will be to integrate more options for touchpads, but this will require writing a kwin-backend for our touchpad control module.

Running Plasma-Wayland on KDE Neon

This morning I installed KDE Neon on my good old ExoPC as I need to test some touch functionality on Wayland. So here are the steps on how to switch Neon to use Wayland:

Install package plasma-workspace-wayland:

sudo apt install plasma-workspace-wayland

Log out and select in session drop down Plasma (Wayland), log in and enjoy your Wayland session.

If you want to verify that it truly is a Wayland session, just open any KDE application, open the “Help” Menu, go to “About <Application Name>”, click “Version” tab, it should say “The Wayland windowing system” is being used.

Starting KWin/Wayland on another virtual terminal

So far when one started KWin/Wayland on a virtual terminal it took over this virtual terminal. This made it difficult to read the debug output and even more difficult to run the complete session through gdb.

The reason for this behavior is that KWin interacts with logind and needs to take session control on the current logind session. This is needed to have logind open the restricted device files like /dev/dri/card0 or the /dev/input/event* files.

Latest KWin master changes the interaction with logind to have it based on XDG_SESSION_ID. In addition changing virtual terminals is now based on logind instead of custom code for switching through ioctl calls.

A side-effect of these changes is that one can start KWin for another logind session. E.g. it’s now possible to create a logind session on virtual terminal 1 and start KWin on virtual terminal 2 for the logind session on virtual terminal 1. KWin automatically switches the virtual terminal when starting up, so one gets the graphical output. This makes it much easier to get to the debug output.

But this also makes it possible to run KWin through gdb by connecting through ssh. Log in to your system through ssh and start kwin_wayland through gdb with XDG_SESSION_ID pointing to the session on your virtual terminal 1.

March Wayland update: ASCII Art rendering

Another busy month, another round of great features being worked on. March featured our Plasma 5.6 release which had lots of last minute (aka bug fixing) focus. We had our sprint at CERN. But of course also new code was added.

One of the features on my TODO list for Plasma 5.7 is the ASCII art output rendering. Mplayer supports that and while that is nice, it’s not really a good solution. It would be way more convenient to have the complete workspace rendered through ASCII art to not have to implement this feature in every application.

So without much further ado I’m happy to announce the caca platform for KWin:

Glxgears running in nested KWin on caca platform
Glxgears running in nested KWin on caca platform

Unfortunately it’s only supported by our QPainter based compositor. Thus to run it, you need to make sure to use KWIN_COMPOSE=Q to pick the QPainter backend (yeah we should improve that to have a better automatic selection). This is the first platform which support both nested and full mode. Thus you can run with –ascii in your normal X session which will open a window, just like our nested X11 platform plugin. The rendering is done through the great caca library.

Lock screen in KWin's caca platform
Lock screen in KWin’s caca platform

What I’m personally impressed about is how small this new platform plugin is. Less than 400 lines of code, great job by libcaca. But it also shows how easy it is to get KWin on a new platform.

Wayland accelerated at CERN

Last week I attended the KDE sprint at CERN to discuss the next steps in Wayland development with my fellow Plasma developers and our awesome crew of the visual design group.

With the feature freeze just behind us it’s a good time to look at where we stand and what still needs to be done. For me it was a quite exciting sprint as it meant one week of working on Wayland. My notebook is my Wayland test system, so also during the sprint I worked on Wayland. The only exceptions to be on X11 was to investigate problems in Wayland which is still easier from a nested setup on X11 (mostly due to the fact that my notebook has only Qt 5.5 which doesn’t support pressing Ctrl+C in a Konsole). But I was not the only one testing Wayland: my fellow developers did as well and reported interesting bugs I never hit (if you hit a Q_UNREACHABLE something is seriously wrong) due to running a Qt release build. At some point we had the majority of attending Plasma devs running a Plasma Wayland session.

While using a full session one realizes what’s missing in the workflow. As a result we have KRunner working on Wayland now (including a nice improvement for X11) and Yakuake working (with the Ctrl+C exception mentioned above). Some devs also need Ctrl+Alt+Backspace to kill the windowing system in their workflow. So that’s now also implemented – don’t be afraid: it follows the X-server settings and is by default disabled in the keyboard layout.

One problem we noticed is that our possibilities to debug a running Wayland session is rather limited. Especially we are lacking tools like xprop and xwininfo. A window is misplaced, why is that? Currently answering that is still rather tricky. So for the next event cycle I want to improve here to make it very easy for debugging a running KWin instance without needing external tools like xprop and xwininfo. As a start I hacked together a small debug console which can be invoked through DBus:
KWin Debug Console

Of course we did not just hack, but also plan the future. Most importantly we had a session together with the visual design group to explain what new possibilities are there on Wayland. The VDG will hopefully soon come up with some new ideas on how to improve the workflows :-) Also we had a good discussion about dynamic window decorations – Ken is going to report about that, so I won’t steal the show on that.

KWin was not just a topic for the Plasma crowd. Also WikiToLearn has a very interesting setup with KWin in the cloud. This is really exiting and looking promising. It might be that some of my crazy ideas might finally get implemented as I now see use cases for KWin streaming textures into a HTML5 web socket.

And of course we saw KWin in the CMS control room :-)
CMS Control Room

Thanks to CERN for hosting us and making it possible to visit the LHC. Thanks to KDE e.V. for sponsoring the sprint. WikiToLearn and especially Riccardo for organizing the sprint.

If you want to support our sprints in future consider becoming a supportive member.

February KWin/Wayland update: all about input

I haven’t blogged for quite some time about the progress on KWin/Wayland and had a few people requesting an update. As we are now approaching a feature freeze and I have most of the things I wanted to do for Plasma 5.6 done, it’s time to blog again. I use this also as a public service announcement: thanks to Let’s Encrypt my blog is also available through an encrypted connection.

Last month my development focus was on the input handling in KWin. That is the part between input events enter through libinput and are sent to the Wayland client. There are many things the compositor needs to consider for input events: updating the window which has focus, ensuring while the screen is locked to not pass events to normal windows, handling focus follows mouse, etc. etc. On X11 KWin has already code for most of these things, but the code is quite dependent on X11, so it needed to be partially adjusted and partially rewritten.

The code we had in KWin/Wayland for input handling so far already showed it’s age. It was written and designed before KWin really became a Wayland compositor, from the time when KWin could render X11 windows to another Wayland compositor. So it mostly cared about sending the events to the X server. Everything else continued to work as the X11 event handling was still in place.

So the first task was to untangle the code so that it’s easier to extend and at the same time guarantee that it won’t break. As we are now able to start KWin/Wayland on a virtual framebuffer, we can run it during our auto tests. This was a rather important corner stone for reworking the input as it allowed to write test cases for everything KWin does.

With that done existing features from X11 could be ported to Wayland including mouse actions (what to do when clicking inactive window), unrestricted move/resize with alt+(left/right) mouse button, focus follows mouse and auto raise, etc. All those features are now also under test coverage and as the code is mostly shared with the X11 implementation we now also have test coverage for these features on X11. That’s quite an improvement for our X11 implementation thanks to Wayland.

Another area of work is keyboard layout handling. So far KWin defaulted to use the us layout without any possibility to change. That was a huge drawback for my own usage as I couldn’t even write my name. Like with many other input related areas I’m not really familiar with the technology, so I had to look into it in more detail. I am very pleased with xkbcommon, it was really easy to get this working and hooked up properly in KWin. The result is that KWin/Wayland now fully supports keyboard layout switches and also the integration with Plasma’s keyboard layout configuration module. I was rather pleased to see that the configuration module was hardly X11 dependent and just works on Wayland. With KWin listening to the correct DBus signal it allows to reconfigure layouts. But there is still work in that area. So far I have not added support for compose keys, the systemtray applet for switching layouts is not ported yet, accessibility features are still lacking. If you are interested in these areas some help is appreciated.

In case you tried Plasma 5.5 on Wayland you might have noticed that the cursor was sometimes rather incorrect. Not anymore in Plasma 5.6. The cursor image handling got also redesigned and put under test coverage (unfortunately our CI system doesn’t like them yet, locally they pass). But having KWin handle cursors correctly is unfortunately not sufficient to have proper cursor images. Cursor images are set by the clients and if they set an incorrect cursor image, KWin cannot do anything about it. For example QtWayland doesn’t support animated cursors and doesn’t support custom cursors. With the feature freeze for Plasma 5.6 behind us I’m also looking into these issues now. Overall client bugs make it really hard to test features as you never know whether it’s a bug in your application or in the client (or XWayland). The fact that GTK+ and wayland-demos just crash, because KWin doesn’t support xdg-shell yet, doesn’t make it easier to test new features.

The last input area I looked at and landed just in time for feature freeze is drag’n’drop. The implementation is not yet using the updated protocol from Wayland 1.10 as we had already passed dependency freeze when I started to implement it.

Overall the improved input handling gives us a nice feature set for Plasma 5.6 on Wayland. On a single screen setup it’s quite useable already. Of course there are bugs and those need you. Please give it a try and report all bugs you see.

In the area of input there is also more work to be done. We need support for graphic tablets. If you are interested in it: I’m willing to mentor a GSoC project about it. We have a few GSoC ideas for Wayland, check them out!

So what’s next in Wayland world? Till the release of Plasma 5.6 I want to concentrate on bug fixes to make the experience better. Than there’s xdg-shell quite high on my priority list and making multi-screen work correctly (that’s blocking me to switch my main system, the single screen notebook is mostly used on Wayland nowadays).