Tag Archives: KWin

Enhancing user experience through the Compositor

This article is an April Fool. For an explanation please see the update.

Some time ago Ken introduced the conecept of dynamic window decorations (DWD) and during the last Plasma sprint there was already some work on experimenting with an implementation. DWD are extremely promising to enhance the user experience by morphing the application and window manager scope together.

The basic concept of DWD is to get content of the application inside the scope of the window manager. But what works one way works also the other way: we can enhance the user experience by providing additional information relevant for the window inside the window decoration.

An example for this could be a volume applet for music and audio players. But of course it doesn’t stop there, today is all about the user content and integrating the web. For web browsers we can integrate a global “Like It on Facebook” button. And by analyzing the user behavior we can provide even more help. We can notice if you struggle using Krita and provide a link to where to get a better drawing tablet.

If you use digiKam you might be interested in some books about photogrophy. We know which music you listen to on Amarok and show you that the new album of your favorite music group just got released. Not every application integrates stores in a proper way. This is a shame as it does not provide the best possible user experience and KWin as a window manager can help there to add missing store functionality where it is needed.

Of course this also allows to show you better alternatives. If we see that you struggle with wine emulated applications we can provide the Linux alternative. We can show that new updates are available for your currently used application. If you use sub-par free software applications we can show you were to get a proper (slightly more expensive) application.

But DWD are not the only place where KWin can help to increase the user experience by offering useful content. We always went a great way to make the desktop more useable by for example allowing to click on a splash screen to hide it. I think we can all agree that splash screens are something really useless: look at a stupid window for a few seconds and having to wait till it hides again. Meh.

We can do better: we know which application starts and what it is used for. And as a compositor we can exchange the content. Wouldn’t it be way more awesome to show a short video giving you advice where to buy the latest tips on C++11 book when launching kdevelop?

And there is so much more room to integrate useful content. When starting a video player we can overlay the window with a nice trailer of the latest movie (of course that will be skipable after 5 sec). This is a feature which might also be handy in Present Windows and Alt+Tab. Of course such videos would be perfectly localized as we know your geolocation and we would take great efforts to only offer trailers which might interest you: after all we have baloo to scan your video data and can also look at your Facebook profile. Now trailers are only one part, of course also other areas could be covered. If you have many pictures of cars you might be rather interested in spots about the latest car of your favorite brand instead of move trailers. Of course we will make it possible to configure which areas you are interested in. As we all know KDE is all about configurability.

Why don’t you just…

Relatively regularly I’m asked why I don’t “just” integrate QtCompositor or libweston and call it a day. Well it should be obvious, if it were as simple as “just” we would do that ;-) So instead of replying again and again I thought to give a full blog post explaining the options and why they don’t really suit our needs.

How KWin’s compositor works

Let’s start with a look at the Compositors in KWin:

  • No Compositing (X11 only)
  • XRender (X11 only)
  • QPainter (Wayland only)
  • OpenGL (ES) (X11 and Wayland)

The task of the compositors is to render the windows into a global scene. The information about the windows is taken from the window manager (e.g. the stacking order defines which window is rendered on top) and get transformed by the effect system.

The compositors use platform specific API to get the content of the window and to render it to the screen. E.g. OpenGL can use GLX and use texture from pixmap to map the content of an X11 window to an OpenGL texture and use an XWindow to create the OpenGL context on. The actual compositor which renders doesn’t care whether it’s an X11 Window or a Wayland window or whatever, all it cares about it’s a texture (OpenGL), a pixmap (XRender) or a QImage (QPainter). The platform specific API is abstracted allowing to make the actual compositors completely windowing system agnostic. Designed as a class diagram it looks (simplified) like this:

Now with that much abstraction one could think that there a huge differences between the code and that there is not much sharing going on. But the opposite is the case. The majority of the code is shared. To give an overview I provide the line count of all the files relevant to the OpenGL compositor.

  • Shared code: 3733 (cloc scene_opengl.* abstract_egl_backend.* scene.*)
  • Platforms specific: 1424 (cloc egl* glxbackend.*)

From the platform specific code the glxbackend is about 50 %. The sharing between the egl backends is larger, for the new X11 mode for kwin_wayland the difference is just 20 lines. For QPainter the differences are also rather small – in fact the differences are so small that all is in one source file making something like 200 lines of code for the three backends. Last week I worked on integrating a framebuffer backend into QPainter – although I prior had no idea about it, it was integrated into KWin in less than two hours fully functional with no change to the SceneQPainter at all except writing 50 lines of the backend code.

Now the backends are only part of the whole story. By using our own existing compositor we ensure a similar feature set between X11 and Wayland and can reuse all our effects directly without any adjustments. According to CLOC we have 22000 lines of effect code.

What about QtCompositor

QtCompositor seems to be the natural choice for bringing Wayland support to KWin. After all it’s part of Qt which is what we use in KWin. Except it doesn’t. For a starter it’s not yet a released component which makes it difficult to use for us to develop against, for our packagers to distribute and for our users to test. “But Martin”, you say, “you could develop on it and get it into a release state!”

Yes I could, but would it help KWin? What is the strength of QtCompositor? Providing a QtQuick API to write a Wayland compositor. But KWin doesn’t use QtQuick in the compositor. “But Martin”, you say, “QtQuick is the new cool thing, everybody ports to it, so should you!”. Yes, of course, but it means:

  • Throwing away all our effects
  • Writing new integration code between QtCompositor and the window manager
  • having no code sharing between the Wayland and the X11 compositor
  • No longer being able to provide non-OpenGL compositors as QtQuick requires OpenGL

If I look at this I rather doubt that this would be a “just integrate it”. That looks like a lot of work and duplication to have the same basic window manager for X11 and Wayland. Code sharing would be kind of impossible as the technologies involved are too different.

“But Martin”, you say, “you could at least use QtCompositor for wrapping the Wayland API and use it to map a Wayland buffer to OpenGL!” Yes, that is true and I did consider it and even planned to do so. In the end I decided against it for multiple reasons. First of all this still shows the problem of the unreleased component (yes, I could work on it), but then I wanted to integrate Wayland in KWin with a Facade. Using a facade is something I consider here as rather important. After all we want to use Wayland for quite some time and not run into the situation that we need to port away from a library in a few years. Using a facade makes such a situation much easier. Now if we write a facade anyway what is the benefit of being a facade to QtWayland compared to being a facade to Wayland directly? It just adds another layer of abstraction which might hinder us (and compared we ported lots away from Qt to XCB during the Qt 5 porting removing the abstraction added by Qt). Oh and of course we will have to use Wayland directly, because we will have to use our own protocols for e.g. interfacing with Plasma. Which would mean that we have to interact with two technologies instead of one. Overall I didn’t see an advantage and that’s why KWayland is a facade over Wayland and not over QtCompositor.

What about (lib)weston

“But Martin”, you say, “you could integrate (lib)weston’s backend!”. Yes, weston uses plugins for it’s backends, so in theory it would be possible to load the plugins and interface with them. There were also patches to make this possible through a libweston, but I actually don’t know whether they got integrated (quick check looks like they didn’t).

I would love to have a library which abstracts e.g. kms or framebuffer for me, but honestly I doubt weston is that. Weston was developed as the reference compositor and not as a library to do that. Weston code is always open in my editor, so I know a little bit about the code and can see that it is – well – Weston-specific. I doubt that the code base could be shared in it’s current state. It would need an effort like what has been done for libinput to move this into a well designed API. As long as that doesn’t happen weston’s backends are not directly useable for us.

And even if we would get back to a different feature set. We use QPainter for non-GL rendering, Weston uses pixman. And that is nested into the compositor implementations of Weston. Again: that needs lots of work to get in a state which would allow integration, so overall it’s not a “just use it”.

And all of that does not consider what is more difficult: integrating the C API or writing the code yourself tailored in a way which makes sense in a C++/Qt world.

KWinception

Last week I merged in a few important changes for the upcoming KWin 5.3 release. The rootless Xwayland support is integrated, which means we are a huge step closer to managing Wayland clients. Rendering, input and cursor is already using the Wayland code paths and will be shared with proper Wayland clients. I have already started working on code for that and have it working quite nicely already but decided to delay the integration for Plasma 5.4.

This means that kwin_wayland starts a Wayland server, but except Xwayland there is no application which can properly connect to it. Nevertheless it’s an important step and allows to also test the code in a better way.

In addition I worked on a better support for nesting compositors. So far one had to start Weston in order to test kwin_wayland. This is of course not optimal as it makes the development setup more difficult than it has to be. So last week I looked into developing a new backend which can use an X11 window to render to. This is comparable to Weston which can be used nested on X11. The backend is relatively straight forward: it can render to the created window using either the OpenGL or QPainter compositor, accepts input events and delegates them and passes the cursor from Wayland windows to the X11 window. The tricky part is that we cannot use any of our X11 specific libraries to create the window: we don’t use the xcb QPA, so no support from Qt and we cannot use KWindowSystem as it only allows one xcb_connection and that’s already needed for the Xwayland window manager. So I had to do things myself and as I consider this just a development feature it’s probably the worst X11 client in existance ;-) (Long term KWindowSystem should be changed to support multiple connections, would be very useful for unit tests).

The new backend triggered a small refactoring which makes it easier to implement further backends like e.g. a plain framebuffer or a drm/kms. As a small reminder: there’s an open GSoC idea for implementing the drm/kms backend and application period is to close soon.

Anyway with the nested KWin it’s now possible to create a kwin_wayland instance on the X Server using kwin_x11 and on the kwin_wayland one can create yet another kwin_wayland so that we have a KWinception:

Aus 2015-03-19

To start such a nested compositor use:

kwin_wayland --windowed --xwayland

Please watch the debug output for:

X-Server started on display :1

This tells the id of the created X Server (normally the next free id) and allows to start an application on the nested KWin:
DISPLAY=:1 kwrite

KWin_Wayland picks the windowing system depending on the environment variables which are defined. For DISPLAY it uses X11, for WAYLAND_DISPLAY it uses Wayland. If both are defined Wayland is preferred. If one wants to specify explicitly one can use the command line arguments –x11-display or –wayland-display. E.g.


kwin_wayland --windowed --xwayland --x11-display=:0

For X11 there are also the options –width and –height to specify an initial size (default is currently 1024×768). On Wayland these options are currently not available, the window always opens fullscreen.

Please remember that this is pretty new code and the support is still incomplete. Feature might be missing or broken. This is at the current state expected and because of that we do not yet accept bug reports. If you find a bug which you think should be fixed, please open an editor ;-)

Rootless Xwayland server integration into KWin

Over the last weeks I concentrated my KWin related work on trying to integrate the Xwayland server properly. Xwayland is an interesting step on the way to Wayland as it maps X11 windows to Wayland surfaces. But it also needs an X11 window manager to manage the X11 windows. Overall it allows us to start integrating Wayland into the compositor without too much breakage. It’s still X11 after all, so our existing code base continues to work. And gruadually functionality can be replaced with the Wayland equivalent, so that we can afterwards start integrating proper Wayland clients.

Integrating Xwayland showed interesting challenges. KWin as an X11 window manager requires a running X server prior to start. It also can only communicate with one X server, the code base has many hard constraints on being only one X server. This means if we want to use Xwayland the Xwayland server must be the one KWin uses. Which means Xwayland must be started prior to KWin’s X11 usage. So far KWin also enforced the usage of the “xcb” QPA plugin, the windowing system abstraction plugin for the X world in Qt. This plugin gets loaded directly at application startup and aborts if there is no X server it can connect to.

As you can see KWin needs to ensure that Xwayland is running prior to the application startup. But Xwayland requires a running Wayland server, which is supposed to be provided by KWin and for processing Wayland events we need a running event loop, which is only present after the application started. Looks like we are caught in a dependency loop:
circular-dependencies

After some hacking and experimenting I found a solution which can start the Wayland server and Xwayland prior to the application startup, but it’s considerable fragile and it can only be a temporary solution. In the long run it would of course be better if KWin could use the Wayland QPA plugin provided by QtWayland to connect to it’s own Wayland server and start Xwayland later-on.

Nevertheless I succeeded in getting KWin connect to the Xwayland server and to start transitioning X clients to be rendered using Wayland buffers instead of performing texture from X11 pixmap:

Aus 2015-02-11

But it just hit another problem: it didn’t support accelerated OpenGL rendering on the X server. That’s quite a problem if no X11 client connected to the Xwayland server can use proper OpenGL and it’s even a problem for KWin as KWin uses QtQuick scenes which use the X server. So KWin itself fails to render accelerated UI. Compositing is not affected as we don’t use Qt for that.

The reason for the problem is that Xwayland expects the Wayland “wl_drm” interface to be present. This interface gets created when binding an EGLDisplay to a Wayland display. In case of KWin the EGLDisplay exists after the Compositor is fully initialized. And again we are in a dependency chain: the Compositor gets created and uses the Workspace class. This class controls the complete startup of the X11 window manager which means it’s highly X11 dependent and requires Xwayland to be present. Again we are in a dependency loop.

Breaking up this dependency loop is quite tricky. The Compositor is too deeply nested into the application to be considered started before creating the QApplication. This means we must be able to create the QApplication before we have an X Server running. This means no usage of xcb QPA plugin. As KWin is going to start a Wayland server anyway, it would be good to get KWin to use the wayland QPA plugin.

This was again quite a challenge. The QtWayland QPA plugin performs blocking roundtrips to the Wayland server in the main thread during startup. But the Wayland server is running in the same thread. So a blocking call to the Wayland server dead locks the server. It’s not possible to move the Wayland server at that point into a thread as one cannot start a QThread prior to having the QCoreApplication created.

The solution I developed for this problem involves creating an own event dispatcher prior to creating the QApplication. So we have the event dispatching for the Wayland server ready to use. Just the event loop is not yet running. This allowed to provide a small patch for QtWayland to run any event dispatcher set before creating the QApplication. A test application in the kwayland repository is also adjusted to make use of it (the test application is also able to start an Xwayland server prior to creating the QApplication).

Unfortunately there is still another issue: QtWayland might call eglInitialize in the main thread which again performs a blocking wayland call. This is a problem I haven’t solved yet and currently just hacked around by disabling OpenGL in Qt (which breaks the QtQuick views).

Being able to use the Wayland QPA just creates a new bunch of problems for KWin. KWin still needs to use X11 and thus needs to create an xcb connection. Just that this wouldn’t help much. KWin on X11 doesn’t init the xcb connection, it’s the xcb plugin in Qt which does it and we only access it via QX11Info. Thus KWin needs to be moved away from the usage. Luckily in many cases we already wrapped the functionality as going through the QPA interface (which QX11Info does internally) is too expensive for our use cases. So the changes are not that invasive. But KWin also uses frameworks like KF5WindowSystem which use QX11Info. Even more those frameworks were properly fixed to perform platform checks and don’t do the X11 specific code if it’s not on platform xcb. But it’s providing important X11 window manager functionality for KWin. For some classes like KWindowInfo and KWindowSystem the fix was trivial: don’t use in KWin. There’s also the more low-level NETRootInfo and NETWinInfo and that’s what KWin should use – usage of the KWindowInfo or KWindowSystem inside KWin can be considered a bug. For some other classes it was already partially possible to be used without the xcb plugin on X11. The classes are only used if we compile with xcb present, so it was possible to add more xcb specific methods which can then be used by KWin even if we do not use platform xcb. The required changes will be part of frameworks 5.8 release.

To summary where we are now: we can start kwin_wayland on platform wayland connecting to a Wayland server started by kwin_wayland, we do not require QX11Info in (most of) KF5WindowSystem and KWin. We are a good step closer to the aim, but still not there. The dependency loop is still in place: Workspace starts the Compositor, the Compositor creates the EGLDisplay which is needed to start Xwayland, which provides X11 which is needed for starting the Workspace.

This means: reorder the startup. We need to be able to start the Compositor prior to the Workspace (which could be interesting for kwin_x11, too, as it could improve persumed startup time). This task was easier than expected. Workspace got split into many modules over the last years and most modules which need to be created prior to creating the Compositor do not depend on Workspace and do not depend on X11. In the few cases where it actually does depend on X11 it was not difficult to delay the X11 specific code till after the X11 connection is created.

With all that in place I was able to delay starting Xwayland to after the Compositor is created and Xwayland can provide OpenGL to the connected clients:

This screenshot shows kwin_wayland running on top of Weston connected to an Xwayland server supporting proper OpenGL as can be seen by the output of glxinfo (in the konsole) and glxgears. Also plasmoidviewer just works on top of this X stack.

Of course there is still some work to be done till this is production ready code, but it looks really promising and I hope to have this ready for the KWin 5.3 release. The aim is also to get more and more features changed to use the Wayland functionality instead of the X11 functionality. E.g. for damage event handling it already uses the damage event of a wl_surface instead of creating an X11 damage handle. This makes supporting Wayland clients easier afterwards.

Looking at the memory improvements of KDecoration2

This week I finally moved kdecoration2 to the kde/workspace project structure and merged in the required backend code in kwin. This means the upcoming 5.2 release will ship with the new Breeze window decoration by default.

Thanks to the usage of the new library we get a nice performance boost in KWin. Some of this I already explained in my blog post announcing KDecoration2. The huge advantage here is that we do no longer use a QWindow as a backend and only require to hold an image the same size as the decoration. But there are more nice improvements around it. When I started designing KDecoration2 one of my aims was to focus it more on the use cases of a composited setup and design the API in such a way that a default decoration could be more efficient. Our previous default decoration Oxygen was rather heavy and KWin was not able to provide features to Oxygen which would make it more efficient. From a memory point of view the main issue with Oxygen were the shadows. With the old decoration API the shadows were part of the decoration. So the renderer got one big image for the shadow and the decoration part. But the shadow is the same for all inactive windows and could be defined by a much smaller image (lots of information is redundant). For KDecoration2 I looked at our existing shadow sharing mechanismn on X11 level and adopted a similar API which allows to provide a compressed shadow to the compositor. That way the shadow is no longer part of the window decoration and does not need to be part of the OpenGL texture. Of course this comes at the cost of needing more rendering calls (I deliberately moved the Time-memory-tradeoff towards better memory usage).

The real improvement comes when the decoration plugin can share the shadow between the decorations. This is being done for Breeze: there is never more than one shadow being created. In Breeze the shadow always has a size of 60×60 pixels, so we save 3600 pixels per decorated window. This is in addition to the saved memory for the compressed shadow instead of having the shadow as part of the window.

Today I went a step further and also modified the code in our OpenGL compositor to share the textures being created for the decoration shadow. So if two decorations have the same shadow, they will also use the same texture. This does not only save us some texture memory, but also means less pixel transfers from CPU to GPU. So a rather nice improvement. For our default decoration Breeze this means that only one shadow is created in the decoration plugin and also only one OpenGL texture needs to be created.

Unfortunately not all decoration plugins can benefit from this change. Also with KDecoration2 the problems outlined for Aurorae still exist and the change in the shadow mechanismn doesn’t improve the situation. The hope to improve this is in Qt 5.4 which gives us new useful features like QQuickRenderControl which allows us to integrate the OpenGL context used for QtQuick with our compositor’s OpenGL context. The hope is that we can render the QtQuick based decoration into an FBO and share the texture with our compositor, so that we can just bypass the normal decoration rendering process which is too costly in the case of Aurorae. I already switched the normal rendering to QQuickRenderControl if kwin is compiled with Qt 5.4 and the result is looking rather promising. Still the memory usage and performance of Aurorae will never be as good as the memory usage and performance of a native decoration. This has never been the aim of Aurorae.

Window and Desktop Switcher moved to Look’n’Feel Package

Today we did an important change in how KWin will distribute its assets in the upcoming 5.2 release. When we started our thoughts about the Look’n’Feel Package and how we want to have meta themes for the complete Plasma workspace we also wanted to have this for the Window and Desktop switcher provided by KWin. So the structure of the Look’n’Feel Package already has all the pieces for including the Window and Desktop Switcher, but it was not used. Now we finally addressed this for the 5.2 release and moved the default switcher into the Look’n’Feel Package and KWin can locate the switchers from the Look’n’Feel Package.

At the same time we want to follow in a better way the “Simple by default, powerful when needed” approach. Our configuration should be simple and thus not offer an overwhelming amount of switchers. But it also should be powerful and thus one can install additional switchers either through GHNS or through your package manager. The result is that all the additional switchers so far shipped with KWin were moved to kdeplasma-addons repository. So all switchers are still available (powerful when needed), just not shown when opening the configuration menu (simple by default).

This also opens the door for including more switchers in the plasma addons repository. So far we had been very reluctant to add more switchers to the KWin repository. There were already too many switchers installed by default and it gives a feeling of we don’t know what we want. By only installing one switcher by default this improves significantly and allows to add high quality switchers with enough differentiation to other switchers to the plasma addons repository.

The change has also some implications for users of non Plasma desktop environments wanting to use KWin as their window manager. By moving the switchers out, KWin removes some of the Plasma dependencies. All switchers provided by KWin are using Plasma components, the default switcher is part of the design concept for Plasma 5 following the same idea as other similar components. Thus KWin had a direct dependency on Plasma with the window switchers. This is now kind of solved by not offering any switcher at all.

My suggestion for desktop environment projects wanting to use KWin is to provide their own default Look’n’Feel package with a Window and Desktop switcher specific for their environment. On the other hand I don’t see a problem with providing a simple fallback theme in KWin. And of course there are still the Desktop Effects for switching between Windows (CoverSwitch and FlipSwitch) installed by default and just need to be enabled.

Window decoration themes in KDecoration2

Most of the window decorations available for KWin are not native decorations but themes for a native theme engine, such as deKorator, Smaragd, QtCurve or my own Aurorae. Themes are much easier to design and to distribute than a native decoration which has to be implemented in C++ and be distributed by the Linux distribution. Thus themes are an important part of the decoration system.

But we did a very bad job of integrating the themes into our configuration system. The configuration system only knows about native decorations and doesn’t know that the native decoration is in fact a theme engine. This makes selecting a theme difficult, because a user has to first select the theme engine and then configure this to select the theme. Downloading new themes through GHNS is also difficult as again it requires to go through the configuration of the theme engine. We can do better.

With Aurorae I tried to address some of the problems by extending the configuration system to know about Aurorae, to be able to find the themes and render previews for it. This only worked because Aurorae and the configuration module are in the same source tree and could share code. Nevertheless it needed to have multiple code paths in the configuration module to load the native themes, Aurorae’s SVG themes, Aurorae’s QML themes and to render the three different kind of themes.

The solution works for one theme engine, but others are still not supported. Which is something I find very sad as it turns the theme engines to second class citizens and also looks bad as my theme engine has full support while others don’t, while doing as a good or even a better job at themeing than Aurorae.

When I started to think about KDecoration2 and started to draft the API design I wanted to make sure that theme engines become a first class citizen in KWin. Last week I started to port the Aurorae theme engine to KDecoration2 and added the missing pieces to make KDecoration2 fully theme aware. The configuration of the selected theme is moved into the framework and the selected theme is passed to the native plugin when a Decoration gets created.

As a result of this work the command line options for kdecoration-viewer changed to:

kdecorationviewer [plugin] [theme]

which allows us to load for example the plugin for Aurorae with one of the SVG based themes:

To announce support that the decoration plugin is a theme engine, the decoration plugin has to put some information into the JSON meta data:

"org.kde.kdecoration2": {
        "themes": true,
        "defaultTheme": "kwin4_decoration_qml_plastik",
        "themeListKeyword": "themes"
    }

If the value for themes is present and true the framework will pass theme information to the Decoration. The framework looks in its configuration for the theme to be used, if there is none it falls back to the defaultTheme from the JSON meta data. The configured theme is passed to the Deocration when being created through KPluginFactory::create which takes a QVariantList as argument. As first element the framework passes in a QVariantMap with a key/value pair of “theme” as key and the configured theme as the value.

The last value of the meta data above is themeListKeyword which is used by the configuration module to locate all themes and provide them. I have not yet finalized the mechanismn so this is still experimental code. The keyword is used to create a different Object through the KPluginFactory. Right now this is a QObject with a QVariantMap property called “themes”. Each key is the user visible name and the value is the internal plugin name. That’s enough information for the configuration module to create a dedicated instance for each of the themes, create a preview for it and properly load/store the information. It allows us to have a configuration module which currently looks like this:

This new configuration module does not have any Aurorae specific code any more. It just knows about plugins and themes and can display them all. As one can see in this screenshot the KCM does not need to know whether it’s a QML based theme (e.g. Plastik) or a SVG based theme (all the others). Which is a nice improvement to the situation before.

The mechanismn for locating the themes will probably change and be moved to KDecoration2 directly. It needs some more tweaks to expose GHNS information and support looking for new themes and deleting existing themes. So there is still a little bit of work to be done. But overall the state is now looking really good and I will soon start the review process for the new API so that KDecoration2 will land in Plasma 5.2. Of course that will be faster if we get more help to finalize the last missing pieces.

Libinput integration in KWin/Wayland

Today I pushed my outstanding branch to get libinput support into kwin_wayland. Libinput is a very important part for the work to get a full Wayland session in Plasma which means we reached a very important milestone. As the name suggests it allows us to process input events directly. KWin needs to forward the input events to the currently active application(s) and also interpret them before any other application gets them. E.g. if there is a global shortcut KWin should intercept it and not send it to an application.

Why libinput integration in KWin?

KWin/Wayland already supported input handling by being a Wayland client and connecting to a Seat. But especially for pointer events this was not sufficient at all. We have quite some code where we warp the pointer and Wayland doesn’t support this (and shouldn’t). Warping the pointer is normally considered evil as it can introduce quite some problems if applications are allowed to warp the pointer. E.g. it can create security issues if you start typing your password and a malicious applications warps the pointer to trick you entering your password into a password field of the malicious application. Also from a usabililty perspective it can be problematic as it makes the system behave in an unpredictable way.

On the other hand if the application is a window manager/compositor the need for warping cursors arises. For example the screen edge handling pushes the cursor slightly back which needs cursor warping. Or you can move a window with the cursor keys (hold Control key to have very precise moving) and in these cases we need to warp the pointer. With libinput this is possible again as KWin gets put in control of the input events directly. It completely bypasses the currently used Wayland compositor.

Libinput is also an important piece in the puzzle for a full Wayland session which does not rely on another Wayland compositor. So far KWin/Wayland can only be used in a nested scenario – which is important for development and interesting new possibilities like the idea for a SoK project – but we also want full support without the need for a Wayland session. This means we need to handle input (which libinput does) and need to interact with DRM directly. DRM support is still missing. This could be an interesting GSoC project next year ;-)

The merged implementation does not support all of libinput yet. Most important touch screen support is omitted as I don’t have a touch enabled device. I plan to sit down with fellow KDE developers who have a touchscreen enabled device and implement that part together. Also I will contact the VDG to define some global touch gestures to interact with the system (I’m quite interested in having a touch gesture to activate Present Windows). There’s lots of work to be done here and I would welcome any helping hand.

Security challenges

Processing input events directly comes with a slight problem, though: one needs to be root to read the events. And that’s obviously an absolute no-go for KWin. KWin may never ever be executed with root privileges and also not an suid which drops privileges again (which also wouldn’t help in that case but so what). The application has not been designed for running as root. The same is the case for Weston and obviously I looked at how it’s solved there and there is a very neat solution to support the use case we have in logind. The session controller can ask logind to open devices and logind provides a file descriptor to the opened device. In addition logind automatically takes care to close the file descriptors when a VT switch happens, which is extremely convenient for the use cases of Wayland compositors. So obviously I went for this solution as all it needs is connecting to very few D-Bus calls. This means the libinput integration in kwin_wayland will have a runtime dependency to a logind D-Bus interface. Of course this does not affect kwin_x11, neither does it affect kwin_wayland without libinput integration, but of course without libinput you won’t get support for all features. There is one caveat though: logind will blank the VT when the session controller goes away. So make sure to not run kwin_wayland with libinput support on your primary X session. Because of that libinput support must be explicitly enabled with the –libinput command line switch of kwin_wayland.

Current state and downsides of libinput and logind

As libinput does not yet have a stable release, the dependency is still optional and it’s possible to build kwin_wayland without libinput support. This is currently very important for the non-Linux operating systems, which might want to provide kwin_wayland, as libinput only supports Linux.

I hope that libinput will become available on other platforms. At XDC during the BSD presentations I heard at least one presenter touch the topic. So I’m optimistic that in the long run this will happen as we also see that DRM and KMS is nowadays in quite a good shape on the BSDs. For KWin development it’s of course important that we have only one library to interact with. Otherwise it means platform dependent code which is hard to develop and extremely difficult to test for the main developers not using such a platform. So if you want to get kwin_wayland on non-Linux, please consider putting the energy into getting libinput working (challenge is udev) as that will help all systems and not just KWin. After all we want to stand on the shoulders of giants ;-)

Logind is in a similar situation. It is developed as a component in systemd, which isn’t available on all systems which run KWin. Luckily we don’t depend on logind directly but only use a subset of a well defined D-Bus interface and that interface can be provided by other tools as well. Something like that is already being worked on for the BSD’s.
Like with libinput, I would much prefer to keep KWin lean and efficient and not complicate the code base and development by including libraries for specific platforms or having security relevant code around. As written above: using suid wrappers is very much a no-no to me. But of course it would be possible to implement the subset of the D-Bus in an independent project and provide it. KWin would happily use it, it just needs someone to write the code. So if enough people care, I’m quite sure that there will be a developer stepping up and writing the code.

I decided to put out a small FAQ here for those who have questions about the implications of the above:

FAQ

Does that mean KWin (and Plasma) depend on systemd?

No.

But it depends on logind?

No. It uses one D-Bus interface provided by logind. It doesn’t care which program is providing this D-Bus interface. It can be logind or logind-shim or the implementation being worked on for the BSDs. Even a small binary just providing the used D-Bus interfaces would work.

You should not use logind, there must be a different solution!

I’m sorry I did not find any solution which was as efficient and secure as the one provided by logind. Of course there are solutions like weston-launch, but they introduce a lot of complexity – both on the coding side and on the installation side. As such a solution would need to be suid, I’m very reluctant to the idea. We shouldn’t introduce such possible security risks, if there are better solutions available. Logind is simply providing a feature which is needed by kwin_wayland.

Does that affect KWin on X11?

No, that only affects kwin_wayland.

But there is no logind for the BSDs! So I won’t be able to run kwin_wayland on BSD systems?

Unfortunately the fact that logind is missing is least of your problems on BSD. Logind support is only needed for libinput which right now is not available on BSD. The kwin_wayland binary on BSD will not try to interact with logind. I’m sorry I don’t have a solution for the input stack on BSDs. I really hope the BSD developers can come up with a solution for this as we don’t have the resources to build a separate input solution for one platform.

How can I change KWin to not use logind?

As I noted, it is important to me that KWin is secure and that the code base is as easy to understand as possible. I don’t like the idea of having ifdefs all over the place and multiple solutions as that results in bitrot. When I pushed the libinput change it directly failed to build on the CI system as the ifdefs introduced a variation which I couldn’t test on my system. Each ifdef and each platform-specific solution increases the development and maintenance costs significantly. This means that I will only accept patches which don’t introduce the above mentioned problems. Preferrable a small wrapper binary could provide the needed D-Bus interface for KWin and other applications which need this functionality. This would not need changes in KWin at all and would be from my perspective the perfect solution.

Why won’t you implement such a wrapper binary?

Honestly there are a million things I would do if I had time, but a day has only 24 h and I have to prioritize my work. Just check the Wayland TODO list for what we all need to do to get KWin/Wayland ready. Why don’t you open your editor and get some work done? ;-)

But if KWin uses logind, Slackware will drop all of KDE!

Yes, I have read that (see comments). Luckily the situation for Slackware is similar to the BSDs: it doesn’t matter right now. Slackware doesn’t provide Wayland packages yet, so the logind support won’t be used as there is no kwin_wayland binary which could be built. And if enough people care, as I said, one or more of them can write the wrapper binary and KWin and other compositors will work just fine.

How can i help?

Best by writing code :-) See the TODO list I linked in an above answer. Also it would be good if someone documented the steps to get kwin_wayland running and how to develop on it cough.

XDC 2014

Last week I had been at the 2014 X.Org Developer’s Conference in Bordeaux, France. After Nuremberg two years ago this was my second XDC and I am very glad that I was able to attend and want to thank KDE e.V. for sponsorship. With the X.Org windowing systems (X11 and Wayland) being at the heart of KWin it is good to get an update on the latest development efforts.

Especially the talks about Wayland related topics were quite important. It was interesting to see what the Tizen developers are working on with their IVI-Shell and the discussions around the talk highlighted the need for a careful put out standardization process for the Wayland shell interfaces: if the interface does not contain enough functionality, multiple incompatible interfaces could be the result making it difficult for applications to interact with the system. On the other hand if the interfaces are too broad it will result in compositors not being able to completely implement the interface resulting again in compatability issues.

More directly touching our development were the talks concerning input handling with libinput. I already have some code prepared and expect that KWin/Wayland 5.2 will come with libinput support (more about that in a future blog post when the work is ready). The talks also showed that I need to reach out to the Krita community to get support for tablet devices. I don’t have access to such a device and even if, I would not know or understand the needs for these kind of input devices. There I will need the expertise of developers using tablet devices and I hope to find this within the Krita community.

KWin/Wayland 5.1 gained support for the fullscreen shell interface. My idea when adding this was to not have to implement DRM support in KWin, but (for the time being) leverage Weston. This simplifies development and allows us to move forward on a higher speed. Jason Ekstrand’s talk showed that the fullscreen shell provides more interesting aspects than our use case. The shell can also be used for use cases such as screen sharing: a compositor renders in addition to a fullscreen shell provided by a different compositor which can use it to e.g. capture a video stream or forward an rdp session. Very interesting and quite useful that we already support it and won’t have to add additional support for rdp into each compositor.

Last but not least I want to highlight Steve Dodier-Lazaro and Martin Peres’s talk about security in a Wayland world. They presented a solution for privileged clients which is very important for our development in the Plasma world: there will be interfaces between Plasma and KWin and nobody except Plasma should be allowed to bind to them. The library presented in this talk will solve this problem and many more. Even more the talk touched various interesting security related topics which I had never thought about in that way. For example setting a window fullscreen provides a security risk: it could fake a lock screen interface tricking the user into entering the password. On X11 the Window Manager/Compositor is unable to mitigate this risk. But on Wayland a compositor can e.g. overlay the window and explicitly ask the user whether the window is allowed to go fullscreen – compare to what browsers already do today. I’ll reach out to the VDG to come up with a good solution.

More interesting is the question on how we can trust password dialogs. This is something I already spent some thoughts about in the past. The problem right now is that a malicious application can easily trick the user into entering the password by faking a system dialog (e.g. polkit authentication or kwallet dialog). We need to find a way to show to the user that this is a trusted dialog and that the password is entered for a trusted application. A difficult task. There are many ways we can make the dialog in a way that nobody would be able to fake it: color the decoration, wobble it, put it into the Plasma’s notification area, but none of them are obvious to a user that no other process can fake it. As explained in the talk it needs an out-of-band communication channel to inform the user that this dialog is trusted. An idea which came to my mind is using KDE Connect to send a token to the connected smartphone to indicate that the dialog is secure.

Unfortunately the state of KWin/Wayland has not processed enough compared to Akademy that I could have done a talk about it. After all it was just a few weeks. So instead I decided to give a talk about what we mostly did last year: port KWin to XCB.

An Update on kwin_wayland

With the initial release of Plasma 5.0 behind us I also started to look more in the direction of Wayland again. Now I’m kind of in full flow on Wayland work and kwin_wayland is progressing nicely. Yes, KWin 5.1 will introduce a new binary called kwin_wayland to complement the kwin_x11 binary which got introduced in KWin 5.0.

Now I do not want to list all the changes as you can hardly express them all in a blog post, but I can point to my Akademy talk. I will provide a small overview of the current state, what is new in KWin 5.0, what will be new in KWin 5.1 and where the journey is going.

I'm going to Akademy 2014

Of course there is lots of work going on and help is always appreciated. We started to use a public available task tracker on todo.kde.org. Also I have to say that there are still quite some open tasks for kdecoration2. Please help as I cannot split myself and it would be super important to release KWin 5.1 with kdecoration2.