Category Archives: KDE

Turning the world upside down

This blog post is a rather important one for me. Not only is it the first blog post which I write in a nearby cafe sitting in the sun and enjoying spring, it is also the first blog post written in a Plasma session running inside kwin_wayland.

This marks an important step in the process of getting Plasma and KWin ready for Wayland as we have reached a state where I can dogfood kwin_wayland on one of my systems. It means that I’m confident enough to write a blog post without having to fear that the session crashes every few seconds (granted I press save a lot nevertheless).

So what do I mean when saying that I’m running a kwin_wayland session? Does it mean that everything is already using Wayland? No, unfortunately not, rather the opposite: all running applications are still using X11, but we use a rootless Xwayland server. The only Wayland application in this setup is Xwayland and KWin itself.

Nevertheless it’s a noteworthy achievement as we now have the architecture pulled upside down. No longer does KWin connect to either an X-server or a Wayland-server which does the rendering to the screen, it does it now all by itself. Over the last week I worked on the DRM (Direct Rendering Manager) backend for kwin_wayland. The DRM backend will become the primary backend for KWin, doing mode-setting, creating buffers for rendering and performing page flips of those buffers on the connected outputs and rendering the mouse cursor on the outputs. In the current state it’s able to get all connected outputs and perform mode setting on them. In addition it can power both compositors: OpenGL and QPainter. The QPainter one is the rather trivial: all we need are two buffers which we map into a QImage to render on. With each rendered frame the buffers are swapped and the last rendering buffer gets presented.

The OpenGL compositor is slightly more difficult. Here we need to introduce another component: GBM (generic buffer management). Gbm allows us to create an EGLDisplay for our DRM device, create an EGLSurface for each of the outputs and get the buffer from the surface to be passed to DRM for presenting. Unfortunately GBM is currently only available on Mesa and given NVIDIA’s presentation at last years XDC it looks like NVIDIA will come up with a different solution. My hope was that this would have settled at the time when we start implementing the solution, but unfortunately that’s not the case yet. So for the foreseeable future it looks like we will first have a Mesa specific backend, then later on can add a new NVIDIA-specific backend and hope that at some future point in time Mesa also supports NVIDIA’s solution (after all it was presented as a generic solution) and we can drop the GBM backend. The situation is none I like, but at the moment we can only ignore the proprietary drivers.

Another noteworthy detail is how KWin opens the DRM device. KWin runs as the normal non-privileged user – thus is not allowed to open the device. Just like libinput integration the DRM backend uses logind to open the device file. Given that we already have support for that in KWin it was a straight forward, elegant and secure way to implement it. I know, that not everybody will agree with this solution. But of course it’s just a DBus interface and anybody can implement it.

There is still quite some work needed before this backend, which got merged into master today, will be fully functional. For example all outputs are set to (0/0) that is render overlapped sections. Here we will in one way or another integrate with kscreen to get sane defaults from the start. I’m so looking forward to being responsible for screen layouts. If I look at all the terrible things one has to do to keep properly updated on XRandR… Now we won’t render with a wrong viewport because different parts of the stack are stuck in different states and no, we won’t freeze just because applications start and call an XRandR call. There are so many things which “just work” and which have been painful on X11. From my first tests on hardware I’m confident that we will finally have working multi-screen handling which is adequate for 2015.

Of course this is a wonderful time to start working on KWin. There are lots of small tasks to work on. I am constantly adding new tasks to our todo list. And if you ask, I can always provide some more small tasks.

There is no KEndorsements

You might have noticed that my yesterday’s blog post was an April Fool. With this post I want to explain a little bit about the idea of the article.

First of all I want to thank the VDG, especially Jens and Ken, for their support by providing the great mockups. That was kind of the first element to notice that it’s an April Fool: normally I do only integrate screenshots by myself and I would not have been able to show a drawing in Krita.

The idea for this April Fool is based on the reports about the inclusion of an adware tool called superfish on Lenovo notebooks. Especially the initial response by Lenovo claiming that this malicious software got installed “to improve the shopping experience”.

Nevertheless the article contains truth, half truth and thing we would never do. So let’s look at some of the ideas presented and evaluate them. The first idea presented was share buttons integrated in the browser’s window decoration. This is an idea which already got introduced similarily through Share-Like-Connect. Personally I think it’s a good idea if browsers would provide such global buttons and in return filter out all the user tracking share buttons on the web sites. A global area for share buttons directly integrated in the browser could increase the security of users a lot.

The next idea is about monitioring the usage behavior of the user. For this I must say that in a Wayland world KWin would have enough information to do that, but we don’t look at the information at all. We do not care what an application is: we don’t know that a given window is “Krita”, we don’t know that it should be used with a drawing tablet, we don’t analyze any input and just pass it through to the application. KWin is completely ignorant on what the application does with input events passed to them and as ignorant to what a window content looks like. To us a window is just a texture we render to the screen. We don’t know that there is a “drawing area” or a “video” element, it’s just one texture. Similarly we do not know (and care) about input events. All we do is determine which window should get the event and pass it to it. We are quite aware that by passing all input events through KWin we could do evil things. The good thing is that our software is open source and you can see what we do (relevant code file is called input.cpp, events come in from the files under the libinput/ directory).

Last but not least I want to point out that we as a window manager will not integrate advertisement into the window management. We do not care whether you watch a video, we might not even know it, because as explained: we are ignorant about what you do. I consider applications which integrate advertisement without the user’s explicit acknowledgement as malware.

The bad part of my blog post is that I expect these things to happen. And if we look around us we can see that it already happened: adware is unfortunately also available in FLOSS. And in the walled garden’s of the proprietary world we will see more attempts to gain more information about the user’s to provide even better matching advertisement. It will blur more and more.

Enhancing user experience through the Compositor

This article is an April Fool. For an explanation please see the update.

Some time ago Ken introduced the conecept of dynamic window decorations (DWD) and during the last Plasma sprint there was already some work on experimenting with an implementation. DWD are extremely promising to enhance the user experience by morphing the application and window manager scope together.

The basic concept of DWD is to get content of the application inside the scope of the window manager. But what works one way works also the other way: we can enhance the user experience by providing additional information relevant for the window inside the window decoration.

An example for this could be a volume applet for music and audio players. But of course it doesn’t stop there, today is all about the user content and integrating the web. For web browsers we can integrate a global “Like It on Facebook” button. And by analyzing the user behavior we can provide even more help. We can notice if you struggle using Krita and provide a link to where to get a better drawing tablet.

If you use digiKam you might be interested in some books about photogrophy. We know which music you listen to on Amarok and show you that the new album of your favorite music group just got released. Not every application integrates stores in a proper way. This is a shame as it does not provide the best possible user experience and KWin as a window manager can help there to add missing store functionality where it is needed.

Of course this also allows to show you better alternatives. If we see that you struggle with wine emulated applications we can provide the Linux alternative. We can show that new updates are available for your currently used application. If you use sub-par free software applications we can show you were to get a proper (slightly more expensive) application.

But DWD are not the only place where KWin can help to increase the user experience by offering useful content. We always went a great way to make the desktop more useable by for example allowing to click on a splash screen to hide it. I think we can all agree that splash screens are something really useless: look at a stupid window for a few seconds and having to wait till it hides again. Meh.

We can do better: we know which application starts and what it is used for. And as a compositor we can exchange the content. Wouldn’t it be way more awesome to show a short video giving you advice where to buy the latest tips on C++11 book when launching kdevelop?

And there is so much more room to integrate useful content. When starting a video player we can overlay the window with a nice trailer of the latest movie (of course that will be skipable after 5 sec). This is a feature which might also be handy in Present Windows and Alt+Tab. Of course such videos would be perfectly localized as we know your geolocation and we would take great efforts to only offer trailers which might interest you: after all we have baloo to scan your video data and can also look at your Facebook profile. Now trailers are only one part, of course also other areas could be covered. If you have many pictures of cars you might be rather interested in spots about the latest car of your favorite brand instead of move trailers. Of course we will make it possible to configure which areas you are interested in. As we all know KDE is all about configurability.

Why don’t you just…

Relatively regularly I’m asked why I don’t “just” integrate QtCompositor or libweston and call it a day. Well it should be obvious, if it were as simple as “just” we would do that ;-) So instead of replying again and again I thought to give a full blog post explaining the options and why they don’t really suit our needs.

How KWin’s compositor works

Let’s start with a look at the Compositors in KWin:

  • No Compositing (X11 only)
  • XRender (X11 only)
  • QPainter (Wayland only)
  • OpenGL (ES) (X11 and Wayland)

The task of the compositors is to render the windows into a global scene. The information about the windows is taken from the window manager (e.g. the stacking order defines which window is rendered on top) and get transformed by the effect system.

The compositors use platform specific API to get the content of the window and to render it to the screen. E.g. OpenGL can use GLX and use texture from pixmap to map the content of an X11 window to an OpenGL texture and use an XWindow to create the OpenGL context on. The actual compositor which renders doesn’t care whether it’s an X11 Window or a Wayland window or whatever, all it cares about it’s a texture (OpenGL), a pixmap (XRender) or a QImage (QPainter). The platform specific API is abstracted allowing to make the actual compositors completely windowing system agnostic. Designed as a class diagram it looks (simplified) like this:

Now with that much abstraction one could think that there a huge differences between the code and that there is not much sharing going on. But the opposite is the case. The majority of the code is shared. To give an overview I provide the line count of all the files relevant to the OpenGL compositor.

  • Shared code: 3733 (cloc scene_opengl.* abstract_egl_backend.* scene.*)
  • Platforms specific: 1424 (cloc egl* glxbackend.*)

From the platform specific code the glxbackend is about 50 %. The sharing between the egl backends is larger, for the new X11 mode for kwin_wayland the difference is just 20 lines. For QPainter the differences are also rather small – in fact the differences are so small that all is in one source file making something like 200 lines of code for the three backends. Last week I worked on integrating a framebuffer backend into QPainter – although I prior had no idea about it, it was integrated into KWin in less than two hours fully functional with no change to the SceneQPainter at all except writing 50 lines of the backend code.

Now the backends are only part of the whole story. By using our own existing compositor we ensure a similar feature set between X11 and Wayland and can reuse all our effects directly without any adjustments. According to CLOC we have 22000 lines of effect code.

What about QtCompositor

QtCompositor seems to be the natural choice for bringing Wayland support to KWin. After all it’s part of Qt which is what we use in KWin. Except it doesn’t. For a starter it’s not yet a released component which makes it difficult to use for us to develop against, for our packagers to distribute and for our users to test. “But Martin”, you say, “you could develop on it and get it into a release state!”

Yes I could, but would it help KWin? What is the strength of QtCompositor? Providing a QtQuick API to write a Wayland compositor. But KWin doesn’t use QtQuick in the compositor. “But Martin”, you say, “QtQuick is the new cool thing, everybody ports to it, so should you!”. Yes, of course, but it means:

  • Throwing away all our effects
  • Writing new integration code between QtCompositor and the window manager
  • having no code sharing between the Wayland and the X11 compositor
  • No longer being able to provide non-OpenGL compositors as QtQuick requires OpenGL

If I look at this I rather doubt that this would be a “just integrate it”. That looks like a lot of work and duplication to have the same basic window manager for X11 and Wayland. Code sharing would be kind of impossible as the technologies involved are too different.

“But Martin”, you say, “you could at least use QtCompositor for wrapping the Wayland API and use it to map a Wayland buffer to OpenGL!” Yes, that is true and I did consider it and even planned to do so. In the end I decided against it for multiple reasons. First of all this still shows the problem of the unreleased component (yes, I could work on it), but then I wanted to integrate Wayland in KWin with a Facade. Using a facade is something I consider here as rather important. After all we want to use Wayland for quite some time and not run into the situation that we need to port away from a library in a few years. Using a facade makes such a situation much easier. Now if we write a facade anyway what is the benefit of being a facade to QtWayland compared to being a facade to Wayland directly? It just adds another layer of abstraction which might hinder us (and compared we ported lots away from Qt to XCB during the Qt 5 porting removing the abstraction added by Qt). Oh and of course we will have to use Wayland directly, because we will have to use our own protocols for e.g. interfacing with Plasma. Which would mean that we have to interact with two technologies instead of one. Overall I didn’t see an advantage and that’s why KWayland is a facade over Wayland and not over QtCompositor.

What about (lib)weston

“But Martin”, you say, “you could integrate (lib)weston’s backend!”. Yes, weston uses plugins for it’s backends, so in theory it would be possible to load the plugins and interface with them. There were also patches to make this possible through a libweston, but I actually don’t know whether they got integrated (quick check looks like they didn’t).

I would love to have a library which abstracts e.g. kms or framebuffer for me, but honestly I doubt weston is that. Weston was developed as the reference compositor and not as a library to do that. Weston code is always open in my editor, so I know a little bit about the code and can see that it is – well – Weston-specific. I doubt that the code base could be shared in it’s current state. It would need an effort like what has been done for libinput to move this into a well designed API. As long as that doesn’t happen weston’s backends are not directly useable for us.

And even if we would get back to a different feature set. We use QPainter for non-GL rendering, Weston uses pixman. And that is nested into the compositor implementations of Weston. Again: that needs lots of work to get in a state which would allow integration, so overall it’s not a “just use it”.

And all of that does not consider what is more difficult: integrating the C API or writing the code yourself tailored in a way which makes sense in a C++/Qt world.

KWinception

Last week I merged in a few important changes for the upcoming KWin 5.3 release. The rootless Xwayland support is integrated, which means we are a huge step closer to managing Wayland clients. Rendering, input and cursor is already using the Wayland code paths and will be shared with proper Wayland clients. I have already started working on code for that and have it working quite nicely already but decided to delay the integration for Plasma 5.4.

This means that kwin_wayland starts a Wayland server, but except Xwayland there is no application which can properly connect to it. Nevertheless it’s an important step and allows to also test the code in a better way.

In addition I worked on a better support for nesting compositors. So far one had to start Weston in order to test kwin_wayland. This is of course not optimal as it makes the development setup more difficult than it has to be. So last week I looked into developing a new backend which can use an X11 window to render to. This is comparable to Weston which can be used nested on X11. The backend is relatively straight forward: it can render to the created window using either the OpenGL or QPainter compositor, accepts input events and delegates them and passes the cursor from Wayland windows to the X11 window. The tricky part is that we cannot use any of our X11 specific libraries to create the window: we don’t use the xcb QPA, so no support from Qt and we cannot use KWindowSystem as it only allows one xcb_connection and that’s already needed for the Xwayland window manager. So I had to do things myself and as I consider this just a development feature it’s probably the worst X11 client in existance ;-) (Long term KWindowSystem should be changed to support multiple connections, would be very useful for unit tests).

The new backend triggered a small refactoring which makes it easier to implement further backends like e.g. a plain framebuffer or a drm/kms. As a small reminder: there’s an open GSoC idea for implementing the drm/kms backend and application period is to close soon.

Anyway with the nested KWin it’s now possible to create a kwin_wayland instance on the X Server using kwin_x11 and on the kwin_wayland one can create yet another kwin_wayland so that we have a KWinception:

Aus 2015-03-19

To start such a nested compositor use:

kwin_wayland --windowed --xwayland

Please watch the debug output for:

X-Server started on display :1

This tells the id of the created X Server (normally the next free id) and allows to start an application on the nested KWin:
DISPLAY=:1 kwrite

KWin_Wayland picks the windowing system depending on the environment variables which are defined. For DISPLAY it uses X11, for WAYLAND_DISPLAY it uses Wayland. If both are defined Wayland is preferred. If one wants to specify explicitly one can use the command line arguments –x11-display or –wayland-display. E.g.


kwin_wayland --windowed --xwayland --x11-display=:0

For X11 there are also the options –width and –height to specify an initial size (default is currently 1024×768). On Wayland these options are currently not available, the window always opens fullscreen.

Please remember that this is pretty new code and the support is still incomplete. Feature might be missing or broken. This is at the current state expected and because of that we do not yet accept bug reports. If you find a bug which you think should be fixed, please open an editor ;-)

Rootless Xwayland server integration into KWin

Over the last weeks I concentrated my KWin related work on trying to integrate the Xwayland server properly. Xwayland is an interesting step on the way to Wayland as it maps X11 windows to Wayland surfaces. But it also needs an X11 window manager to manage the X11 windows. Overall it allows us to start integrating Wayland into the compositor without too much breakage. It’s still X11 after all, so our existing code base continues to work. And gruadually functionality can be replaced with the Wayland equivalent, so that we can afterwards start integrating proper Wayland clients.

Integrating Xwayland showed interesting challenges. KWin as an X11 window manager requires a running X server prior to start. It also can only communicate with one X server, the code base has many hard constraints on being only one X server. This means if we want to use Xwayland the Xwayland server must be the one KWin uses. Which means Xwayland must be started prior to KWin’s X11 usage. So far KWin also enforced the usage of the “xcb” QPA plugin, the windowing system abstraction plugin for the X world in Qt. This plugin gets loaded directly at application startup and aborts if there is no X server it can connect to.

As you can see KWin needs to ensure that Xwayland is running prior to the application startup. But Xwayland requires a running Wayland server, which is supposed to be provided by KWin and for processing Wayland events we need a running event loop, which is only present after the application started. Looks like we are caught in a dependency loop:
circular-dependencies

After some hacking and experimenting I found a solution which can start the Wayland server and Xwayland prior to the application startup, but it’s considerable fragile and it can only be a temporary solution. In the long run it would of course be better if KWin could use the Wayland QPA plugin provided by QtWayland to connect to it’s own Wayland server and start Xwayland later-on.

Nevertheless I succeeded in getting KWin connect to the Xwayland server and to start transitioning X clients to be rendered using Wayland buffers instead of performing texture from X11 pixmap:

Aus 2015-02-11

But it just hit another problem: it didn’t support accelerated OpenGL rendering on the X server. That’s quite a problem if no X11 client connected to the Xwayland server can use proper OpenGL and it’s even a problem for KWin as KWin uses QtQuick scenes which use the X server. So KWin itself fails to render accelerated UI. Compositing is not affected as we don’t use Qt for that.

The reason for the problem is that Xwayland expects the Wayland “wl_drm” interface to be present. This interface gets created when binding an EGLDisplay to a Wayland display. In case of KWin the EGLDisplay exists after the Compositor is fully initialized. And again we are in a dependency chain: the Compositor gets created and uses the Workspace class. This class controls the complete startup of the X11 window manager which means it’s highly X11 dependent and requires Xwayland to be present. Again we are in a dependency loop.

Breaking up this dependency loop is quite tricky. The Compositor is too deeply nested into the application to be considered started before creating the QApplication. This means we must be able to create the QApplication before we have an X Server running. This means no usage of xcb QPA plugin. As KWin is going to start a Wayland server anyway, it would be good to get KWin to use the wayland QPA plugin.

This was again quite a challenge. The QtWayland QPA plugin performs blocking roundtrips to the Wayland server in the main thread during startup. But the Wayland server is running in the same thread. So a blocking call to the Wayland server dead locks the server. It’s not possible to move the Wayland server at that point into a thread as one cannot start a QThread prior to having the QCoreApplication created.

The solution I developed for this problem involves creating an own event dispatcher prior to creating the QApplication. So we have the event dispatching for the Wayland server ready to use. Just the event loop is not yet running. This allowed to provide a small patch for QtWayland to run any event dispatcher set before creating the QApplication. A test application in the kwayland repository is also adjusted to make use of it (the test application is also able to start an Xwayland server prior to creating the QApplication).

Unfortunately there is still another issue: QtWayland might call eglInitialize in the main thread which again performs a blocking wayland call. This is a problem I haven’t solved yet and currently just hacked around by disabling OpenGL in Qt (which breaks the QtQuick views).

Being able to use the Wayland QPA just creates a new bunch of problems for KWin. KWin still needs to use X11 and thus needs to create an xcb connection. Just that this wouldn’t help much. KWin on X11 doesn’t init the xcb connection, it’s the xcb plugin in Qt which does it and we only access it via QX11Info. Thus KWin needs to be moved away from the usage. Luckily in many cases we already wrapped the functionality as going through the QPA interface (which QX11Info does internally) is too expensive for our use cases. So the changes are not that invasive. But KWin also uses frameworks like KF5WindowSystem which use QX11Info. Even more those frameworks were properly fixed to perform platform checks and don’t do the X11 specific code if it’s not on platform xcb. But it’s providing important X11 window manager functionality for KWin. For some classes like KWindowInfo and KWindowSystem the fix was trivial: don’t use in KWin. There’s also the more low-level NETRootInfo and NETWinInfo and that’s what KWin should use – usage of the KWindowInfo or KWindowSystem inside KWin can be considered a bug. For some other classes it was already partially possible to be used without the xcb plugin on X11. The classes are only used if we compile with xcb present, so it was possible to add more xcb specific methods which can then be used by KWin even if we do not use platform xcb. The required changes will be part of frameworks 5.8 release.

To summary where we are now: we can start kwin_wayland on platform wayland connecting to a Wayland server started by kwin_wayland, we do not require QX11Info in (most of) KF5WindowSystem and KWin. We are a good step closer to the aim, but still not there. The dependency loop is still in place: Workspace starts the Compositor, the Compositor creates the EGLDisplay which is needed to start Xwayland, which provides X11 which is needed for starting the Workspace.

This means: reorder the startup. We need to be able to start the Compositor prior to the Workspace (which could be interesting for kwin_x11, too, as it could improve persumed startup time). This task was easier than expected. Workspace got split into many modules over the last years and most modules which need to be created prior to creating the Compositor do not depend on Workspace and do not depend on X11. In the few cases where it actually does depend on X11 it was not difficult to delay the X11 specific code till after the X11 connection is created.

With all that in place I was able to delay starting Xwayland to after the Compositor is created and Xwayland can provide OpenGL to the connected clients:

This screenshot shows kwin_wayland running on top of Weston connected to an Xwayland server supporting proper OpenGL as can be seen by the output of glxinfo (in the konsole) and glxgears. Also plasmoidviewer just works on top of this X stack.

Of course there is still some work to be done till this is production ready code, but it looks really promising and I hope to have this ready for the KWin 5.3 release. The aim is also to get more and more features changed to use the Wayland functionality instead of the X11 functionality. E.g. for damage event handling it already uses the damage event of a wl_surface instead of creating an X11 damage handle. This makes supporting Wayland clients easier afterwards.

Global Shortcuts and the Lock Screen

With Plasma 5 our lock screen architecture changed significantly. For example we do no longer support screen saver hacks or widgets on top of the locked screen. Both are very unlikely to make a return in future releases. This means that bug reports against the old infrastructure might no longer apply to our current code base. Two weeks ago I went through all bug reports and feature requests to evaluate whether they still apply to our new infrastructure or should be closed.

This is something I do not like to do. I find it extremely sad to close bug reports because they are outdated. Especially if the bugs have been open for several years without any activity. After going through all those reports it is obvious that we offered too many possibilities to configure the screen locker with too few people caring about it. The number of available screen savers was just immense – especially if one considers that there are also 3rd party savers. While it’s easy to install them, there is basically nobody who cared about them. Some are decades old with the devs having moved on years ago.

Apart from that one could also notice that there were important features missing in our lock screen: audio and multimedia control. The problem is obvious: you suspend your notebook while audio is playing, resume in a place where it should not play audio (e.g. classes) and first need to unlock the screen before being able to mute the audio. An unpleasant experience.

The problem here is that media keys are not supposed to work. The lock screen grabs all keyboard input and prevents other applications to get the keyboard input. This includes our global keyboard infrastructure. We cannot just forward all keys to the global keyboard infrastructure as that could be used to create a key logger when the screen is locked. Even more: most short cuts shouldn’t be invokable when the screen is locked, e.g. you don’t want the desktop to switch.

After brooding over it for a few days I had an idea on how to resolve the problem: the lock screen needs to integrate with our globalshortcut handling. When the screen gets locked we get the available shortcuts from the daemon and map them against a white list of allowed shortcuts. Whenever a key is now pressed while the screen is locked, it’s verified against the fetched list. If we have a match the shortcut is invoked. Not all shortcuts are supported, though. The architecture ensures that one cannot abuse the infrastructure to turn it into a key logger. All alphanumeric keys are excluded. In addition it uses as mentioned a white list, which is not configurable, but hard coded on purpose. At the moment we support only a very limited set of global shortcuts: volume keys, brightness keys and media control keys.

Media control keys were also an interesting topic to work on. Our Plasma session didn’t have any global shortcuts for media handling, so there was nothing which our lock screen could do about it. It cannot figure out whether there is a media application running and then invoke an action on it.

So I stepped back and thought about whether there is a better way to solve it in a general way. Plasma supports the mpris2 interface allowing to control any mpris2-enabled media application. What if the mpris2 engine inside Plasma binds the multi media keys? They could forward to the currently running media player, give us a consistent way to interact with media players and allow us to expose it in the lock screen. So now we have by default mapped the media player controls as global shortcut delegating to any media application. If you press the “Play” button it will Play/Pause VLC, no matter whether it’s the active application or not. And in addition also in the lock screen.

Now there was just one problem to solve: pressing shortcuts and not having visual feedback is not that good. On the desktop we show an on-screen-display whenever the volume changes, but as the screen is locked, we cannot see it. So this architecture also needed enhancement. With a few more changes our lock screen is now able to listen to the requests for on-screen-display information and integrate them:
New lock screen with audio info

All just for supporting shortcuts in the lock screen.

Why screen lockers on X11 cannot be secure

Today we released Plasma 5.2 and this new release comes with two fixes for security vulnerabilities in our screen locker implementation. As I found, exploited, reported and fixed these vulnerabilities I decided to put them a little bit into context.

The first vulnerability concerns our QtQuick user interface for the lock screen. Through the Look and Feel package it was possible to send the login information to a remote location. That’s pretty bad but luckily also only a theoretical problem: we have not yet implemented a way to install new Look and Feel packages from the Internet. So we found the issue before any harm was done.

The second vulnerability is more interesting as it is heavily related to the usage of X11 by the screen locker. To put this vulnerability into context I want to discuss screen lockers on X11 in general. In a previous post I explained that a screen locker has two primary tasks:

  1. Blocking input devices, so that an attacker cannot interact with the running session
  2. Blanking the screen to prevent private information being leaked

From the first requirement we can also derive a requirement that no application should get the input events except the lock screen and that the screen gets locked after a defined idle time. And from the second requirement we can derive that no application should have access to any screen content while the screen is being locked.

With these extended requirements we are already getting into areas where we cannot have a secure screen locker on X11. X11 is too old and too insecure to make it possible to fulfill the requirements. Why is that the case?

X11 on a protocol level doesn’t know anything of screen lockers. This means there is no privileged process which acts as the one and only screen locker. No, a screen locker is just an X11 client like any other (remote or local) X11 client connected to the same X server. This means the screen locker can only use the core functionality available to “emulate” screen locking. Also the X server doesn’t know that the screen is locked as it doesn’t understand the concept. If the screen locker can only use core functionality to emulate screen locking then any other client can do the same and prevent the screen locker from locking the screen, can’t it? And yes that is the case: opening a context menu on any window prevents the screen locker from activating.

That’s quite a bummer: any process connected to the X server can block the screen locker. Even more it could fake your screen locker. How hard would that be? Well I asked that question myself and needed about half an hour to implement an application which looks and behaves like the screen locker provided by Plasma 5. This is so trivial that I don’t see a point in not sharing the code:

#include <QGuiApplication>
#include <QQuickView>
#include <QQmlContext>
#include <QScreen>
#include <QStandardPaths>
#include <QtQml>

class Sessions : public QObject
{
    Q_OBJECT
    Q_PROPERTY(bool startNewSessionSupported READ trueVal CONSTANT)
    Q_PROPERTY(bool switchUserSupported READ trueVal CONSTANT)
public:
    explicit Sessions(QObject *parent = 0) : QObject(parent) {}
    bool trueVal() const { return true; }
};

int main(int argc, char **argv)
{
    QGuiApplication app(argc, argv);

    const QString file = QStandardPaths::locate(QStandardPaths::GenericDataLocation,
        QStringLiteral("plasma/look-and-feel/org.kde.breeze.desktop/contents/lockscreen/LockScreen.qml"));

    qmlRegisterType<Sessions>("org.kde.kscreenlocker", 1, 0, "Sessions");
    QQuickView view;
    QQmlContext *c = view.engine()->rootContext();
    c->setContextProperty(QStringLiteral("kscreenlocker_userName"),
                          QStringLiteral("Martin Graesslin"));
    c->setContextProperty(QStringLiteral("kscreenlocker_userImage"), QImage());
    view.setFlags(Qt::BypassWindowManagerHint);
    view.setResizeMode(QQuickView::SizeRootObjectToView);
    view.setSource(QUrl::fromLocalFile(file));
    view.show();
    view.setGeometry(view.screen()->geometry());
    view.setKeyboardGrabEnabled(true);
    view.setMouseGrabEnabled(true);

    return app.exec();
}

#include "main.moc"

This looks like and behaves like the real screen locker, but it isn’t. A user has no chance to recognize that this is not the real locker. Now if it’s that simple to replace the screen locker why should anyone go a complicated way to attack the lock screen? At least I wouldn’t.

And is there nothing which could be done to protect the real locker? Well obviously a good idea is to mark the one and only screen locker as authentic. But how to do that in a secure way on X11? We cannot e.g. show a specific user selected image. This would conflict with another problem with screen lockers on X11: it’s not possible to prevent that other windows grab screen content. So whatever the screen locker displays is also available to all other X11 clients. Also the window manager cannot help like preventing fullscreen windows to open fullscreen as can be seen in the code fragment above: it’s possible to bypass the window manager. Built in feature by X11.

Many of these issues could be considered as non-problematic using the old pragma of “if it runs, it’s trusted”. While I personally disagree, it just doesn’t hold for X11. If only clients of one user were connected to the X server one could say so. But X11 allows clients from other users and even remote clients. And this opens a complete new problem scope. Whenever you use ssh -X you open up your local system to remote attack vectors. If you don’t control the remote side it could mean that the client you start is modified in a way to prevent your screen from locking or to install a fake locker. I know that network transparency is a feature many users love, but it’s just a security night mare. Don’t use it!

Overall we see that attacking a screen locker or preventing that it opens up is really trivial on X11. That’s an inherent problem on the architecture and no implementation can solve them, no matter what the authors tell how secure it is. Compared to these basic attack vectors the vulnerability I found is rather obscure and it takes a considerable amount of understanding how X11 works.

Nevertheless we fixed the issue. And interestingly I chose to use the technology which will solve all those problems: Wayland. While we don’t use Wayland as the windowing system we use a custom domain-specific Wayland-based protocol for the communication between the parts our screen locker architecture. This uses the new libraries developed for later usage in kwin_wayland.

As we are speaking of Wayland: how will Wayland improve the situation? In the case of Plasma the screen locker daemon will be moved from ksmserver to kwin, so that the compositor has more control over it. Screen locking is a dedicated mode supported by the compositor. Whether a context menu is open or not doesn’t matter any more, the screen can be locked. Also the compositor controls input events. If it has the knowledge that the screen is locked, it can ensure that no input goes to the wrong client. Last but not least the compositor controls taking screenshots and thus can prevent that clients can grab the output of the lock screen.

KWin on speed

With the 5.2 release basically done, I decided to do some performance investigation and optimizations on KWin last week. From time to time I’m running KWin through valgrind’s callgrind tool to see whether we have some expensive code paths. So far I hadn’t done that for the 5.x series. Now after the switch to kdecoration2 I was really interested in the results as in the past rendering the decoration used to be a bottle neck during our compositing rendering loop.

Unfortunately callgrind doesn’t give us a good look on the performance of KWin as it neither includes GPU times nor roundtrips to the X server. Nevertheless it gives us a good look on our own CPU usage. I was rather surprised by the result as I didn’t find anything which looked bad. Nevertheless I was able to slightly optimize one method which is called whenever the X11 stacking order is changed by improving an internal algorithm which didn’t scope well with the larger than expected number of child windows of the root window.

But callgrind output wasn’t the only performance relevant thing I looked into. I investigated a really interesting bug report about the screen freezing for a short time when a new window opened. While I wasn’t able to reproduce the issue as is, I was able to reproduce a small freeze whenever a Qt 5 application opened. Interestingly only with a Qt 5 application. So I ran the same application in a Qt 4 and Qt 5 variant and only in the latter I got a freeze. Investigation showed pretty quick that KWin is not to blame, for one I got the freeze before KWin started to manage a window for it and I was able to reproduce with different window managers. With the help of xtrace I finally found the culprit and we found the appropriate bug report on Qt side. Also our KDE domain experts started to look into the issue on Qt side.

But still others were able get a small freeze whenever KWin started to manage a window. And in deed further investigation showed that the method handling the managing of a new window can take some time and can cause the compositor to drop frames. Ideally this would be solved by moving the compositor into a dedicated rendering thread but that’s quite a lot of work and might not help in that case as KWin’s main thread grabs the XServer while managing a window. So the better solution was to investigate why the method takes so long. To not drop any frames the method may not take longer than 16 msec, the shorter the better.

While managing a window KWin needs to read quite a lot of properties. Most of them are nicely read in a non-blocking way through the KWindowSystem framework, but some properties are also KWin internal and read in a blocking way. Most expensive was reading the icons which was triggering several round trips especially if the window did not specify the icons in a NETWM compliant way. This could easily cause a delay of 50 to 100 msec during managing a window. Overall the method could trigger up to 14 round trips to the X server which were not needed at all in the case of KWin. Our KWindowSystem framework got an adjustment to prevent the roundtrips if the user of the KWindowSystem framework has all required information already fetched. The result is that reading the icons is now significantly below one msec. For other roundtrip causing methods I introduced two new methods: one to perform the request, one to later read the result. This allowed to remove another set of roundtrips. My measurements showed that each roundtrip takes about half a millisecond on my system. Half an msec here, half an msec there easily adds. Unfortunately there are still some XLib calls (one to read motif hints and one to read WM_SIZE_HINTS) which ideally would get ported and as long as they are not ported delay the managing of a new Client.

Nevertheless this shows quite some nice improvements for our development version which will become 5.3 in a few months. Of course all of that would not have been possible without the switch from XLib to XCB.

Locking the screen before system suspends

Our Plasma workspace has offered the feature to lock the screen when resuming from suspend for a long time. Ideally the screen gets locked right before the system goes to suspend to ensure that the screen is properly locked when the system wakes up.

The process was controlled by powerdevil: when powerdevil decided that it should suspend the system, it invoked the lock screen to get the screen locked. But this has drawbacks. For example the screen locker doesn’t know that the system is going to suspend. Neither does powerdevil know when the screen is fully locked (the lock process includes multiple stages and the actual lock is hold before the screen gets blanked). Also it only works with powerdevil, if one suspends the session in a different way (e.g. through systemctl suspend) the screen doesn’t get locked. The worst drawback was that sometimes it was still possible that the system woke up and expose the session for a split second as the screen locker has no way to indicate to the system to wait with the suspend till everything is settled.

With the upcoming Plasma 5.2 release this will significantly improve by leveraging functionality provided by logind. The lock screen on resume functionality is moved from powerdevil into the screen locker. This already means that the screen locker has a more complete picture of what is going on and allowed us to streamline the settings: whether the screen locks on resume is now found in the same configuration module as all other screen locker related settings.

The most important change is that we are now able to ensure that the screen is locked before going to suspend. For this the screen locker holds an inhibitor lock. When the screen is going to suspend, the screen locker is notified by logind, can start the lock process and remove the inhibitor lock once the screen is properly locked. This means under normal condition the screen will be locked before going to suspend (of course a timeout in logind could be hit if everything takes too long or if the screen cannot be locked).

Obviously this will only work if the system is suspended through logind, which powerdevil does if logind is available. So our users who use logind get an improved experience. For our users who do not yet use logind nothing changes: powerdevil notifies the screen locker that it’s going to suspend and the screen locker starts the lock process. This obviously has the same limitations as described above.