Posted on

Unsetting QT_QPA_PLATFORM environment variable by default

Since the introduction of the Plasma/Wayland session we set the QT_QPA_PLATFORM variable to wayland by default. After a long and hard discussion we decided to no longer do this with Plasma 5.13. This was a hard decision and unliked by everyone involved.

The environment variable forced Qt applications to use the wayland QPA platform plugin. This showed a problem which is difficult to address: if Qt does not have the wayland QPA platform plugin the application just doesn’t start. If you start through the console, the application will tell you:

This application failed to start because it could not find or load the Qt platform plugin "wayland"
in "".

Available platform plugins are: minimal, xcb.

Reinstalling the application may fix this problem.
Aborted (core dumped)

That is not really a useful information and does not tell the user what to do. Neither does it tell the user where the actual problem is and how to solve it. As mentioned when using a graphical launcher it’s worse as the app just doesn’t start without any feedback.

But how can it happen that the qpa platform plugin is missing although Plasma itself happily uses it? The problem is that application installed outside of the system bundle their own Qt and Qt does not (yet) include QtWayland QPA platform plugin. This affects proprietary applications, FLOSS applications bundled as appimages, FLOSS applications bundled as flatpaks and not distributed by KDE and even the Qt installer itself. In my opinion this is a showstopper for running a Wayland session.

The best solution is for Qt including the QPA platform plugin and having a proper auto-detection based on XDG_SESSION_TYPE. The situation will improve with Qt 5.11, but it doesn’t really help as the Qt LTS versions will continue to face the problem.

For now we implemented a change in Plasma 5.13 so that we don’t need to set the env variable any more. Plasma is able to select the appropriate platform plugin based on XDG_SESSION_TYPE environment variable. Non-Plasma processes will use the default platform plugin. With Qt < 5.11 this is xcb, with Qt 5.11 this will most likely change to wayland. KDE’s flatpak applications pick Wayland by default in a Wayland session and are unaffected by the change.

What is really sad about the change is that the Wayland qpa platform plugin gets less testing now. So we would like to ask our users to continue testing application with the Wayland platform plugin by setting the env variable manually or specifying –platform wayland when starting an application.

Posted on

Plasma/Wayland and NVIDIA – 2017 edition

More than a year ago I elaborated whether KWin should or should not add support for NVIDIA’s proprietary Wayland solution. I think it is time to look at the situation of Plasma/Wayland and NVIDIA again. In case you haven’t read my previous blog post on that topic I recommend to read it as I use it as the base for this blog post.

Compared to a year ago not much has changed: NVIDIA still does not support the standard Linux solution gbm, which is supported by all vendors and nowadays even going to enter the mobile space. E.g. the purism phone is going to have a standard graphics stack with gbm. So no additional code required. But NVIDIA doesn’t support gbm. Instead it has a proprietary implementation called EGLStreams, which no other vendor implements. Due to that Plasma/Wayland cannot support OpenGL for NVIDIA users.

But current KWin master branch (what will become 5.12) supports automatical fallback to Qpainter compositing in case OpenGL fails. With that it should be possible to run Plasma/Wayland on NVIDIA hardware. Granted it won’t be accelerated, but as explained above that is only NVIDIA’S fault. Also I have not tested this yet. It might be that NVIDIA doesn’t support dumb buffers on DRM.

In my blog post from last year I elaborated whether KWin should add support for NVIDIA’S proprietary solution. I put out a requirement that the patches for Weston need to be merged first:

For KWin such patches do not exist and we have no plans to develop such an adaption as long as the patches are not merged into Weston. Even if there would be patches, we would not merge them as long as they are not merged into Weston.

A year passed and nothing happened. I don’t expect anything further will happen: the patches won’t be merged into Weston. Given that it doesn’t make any sense to make this a requirement for KWin. If we hold up the requirement it would be like saying we would never accept it.

Last year I hoped that XDC would find a solution for the problem:

We do hope that at XDC the discussions will have a positive end and that there will be only one implementation.

And it looked positive: a new universal buffer allocation library got announced. I hoped that this would solve the situation pretty quickly. The new library would get implemented, Mesa and NVIDIA add support for it, we slightly adjust our code and drop gbm. Everybody happy.

But now another XDC passed and from what I understand not much process was made. The library is not ready and no driver uses it. We waited one year for it, I doubt waiting another year will change the situation. Realistically I think it will take another two to three years for the library to get implemented if at all. Then it will take about a year for Mesa gaining support and shipping a release. Yet another year for distributions shipping with that version so that we can depend on it without people killing us. So maybe in five years there will be a replacement. I don’t think we can wait that long.

So does this mean KWin will gain support for EGLStreams? Certainly not! I do not think that the KDE community should spend any time to support NVIDIA’s proprietary solution! We are a free software community and we should not implement code which only benefits proprietary non-free solutions. There are way more free things to do to improve Wayland without having to write code for proprietary solutions.

Also there is another aspect which I did not consider in my blog post last year: XWayland. XWayland also uses gbm and does not have support for NVIDIA’s proprietary solution. Due to that one does not have OpenGL for any legacy X application. This probably includes most games, but currently also browsers such as Firefox and Chromium. I don’t think that this would give a satisfying experience to NVIDIA users. And it is also hardly better than rendering everything through the QPainter compositor. This is another reason for the KDE Community to not spend any time on implementing support for EGLStreams.

But I think there is lots NVIDIA could do. Today I would accept a patch for EGLStreams in KWin if NVIDIA provides it. I would not be happy about it, but I would not veto it. If it is well implemented and doesn’t introduce problems for the gbm implementation I would not really have an argument against it. But I expect NVIDIA to do it. I don’t want a contribution from a non-NVIDIA developer. This mess was created by NVIDIA, NVIDIA needs to fix it.

Similar I think that NVIDIA should adjust XWayland. I understand that NVIDIA is not happy with the design of XWayland, but nevertheless they should make it work. Their users pay quite some money for the hardware. I think they have a right to demand from NVIDIA to fix this situation. Ideal would of course be NVIDIA adopting gbm. But as that seems unlikely, I think it is the duty of NVIDIA to provide patches for their users.

Posted on

Plasma Wayland and Qt 5.9 and beyond

As you might know Qt 5.8 created challenging problems for our Wayland session and threw our efforts back quite a bit. In this post I want to discuss the actual problems it created, how we are addressing them and looking into the future.

How our integration used to work

Our integration uses additional Wayland protocols. We have a protocol for server side window decorations which we use in our Plasma integration plugin to inform KWin whether the window should have a decoration or not. We have a protocol for client provided shadows which is e.g. used by our widget style Breeze to add shadows to the context menus. We have a protocol for the desktop shell, so that it can mark windows as desktop, panel, auto hiding panel, position the window, etc. Also we have a few protocols for interacting with our effect system, e.g. sliding popups, blur behind.

To use these protocols we need to interact with Qt in a low level way. We use the native interface in the Qt Platform Abstraction to get a wl_surface pointer for the QWindow. In order to not have to keep this simple for our applications our KWayland::Client API provides an API point for it: Surface::fromWindow(QWindow*) -> Surface*.

But when exactly to inject our own integration? We found a very handy way which worked much better than what we had used in the past for X11 (and based on that also transitioned X11 code to use it). Qt emits an event once it has created the native platform surface (in case of Wayland the wl_surface*) for a QWindow. Verbatim quote of the documentation:

The QPlatformSurfaceEvent class is used to notify about native platform surface events.

Platform window events are synchronously sent to windows and offscreen surfaces when their underlying native surfaces are created or are about to be destroyed.

Applications can respond to these events to know when the underlying platform surface exists.

Awesome! We get an event when the surface is created and when it gets destroyed. This made it very simple to create the integration and what is really important for us is that we get this event before the window is shown. So we can prepare everything so that KWin gets a good state.

And the KWin side was to a large part implemented on assumptions on how the sequence will work. We first get the surface, then the (xdg) shell surface, then the integration bits. Sure it would be nice if KWin handled also other sequences, but as the only implementation of this is Qt it doesn’t make sense to really care about it. We know that it was not perfect, we even had the test cases for it, which expect failed.

What broke with Qt 5.8?

In Qt 5.8 our complete integration broke. When the platform surface created event was emitted the wl_surface was not created, see QTBUG 58423. This is in my humble opinion a clear violation of the documented behavior and thus a breakage of the stability guarantees Qt provides, but others might disagree. After some discussion, trying out patches by those who had a Qt 5.8 build we had a patch for Qt which made things mostly work together with a patch to KWin. But at the time we had the patch ready Qt 5.8 was already declared end of life with no prospect of a Qt 5.8.1 bug fix release. For our Wayland session it was just impossible to get Qt 5.8 compatible again. All we could do was to advise distributions to not combine Qt 5.8 with the Wayland session. From our side it was fine for distros to ship Qt 5.8, but if they do they should make it impossible to install the Wayland session. The state was just too broken.

Qt 5.9

With Qt 5.9 the situation looks better. The required patch is merged, it’s an LTS release and we already had the first bug fix release. Qt wants to create more bug fix releases for it and this allows us to use it as a new target for integration. But still the situation is not as good as it used to be. If you currently use our Wayland session with Qt 5.9 you will still see quite some rough corners compared to where we were with Qt 5.7.

The main problem for us is that the platform surface regression was not the only change affecting us. Pre Qt 5.7 a wl_surface lived as long as a QWindow. Now the wl_surface gets destroyed whenever the window gets hidden and a new one created on every show. Unfortunately without a platform surface created event. This means our integration breaks as soon as a window gets hidden. E.g. after closing KRunner the integration for KRunner is broken.

We tried to address this problem in various places, but it is challenging. On the show event we don’t have the wl_surface yet (too early), on the expose event the window is already mapped (too late). This creates problems for KWin which is not prepared for the integration bits to hit when the window is mapped. Our protocols were designed with the platform surface created event semantics. For example in KRunner we face an issue with the integration. KRunner is a panel, which accepts focus and allows windows to go below, also it positions manually. Now when it gets re-shown KWin doesn’t know that this window is supposed to position itself and positions it. Now we get the request to put it as panel, KWin adjusts and moves maximized windows around. And then we get the request that the panel allows windows to go below. KWin shuffles the windows again as the maximized area changed again.

This is a rather tricky situation as we cannot really do something about it. If the window is mapped it’s too late. Even if we improve our API to handle the situation better it will be too late.

There are two possibilities to handle this: Qt stops to destroy the surface or sends a platform surface created event when it recreates the surface. The latter would be my personally preferred solution as this would match the documentation again and allow us to just use the one event handler.

Other regression

The situation around the changed behavior in Qt 5.8 caused a few steps back. Our code needed to be adjusted and that sometimes caused issues. We had a few regressions which also affected the compositor, so the stability of the whole system suffered. These issues are luckily investigated and fixed. But there are still bugs lurking in the system.

For me personally the most annoying bug is a crash in Qt which affects the auto completion of kdevelop. This makes hacking in a Wayland session rather difficult. I’m running currently a patched KWin which disables the virtual keyboard integration to not hit this issue.

A huge problem is that context menus are not marked as transient windows. This means that the Wayland compositor does not know that it is a menu and positions the menu anywhere on the screen. It gives the system a very unfinished touch.

If KRunner is closed through the escape key, the key starts to repeat on the window constantly and due to that it is not possible to open KRunner again. Similar if you start an application in Kickoff through the enter key, when opening Kickoff again it automatically launches the currently selected item. This again makes it very difficult to use the session and gives the whole system an unfinished look. We are working on a workaround for this issue in the server.

Towards the future

Qt 5.9 is here to stay and that’s what we have to use as integration target. Given that Qt 5.9 and Qt 5.7 behave very differently it will become difficult for us to maintain support for both. My suggestion is that we drop support for Qt 5.7 and require Qt 5.9 for the Wayland session. In addition there is hope that we can improve the integration. Marco and David have been working on adding support for XDG Shell unstable v6 which is already supported by Qt and makes it easier to integrate with. Once this landed in KWin Qt will be switched from wl_shell to xdg shell. This will improve the situation for us quite a bit as we then have one code path for both Qt and GTK applications.

How to prevent such regressions in the future

The change of behavior in Qt 5.8 threw our Wayland efforts back a few months. This is something we communicated to Qt quite early and it’s something which worries me a lot. We cannot spend time on changing our integration every time Qt releases a new version. Given that we need to look into how to prevent that such a situation happens again.

I hope that we can improve the integration on the testing front between Qt and KDE. We have a huge test suite which can find regressions in Qt. If Qt would run KDE’s tests during the integration phase Qt would notice regressions before they hit the code base. Given that all our tests are free software it should be possible for Qt to integrate them.

But also the other side would be interesting: if we could get the latest Qt into our CI system we could also discover breakages early. We have now a new docker based CI system which allows running multiple builds of the same change (e.g. Plasma gets build on openSUSE tumbleweed and on FreeBSD) – an image with a daily or weekly Qt snapshot could help us and Qt a lot to detect breakages early.

I also hope in openQA which allows to test the full operating system. This would spot regressions like the misplaced context menus even if KWin’s own auto tests would not spot them (KWin doesn’t care about Qt there, only about the Wayland and X11 protocols). There we might need to invest some work to make sure that KWin/Wayland can be properly run in the openQA tests.

I hope that our Plasma devs can discuss this in more detail with Qt devs during Akademy in person. Unfortunately I cannot be at Akademy this year 🙁 so I cannot discuss in person.

Last but not least it is important that developers test. It would help a lot if the developers working on QtWayland test their changes in a running Plasma Wayland session. We are now overall in a state that the session is suited for hacking on. I do all my Wayland hacking in a Wayland session, experiencing all the glitches like kdevelop crashing.

Of course you might wonder what about us? Shouldn’t we KDE devs also test against the latest Qt? For me personally that is not the case. I’m working on the server side and not on the client side. I’m also not testing the latest GTK for example. Nevertheless I tried to use Qt 5.9 before it got released. Used the installer, spent a day to compile everything on top of it it, just to notice that it doesn’t have QtWayland and won’t get it. I didn’t give up that easily. So I tried to compile QtWayland myself. But when I tried to use it, it turned out to not have any keyboard support, because qtbase was compiled without support for xbkcommon. At that point I gave up. Not having QtWayland is one thing, but not being able to use keyboard is another, it’s rather pointless. The only other option is to compile Qt, but that is hardly an option as it’s really difficult to compile an actually working Qt with all components. The last times I tried, I failed, wasting days compiling. If there were usable weekly images for Qt I would be happy to try it, but of course only with a properly compiled and included QtWayland.

Posted on

Virtual Keyboard on Lock Screen

As part of our efforts to improve out of box experience for touch screens I’m pleased to announce that Plasma 5.10 will provide integration for virtual keyboard.

Plasma 5.10 finally integrates qtvirtualkeyboard directly in the lockscreen component of the look and feel package. Support for Qt’s virtual keyboard is not new in Plasma in general, it is already fully integrated into the Plasma Wayland session. This change, though, can be used on both Wayland and X11.

Greeter with new Virtual Keyboard button

The lockscreen got extended by a new button next to the keyboard layout switcher: “Virtual Keyboard”. When clicking this button the virtual keyboard gets enabled and one can use it (with both touch and mouse) to enter the password and unlock the screen.

Virtual Keyboard in the lock screen

As a nice side effect this finally enables the possibility to enter east Asian languages in the lock screen. So far we had to actively disable input methods on the lockscreen to prevent that the screen cannot be unlocked. As a side effect this meant that only latin characters can be used in the password. I hope that the new virtual keyboard can help here by providing a better experience.

As a note to our touch screen users. It’s possible to use Qt’s virtual keyboard for all Qt application. One just needs to specify the env variable QT_IM_MODULE=qtvirtualkeyboard. When entering text Qt pops up the virtual keyboard as an additional window. This is only needed in an X11 session. In a Wayland session KWin provides the integration and Qt apps (and other apps) get the virtual keyboard through the Wayland text input protocol.

Posted on

Wayland improvements since Plasma 5.8 release

Two weeks have passed since the Plasma 5.8 release and our Wayland efforts have seen quite some improvements. Some changes went into Plasma 5.8 as bug fixes, some changes are only available in master for the next release. With this blog post I want to highlight what we have improved since Plasma 5.8.

Resize only borders

KWin’s server side decorations have a feature that one can resize the window in the shadow area. With the Breeze window decoration this is available if one uses the border size “No Side Borders” or “No Borders”. For Wayland we just had to adjust the input area of a window slightly and honor it when evaluating the mouse pointer movements.

Global Shortcut handling

We found a few bugs related to global shortcut triggering. There is some unexpected behavior for shortcut triggering in xkbcommon, which will be addressed in the next release by adding new API. For now we had to workaround it to support some shortcuts which no longer triggered. Of course for every kind of shortcut which did not trigger we added a test case so we can also in future ensure that this works once the new xkbcommon release is available. At the moment we are not aware of any not working global shortcuts on Wayland. If you hit one, please report a bug.

Support for Keyboard LEDs through libinput

KWin did not enable the LEDs for num lock, caps lock, etc. This was mostly because I don’t have any keyboard which has such LEDs – neither my desktop keyboard nor my two notebooks have any LEDs. So I just didn’t notice that this was missing. Once we got the bug report we looked into adding this. I want to take this as an example of the “obvious bug” one doesn’t report because it’s so obvious. But if one doesn’t have such hardware it’s not so obvious any more.

Relative pointer support

A feature we added for Plasma 5.9 is support for the relative pointer protocol.

Relative Pointer Events

The protocol is implemented in KWayland 5.28 and KWin is adjusted to support the relative pointer events as can be seen in the screenshot of the input debug console. This is a rather important protocol to support games on Wayland. We also plan to add pointer confinement for Plasma 5.9.

Move windows through the widget style

Our widget styles Breeze and Oxygen have a feature to move the window when clicking in empty areas. This is a feature which needs to interact with the windowing system directly as Qt doesn’t provide an abstraction for it. On X11 it uses the NETRootInfo::moveResizeRequest, on Wayland support for triggering a window move is built into the core protocol. But so far we were not able to provide the feature on Wayland as we just didn’t have enough information from QtWayland. For example we lacked access to the wl_shell_surface on which we have to trigger the move. So some time ago I added support to QtWayland that we can access the wl_shell_surface through the native interface. Now about a year later we can start to use it. To support this feature we need to create an own wl_seat and wl_pointer object and track the serial of pointer button press. This we can then pass to the move request on the ShellSurface. The change is not KWin specific at all and will work on all Wayland compositors.

Color scheme sync to decoration

A new feature we added in KWin 5.0 is the possibility to synchronize the color scheme from the window into the window decoration and the context menu on the decoration. On X11 this works through a property which our KStyle library sets. This was the best we had back in the early days of the 5.x series as Qt didn’t expose enough information. It has the disadvantage that the sync only works with QWidget based applications and only with widget styles inheriting KStyle. For Plasma 5.9 we improved that and brought the relevant code into plasma-integration. The restriction to QWidget is gone and it works now with all kind of windows by listening to the QPlatformSurfaceEvent. This very useful event which got added in Qt 5.5. It informs us when a native window is created for a QWindow. Thus we can add our own X11 properties on the native window directly after creation and before the window is mapped.

Custom color scheme support

While adjusting this code for X11 we also added the relevant bits for Wayland. We use the Qt Surface Extension protocol to pass a property to the server. That’s a small and neat addition the Qt devs did to allow communication between a Qt based client and a Qt based Wayland compositor. As one can see in the screenshot the color scheme now updates also for Wayland applications.

Window icons

Window icon handling in Wayland is different to X11. On X11 the icons are passed as pixmaps. That has a few disadvantages nowadays because the icons provided on the window might not have a high enough resolution to work well on high-dpi systems. The icon from the icon-theme though provides higher resolution. On Wayland there is no way to pass window icons around and the compositor takes the icon from the desktop file of the application. This works well unless we don’t have a desktop file. For such windows we now started to use a generic Wayland icon as the fallback, just like we use a generic X icon as fallback for X11 windows which don’t have an icon.

Proper icons for X windows in task manager

That’s an icon which one might have noticed when using a Plasma Wayland session as every Xwayland window only had the generic X icon in the task manager. The communication between KWin and the task manager also passes the icon name around and not pixmap data. This works well for everything which isn’t Xwayland where we normally just don’t have the name. For Plasma 5.9 we addressed this problem and extended our protocol to request pixmap data for a window icon which doesn’t have a name. Thus we are now able to also support Xwayland windows, which increases the useability of the system quite a lot.

Multi screen effect improvements

On Wayland several of our effects broke in a multi-screen setup. This is because rendering is different. On X11 all screens are rendered together in one rendering pass and we have one OpenGL window to render to. On Wayland we have one OpenGL window per screen and have one rendering pass per screen. That’s something our effects didn’t handle well and resulted in rendering issues. For Plasma 5.9 these issues are finally resolved.

Wobbly windows

One of the affected effects is Wobbly windows. A rather important effect given that this blog is subtitled “From the land of wobbly windows”. We experienced that in a multi-screen setup the effect was only active on one screen. If the window got moved to the other screen it completely vanished.

I was quite certain that this is not a problem with the effect itself, but rather with the way how we render. As we also saw other effects having rendering issues in multi-screen setups I was quite optimistic that fixing wobbly would fix many effects.

The investigation showed that the problem in fact was an incorrect area passed to glScissor due to the general changes in rendering explained above. Rendering on other screens got clipped away. With the proper change we got wobbly working and several other effects (Present Windows, Desktop Grid, Alt+Tab for example) without having to touch the effects at all.

Screenshot

With that knowledge in place we looked into fixing other effects. E.g. the screenshot effect which allows to save a screenshot in the tmp directory. A few example of screenshots taken with this effect can be seen in this blog post. The problem with this effect was that when taking a fullscreen shot over all screens only one got captured. The assumption here was that our glBlitFramebuffer code needs adjustment to be per output and with that we can now screenshot every screen individually or all screens combined.

Multi-screen shot with blur

Blur and Background Contrast

Related to that are the blur and background contrast effect as they also interact with the frame buffer, though don’t use the glBlitFramebuffer extension. With those effects one of the biggest problems was that the viewport got restored to a wrong value after unbinding the frame buffer object. Due to that the rendering got screwed up and we had severe rendering issues with blur on multi screen. These issues are now fixed as can be seen in the screenshot above: both screens are rendered correctly even with blur enable.

Panel improvements

Plasma’s panel got some improvements for Plasma 5.9. This started from bug reports about windows can cover not working and also auto-hide not working. Another example that it is important to report bugs.

Auto hiding panel

On X11 auto hiding panels use a custom protocol with KWin to indicate that they want to be restored if the mouse cursor touches the screen edge. It uses low level X11 code thus we also need a low level Wayland protocol for it. We extended our plasma shell protocol to expose auto hiding state and implemented it in both KWin and Plasma.

Search in widget explorer

We had a bug report that search in the widget explorer doesn’t work. The investigation showed that the reason for that is that the widget explorer is a panel window and we designed panels on Wayland so that they don’t take any keyboard focus. This is correct for the normal panel, but not for this special panel. We adjusted our protocol to provide an additional hint that the panel takes focus and implemented this in kwayland-integration in a way that the widget explorer gains focus without any adjustments to it.

KRunner as a panel

Of course there are more potential users for this new feature. One being KRunner. Once we had the code in place we decided to make KRunner a Panel on Wayland which brings us quite some improvements like it will be above other windows and on all desktops.

Posted on

Animating auto-hiding panels

With Plasma 5 a change regarding auto-hiding panels was introduced. The complete interaction was moved from Plasma to KWin. This was an idea which we had in mind for a long time. The main idea is to have only one process to reserve the interaction with the screen edges (which is needed to show the panel when hidden) and to prevent conflicts there. Also it allowed to have only one place for providing the hint that there is the panel.

On an implementation level that uses an x11-property protocol between KWin and Plasma to indicate when the panel should be hidden. KWin then does the interaction to hide it and to show on screen edge activation.

Unfortunately from a visual perspective this created a regression. In Plasma 4 the auto-hiding panel was animated with our Sliding-Popups effect, but in Plasma 5 this doesn’t work any more. The reason for that is that our effect system can only animate when a window gets mapped and unmapped. With the new protocol the window doesn’t get unmapped when hidden and not mapped when shown, so our effects are not able to react on it. Technically our Client object is kept instead of being released on unmapped.

Thanks to Wayland this will be working again starting with Plasma 5.8. For Wayland windows KWin keeps the object around when a window gets unmapped and uses the same object when it gets shown again. KWin keeps more state for Wayland windows than for X11 windows. But this also means that our animations are not working for Wayland windows. Last week I addressed this and extended the internal effects API to also support hiding/showing again of Wayland windows. As the effects API does not differentiate between Wayland and X11 windows this change also enables the auto-hiding panel animation. All that was needed was emitting two signals at the right place.

Here Wayland shows another strength: not only does it help to bring features to X11 it also allows us to automate the testing. With a pure X11 system we would have had a hard time to properly auto-test this. But for Wayland we have a nice isolated integration test-framework which allowed to add a test-case for auto-hiding panels.

Posted on

Wayland in Plasma 5.7

Last week we released the beta version of Plasma 5.7 which means we know what this release will have for better Wayland support. First of all I need to mention what didn’t make it: unfortunately I missed the freeze of Frameworks 5.23 to land support for xdg-shell. I have a working implementation, but was not yet satisfied with the API. This is a difficult interface to provide an API for due to the unstable nature of the interface. Due to lack of xdg-shell support GTK applications are still going to use X11 on Wayland (like the Firefox window I’m just typing this blog post in).

In the past I already blogged about a few new features in 5.7 like the improved task manager for Wayland, the virtual keyboard support, sub-surface support and improved input device support. So in this blog post I want to focus on a different topic: quality.

For Plasma 5.7 my aim was to get the Plasma session into a state that I can use it as my primary system. And since last week I have not started into an X11 session any more. This means that we needed to get the whole system stable enough to have neither KWin nor applications crash due to Wayland. Given that our Wayland code is quite a fair amount of new code, changing lots of assumptions there are of course bugs to be expected. We still have code which calls into X11 unconditionally, we have things which are not implemented correctly and of course we do stupid mistakes. So for Plasma 5.7 the task was to find these issues and fix one for one.

For Wayland it’s much easier for us to test. KWayland – our framework for Wayland support – is developed in a test driven approach making it possible to create test cases for every problem. They expose the problems and verify that they are fixed and as regression tests ensure that they won’t hit us again. Over the last release cycle we added several thousand lines of test code in KWayland alone.

Finding those issues is not always easy. If KWin crashes we don’t have DrKonqi like normally, it doesn’t work for Wayland (tries to connect to a display server, but that just crashed). What I saw on my Wayland test device was that KWin sometimes randomly crashed – more often when I interacted with X11 windows. But when attaching gdb to KWin it didn’t crash. But once I caught it: it turned out to be an error in KWin in the handling of Xwayland windows. There are two possible code paths it can take and one was with a mistake. Due to running through a debugger it was more likely to take the correct one. So yeah it’s not always easy.

With that problem gone we were able to find a few more and fix also some bugs which caused windows to quit. Unfortunately some of this fixes had to go into KWayland after the 5.23 release. This means the frameworks version used with Plasma 5.7 is not going to have all fixes. If you want to give Plasma/Wayland a try I recommend to not just wait for Plasma 5.7 but also for frameworks 5.24.

This week I will be at the openSUSE conference in Nuremberg, where I will also give a talk about how Wayland helps us to improve our quality and our workflows. I’ll do another blog post about the content of that presentation – don’t want to spoil 😉 Though if you follow our development you are already aware of it. Thanks to openSUSE to give me the possibility to present at the conference and thanks to KDE e.V. for the support to go there.

I can hear you asking now the question of all questions: “When will it be ready?” I think that I am not objective enough to answer the question or to say that it is ready. I’m too close to the code and might just omit important problems because I don’t see them. Thus I cannot say that it is ready. It depends on your workflow and whether that workflow is already fully implemented. This is something only you can know.

Last week KDE had a very important developer sprint (where I did not participate) and is currently running a fundraising campaign for this sprint. We need the money to send our developers to such meetings or to a conference like openSUSE conf where I will be this week. At the moment just 107 people have participated and donated. This is something which makes me sad. I see the statistics for my blog posts and know that this one will have at least 1000 direct hits. In addition there are people reading planetkde and not directly my blog. We are trying to raise 24000 EUR and please do the math yourself to see how close we would be to it if everybody would donate just 10 EUR.

Posted on

A task manager for the Plasma Wayland session

Plasma 5.7 will ship with a new taskmanager library. One of the reasons to implement a new taskmanager library is the port to Wayland. Our old code base was heavily relying on X11 and didn’t support the concept of multiple windowing systems. You can read more on that in Eike’s blog post about the new task manager. In this blog post I want to focus a little bit on the Wayland side of a task manager.

Architectural difference to X11

On X11 every application is able to read every aspect of every other window. This allows to implement a task manager. The task manager can get notified when a window is added, when a window is removed and install event filters for the changes of the window properties. That’s the basics for a task manager.

On Wayland the world is different. An application is not able to see the windows of other applications. This means by default Plasma is not able to see what windows exists, the taskmanager would be empty. The only application which knows about the existing windows is the Wayland compositor.

Merging Desktop Shell and Wayland compositor?

A possible solution to this problem could be to merge the desktop shell and the compositor into one process. This is the architecture used in e.g. Unity and GNOME Shell on X11.

But such an architecture has also disadvantages. It violates the concept of separating concerns and of “doing one thing and do it right”. We have an excellent task manager, an excellent window manager and an excellent desktop shell.

Also it would make code sharing between platforms more difficult. A task manager for Wayland would be very different from a task manager on X11. All the business logic and presentation should be the same. No matter which windowing system is used.

By merging the code bases together we would also make the code less reusable. One couldn’t use KWin/Wayland in another desktop environment (e.g. LXQt) nor could one use Plasma/Wayland with another compositor (e.g. sway). While we of course recommend to use KWin as the Wayland (and X11) compositor we don’t enforce it and also don’t want to enforce it.

Protocols, protocols

So a different solution is needed and that is done through a custom Wayland protocol which is called org_kde_plasma_windowmanagement. We have a few interfaces prefixed with Plasma. This does not mean that it can only be used by Plasma. It just means that the interface was first designed and developed for Plasma’s needs. We are happy to share the interface with other desktop environments.

The protocol is provided by KWin and it announces whenever a new window gets created. Once a new window gets created a bound client can bind an org_kde_plasma_window interface and get notified about all the states of the window.

The protocol exposes pretty much all the information KWin has about the window and it gets updated automatically whenever the state in KWin changes. In addition the protocol has requests the task manager needs, like “close the window”, “minimize it”, “maximize it”, etc. etc. KWin also listens to these requests and honors them.

Although the protocol is added to Wayland, it is windowing system agnostic. The created Plasma Window does not expose the actual windowing system resource (after all on Wayland another client cannot get access to it). KWin exposes a Plasma Window for both X11 managed windows and Wayland windows. By that the task manager is able to manage tasks from multiple windowing systems without knowing that it does so.

Evolution of the protocol

We added the Plasma window protocol initially for the work on the Plasma phone about a year ago. Also on the Plasma phone we had the use case of switching windows and using the X11 based task manager was just no option on a Wayland only device 😉

While the protocol worked fine for our initial needs, it was not yet sufficient for the use in the desktop task manager. So over the last weeks we introduced further support and improved the API to make it work more reliable. Also quite some changes went into KWin to ensure that only “interesting” windows are exposed. E.g. a context menu should not be shown in the task manager.

Our KWayland library of course has support for this protocol through the classes PlasmaWindowManagement and PlasmaWindow on client side. In addition there is a model exposed as PlasmaWindowModel.

Future work

The work on the interface is not done yet. There are still changes to the API needed and the upcoming KDE Frameworks 5.23 release will include new API calls in PlasmaWindow. As well we have already the first change scheduled for KDE Frameworks 5.24. And we also know that we need to expose a few more data elements in the protocol to get the task manager to the same level as on X11.

There is also an interesting idea floating around to use the – windowing system agnostic – Wayland implementation on both X11 and Wayland. For this KWin (on X11) would need to create a dummy Wayland server for the task manager to connect to. It’s an idea which we might or might not implement.

Security considerations

Currently every Wayland client is able to bind this interface, which means that some of the security improvements of Wayland are not available in Plasma/Wayland. We are fully aware of that and were also fully aware of the consequence when we added the interface. I do have ideas on how to address this and this will be implemented before we will recommend the Plasma/Wayland session for daily usage. The design allows to add security checks in these areas. Unfortunately my priority list did not allow me to implement this for Plasma 5.7, next chance 5.8.

Posted on

Looking at the security of Plasma/Wayland

Our beta release of Plasma 5.5 includes a Wayland session file which allows to start in a Plasma on Wayland session directly from a login manager supporting Wayland session. This is a good point in time to look at the security of our Plasma on Wayland session.

Situation on X11

X11 is not secure and has severe conceptual issues like

  • any client connected to the X server (either remote or local) can read all input events
  • any client can get information about when another window rendered and get the content of the window
  • any client can change any X attribute of any other window
  • any window can position itself
  • many more issues

This can be used to create very interesting attacks. It’s one of the reasons why I for example think it’s a very bad idea to start the file manager as root on the same X server. I’m quite certain that if I wanted to I could exploit this relatively easily just through what X provides.

The insecurity of X11 also influenced the security design of applications running on X11. It’s pointless to think about preventing potential attacks if you could get the same by just using core X11 functionality. For example KWin’s scripting functionality allows to interact with the X11 windows. In general one could say that’s dangerous as it allows untrusted code to change aspects of the managed windows, but it’s nothing you could not get with plain X11.

Improvements on Wayland

Wayland removes the threats from the X11 world. The protocols are designed in a secure way. A client cannot in any way interact with windows from other clients. This implies:

  • reading of key events for other windows is not possible
  • getting window content of other windows is not possible

In addition the protocols do not allow to e.g. position windows themselves, or raise themselves, change stacking order and so on.

Security removed in Plasma

But lots of these security restrictions do not work for us in Plasma. Our desktop shell need to be able to get information of other windows (e.g. Tasks applet), be able to mark a panel as a panel (above other windows) and need to be able to position windows itself.

Given that we removed some of the security aspects again and introduced a few custom protocols to provide window management facilities and also to allow Plasma windows to position themselves. At the moment we have no security restrictions on that yet which gives this functionality to all clients.

We will address this in a future release. There are multiple ways how this could be addressed, e.g. using the Wayland security module library or use systemd in some way to restrict access. Overall I think it will require rethinking security on a Linux user session in general, more on that later on.

Security added in Plasma compared to X11

The most important change on the security front of the desktop session is a secure screen locker. With Plasma 5.5 on Wayland we are able to provide that and address some long standing issues from X11. The screen locks even if a context menu is open or anything else grabbing input. The compositor knows that the screen is locked and knows which window is the screen locker. This is a huge change compared to X11: the XServer has no concept of a screen locker. Our compositor can now do the right thing when the screen is locked:

  • don’t render other windows
  • ensure input events are only handled in the lock screen
  • prevent access to screen grabbing functionality while screen is locked

As a matter of fact the Wayland protocol itself doesn’t know anything about screen locking either. This is now something we added directly to KWin and doesn’t need any additional custom Wayland interfaces.

How to break the security?

Now imagine you want to write a key logger in a Plasma/Wayland world. How would you do it? I asked myself this question recently, thought about it, found a possible solution and had a key logger in less than 10 minutes: ouch.

Of course there is no way to get a client to act as a key logger. The Wayland protocol is designed in a secure way and also our protocol additions do not weaken that. So the key to get a key logger is to attack KWin.

So what can an attacker do with KWin if he owns it? Well pretty much anything. KWin internally has a very straight forward trust model: everything is trusted and everything can access anything. There is not much to do about that, this is kind of how binaries work.

For example as a Qt application each loaded plugin has access to the QCoreApplication::instance. From there one could just use Qt’s meta object inspection to e.g. get to the InputRedirection model and connect to the internal signal on each key press:

<code>void ExamplePlugin::installKeyLogger()
{
    const auto children = QCoreApplication::instance()-&gt;children();
    for (auto it = children.begin(); it != children.end(); ++it) {
        const QMetaObject *meta = (*it)-&gt;metaObject();
        if (qstrcmp(meta-&gt;className(), "KWin::InputRedirection") != 0) {
            continue;
        }
        connect(*it, SIGNAL(keyStateChanged(quint32,InputRedirection::KeyboardKeyState)), this, SLOT(keyPressed(quint32)), Qt::DirectConnection);
    }
}

void ExamplePlugin::keyPressed(quint32 key)
{
    qDebug() &lt;&lt; "!!!! Key: " &lt;&lt; key;
}
</code>

But Martin, why don’t you just remove the signal, why should any other aspect of KWin see the key events? Because this is just the example of the most trivial exploit. Of course it’s not the only one. If you have enough time and money you could write more sophisticated ones. For example look at this scenario:

KWin uses logind to open restricted files like the input event files or the DRM node. For this KWin registers as the session controller in logind. Now a binary plugin could just send a DBus call to logind to also open the input event files and read all events. Or open the DRM node and take over rendering from KWin. There is nothing logind could do about it: how should it be able to distinguish a valid from an invalid request coming from KWin?

How to secure again?

As we can see the threat is in loading plugins. So all we need to do is ensure that KWin doesn’t load any plugins from not trusted locations (that is not from any user owned locations). This is easy enough for QML plugins where we have the complete control. In fact it’s easy to ensure for any of KWin’s own plugins. We can restrict the location of all of them.

And even more: by default a system is setup in a way that no binary plugins are loaded from user’s home. So yeah, no problem after all? Well, unfortunately not. During session startup various scripts are sourced which can override the environment variables to influence the loading of plugins. And this allows to also use the well known LD_PRELOAD hack. My naive approach to circumvent this issue didn’t work out at all as I had to learn that already the session startup and the PAM interaction source scripts. So your session might be owned very early.

An approach to black list (unset) env variables is futile. There are too many libraries KWin relies on which in turn load plugins through custom env variables. Most obvious examples are Qt and Mesa. But there are probably many more. If we forget to unset one variable the protection is broken.

A different approach would be to white list some known secure env variables to be passed to KWin. But this also requires that at the point where we want to do the restriction the session is not already completely broken. This in turn means that neither PAM nor the session manager may load any variables into the session before starting the session startup. And that’s unfortunately outside what we can do in our session startup.

So for Plasma 5.5 I think there is nothing we can do to get this secure, which is fine given that the Wayland session is still under development. For Plasma 5.6 we need to rethink the approach completely and that might involve changing the startup overall. We need to have a secure and controlled startup process. Only once KWin is started we can think about sourcing env variables from user locations.

So how big is the threat? By default it’s of course secure. Only if there is already a malicious program running in the system there is a chance of installing a key logger in this way. If one is able to exploit e.g. a browser in a way that it can store an env variable script in one of the locations, you are owned. Or if someone is able to get physical access to your unencrypted hard disk, there is a threat. There are easy workarounds for a user: make all locations from where scripts are sourced during session startup non-writable and non-executable, best change ownership to root and encrypt your home location.

Overall it means that Plasma 5.5 on Wayland is not yet able to provide the security I would have liked to have, but it’s still a huge improvement over X11. And I’m quite certain that we will be able to solve this.

Posted on

Plasma Phone and KWin

As you are probably aware by now we announced the Plasma Phone project during Akademy this weekend. In this blog post I want to discuss the role of KWin in Plasma Phone.

Plasma Phone uses Wayland as the windowing system with KWin being the Wayland compositor. This is our first product which uses Wayland by default and also the first product which uses KWin as the Wayland compositor. The phone project pushed the Wayland efforts in Plasma a lot and is the only reason why we are able to make Wayland a technological preview with the upcoming Plasma 5.4 release.

The phone project gave KWin/Wayland into the hands of many developers who started to actively use it and to develop with it. This obviously helped to find lots of small and larger issues which then could be fixed. It triggered a stabilization process which reached a stage that I can use a Plasma Wayland session on my notebook with enough confidence that I won’t lose data due to crashes. So thanks to the whole team for pushing the system to the limits.

An area which saw lots of work thanks to the Phone is the interaction between Plasma as the desktop shell and KWin as the Wayland compositor. With Wayland we keep the architecture of having a dedicated shell process which is not inside the compositor. This architecture has served us well in the past and we don’t see a reason why we should change this. It means that KWin can serve as a compositor for other desktop projects, but it also means that Plasma can be used with other compositors. Now unlike X11, Wayland’s protocols are secure by default. This means that Plasma as the desktop shell does not know anything about windows from other processes. To solve this problem we designed a special window management protocol which exports the required information. The protocols are still under development, but we hope that they can be also useful for other projects with a similar architecture (LXQt being an obvious candidate). Of course such protocols should only be used by the “blessed” desktop shell process – this is something we still need to implement and one of the reasons why at the moment Plasma/Wayland is only a technological preview.

Window management is not the only area where the shell process needs to be “blessed”. Also for announcing information about its own windows, we need some more information. We need to know whether a window is a “panel” or the “desktop” view. So on the phone the shell background is just a desktop window, the panel on the bottom is just a normal dock. This allows us to share the general layering code with the X11 implementation of KWin. Even more having the panels marked as panels allows us to properly define the maximized area. And the windows on the phone open maximized by using the “Maximizing” placement strategy in KWin.

Working on the phone project also had some surprises. For example the virtual keyboard just didn’t want to show. It turned out that the reason for this was that Qt(Wayland) only requests the virtual keyboard if it has keyboard focus. Keyboard focus inside KWin is bound to the “active” window. So we had to implement activating Wayland clients. But the initial implementation didn’t really solve it, we still had areas where the keyboard didn’t come up. E.g. on the desktop in the KRunner search field we couldn’t get it to show. The reason was related to the described problem: we did not yet support activating Wayland clients when clicking on them. This got solved by implementing mouse action (and touch action) support inside KWin. So thanks to this change (done for the phone) we can properly switch between Wayland windows on the desktop with a mouse driven setup.

Another nice touch is that KWin requires a running Xwayland server. This gives us support for all “legacy” X11 applications such as Qt 4 or GTK2 on the phone. Just we hit a problem with them: they did not react on touch events. The reason for this is quite simple: touch support is not yet implemented in Xwayland. As a solution we implemented simulating pointer events in case a connected Wayland window does not bind the touch interface. Thus all Wayland and X11 applications can now be used with a touch screen – be it on the phone or on a notebook with a touch screen.

So far I have only spoken about progress made in KWin which is relevant for the desktop. So what about the phone specific adjustments? Well there are hardly any. In the core of KWin there is no phone specific code. The only phone specific code is the hwcomposer backend. This is a plugin used for creating an OpenGL context using libhybris and for reading input events (unfortunately libinput cannot read events on a libhybris enabled system). According to cloc this plugin is just about 800 lines of code and about 200 are just very straight forward mapping of Android key codes to “normal” Linux key codes. For comparison: KWin + kwayland/server currently have about 120,000 lines of code. And even this hwcomposer backend is not phone specific. It could also be used to run a normal KWin/Plasma session on any libhybris enabled device. There is one important difference of the plugin to the rest of KWin which is worth to mention: it is GPLv3+ licensed, while everything else is GPLv2+. The reason for this change is the fact that libhybris is Apache 2 licensed and this license requires a change to GPLv3.