Plasma/Wayland and NVIDIA – 2017 edition

More than a year ago I elaborated whether KWin should or should not add support for NVIDIA’s proprietary Wayland solution. I think it is time to look at the situation of Plasma/Wayland and NVIDIA again. In case you haven’t read my previous blog post on that topic I recommend to read it as I use it as the base for this blog post.

Compared to a year ago not much has changed: NVIDIA still does not support the standard Linux solution gbm, which is supported by all vendors and nowadays even going to enter the mobile space. E.g. the purism phone is going to have a standard graphics stack with gbm. So no additional code required. But NVIDIA doesn’t support gbm. Instead it has a proprietary implementation called EGLStreams, which no other vendor implements. Due to that Plasma/Wayland cannot support OpenGL for NVIDIA users.

But current KWin master branch (what will become 5.12) supports automatical fallback to Qpainter compositing in case OpenGL fails. With that it should be possible to run Plasma/Wayland on NVIDIA hardware. Granted it won’t be accelerated, but as explained above that is only NVIDIA’S fault. Also I have not tested this yet. It might be that NVIDIA doesn’t support dumb buffers on DRM.

In my blog post from last year I elaborated whether KWin should add support for NVIDIA’S proprietary solution. I put out a requirement that the patches for Weston need to be merged first:

For KWin such patches do not exist and we have no plans to develop such an adaption as long as the patches are not merged into Weston. Even if there would be patches, we would not merge them as long as they are not merged into Weston.

A year passed and nothing happened. I don’t expect anything further will happen: the patches won’t be merged into Weston. Given that it doesn’t make any sense to make this a requirement for KWin. If we hold up the requirement it would be like saying we would never accept it.

Last year I hoped that XDC would find a solution for the problem:

We do hope that at XDC the discussions will have a positive end and that there will be only one implementation.

And it looked positive: a new universal buffer allocation library got announced. I hoped that this would solve the situation pretty quickly. The new library would get implemented, Mesa and NVIDIA add support for it, we slightly adjust our code and drop gbm. Everybody happy.

But now another XDC passed and from what I understand not much process was made. The library is not ready and no driver uses it. We waited one year for it, I doubt waiting another year will change the situation. Realistically I think it will take another two to three years for the library to get implemented if at all. Then it will take about a year for Mesa gaining support and shipping a release. Yet another year for distributions shipping with that version so that we can depend on it without people killing us. So maybe in five years there will be a replacement. I don’t think we can wait that long.

So does this mean KWin will gain support for EGLStreams? Certainly not! I do not think that the KDE community should spend any time to support NVIDIA’s proprietary solution! We are a free software community and we should not implement code which only benefits proprietary non-free solutions. There are way more free things to do to improve Wayland without having to write code for proprietary solutions.

Also there is another aspect which I did not consider in my blog post last year: XWayland. XWayland also uses gbm and does not have support for NVIDIA’s proprietary solution. Due to that one does not have OpenGL for any legacy X application. This probably includes most games, but currently also browsers such as Firefox and Chromium. I don’t think that this would give a satisfying experience to NVIDIA users. And it is also hardly better than rendering everything through the QPainter compositor. This is another reason for the KDE Community to not spend any time on implementing support for EGLStreams.

But I think there is lots NVIDIA could do. Today I would accept a patch for EGLStreams in KWin if NVIDIA provides it. I would not be happy about it, but I would not veto it. If it is well implemented and doesn’t introduce problems for the gbm implementation I would not really have an argument against it. But I expect NVIDIA to do it. I don’t want a contribution from a non-NVIDIA developer. This mess was created by NVIDIA, NVIDIA needs to fix it.

Similar I think that NVIDIA should adjust XWayland. I understand that NVIDIA is not happy with the design of XWayland, but nevertheless they should make it work. Their users pay quite some money for the hardware. I think they have a right to demand from NVIDIA to fix this situation. Ideal would of course be NVIDIA adopting gbm. But as that seems unlikely, I think it is the duty of NVIDIA to provide patches for their users.

KWin/Wayland goes real time

Today I landed a change in KWin master branch to enable real time scheduling policy for KWin/Wayland. The idea behind this change is to keep the graphical system responsive at all times, no matter what other processes are doing.

So far KWin was in the same scheduling group as most other processes. While Linux’s scheduler is completely fair and should provide a fair amount of time to KWin, it could render the system hard to use. All processes are equal, but some processes are more equal than others. KWin – as the windowing system – is to a certain degree more equal. All input and all rendering events need to go through KWin.

We can now imagine situations where a system can become unusable due to KWin not getting sufficient time slots. E.g. if the user moves the mouse we expect a timely cursor motion. But if KWin is not scheduled the system is quickly starting to lagging. Basically what we expect is that when the mouse moves with the next page flip the cursor must be at the updated position. A page flip happens normally every 16 msec (60 Hz), so if we miss one or two we are in the area where a user visually notices this. For a gamer with a high precision mouse this could be quite problematic already.

Another situation is a few processes running amok and one wants to switch to a virtual terminal to kill the processes. The key combination (e.g. Ctrl+Alt+F1) needs to go through KWin. Again the user wants to have a responsive system in order to be able to switch the session.

While thinking about these general problems I came to the conclusion that we should try making KWin a real time process. This would ensure that KWin gets the CPU whenever it wants to have the CPU. This is actually quite important: KWin is only taking CPU if there is a reason for it, e.g. an input event or a window requesting a repaint, etc. So by adding a real time scheduling policy to KWin we can ensure that all input events and all rendering events are handled in a timely manner. My hope is that this in general results in slightly smoother experience and especially in a smoother experience if the system is under heavy load.

Of course I considered the disadvantages. If KWin is a real time process, it always wins. It’s on the scheduler’s fast lane. This could of course be a problem for other processes actually demanding a higher scheduling policy or which are real time itself. E.g. a game or a video player might already be real time. But here we also see that there is no point in having the game be real time for fast input if the windowing system itself is not real time. So this is actually not a disadvantage, but rather another reason to make KWin a real time process.

There could be other real time processes which are more important. Of course those should win against KWin. To support this KWin requests only the minimum priority, so any other real time process in the system is considered more important than KWin.

The only real “problem” I see would be KWin running amok, because then KWin would get too much CPU time. But in this case the system would be broken anyway. If e.g. KWin would be stuck in an endless loop one would not be able to switch to another VT as KWin is stuck in the endless loop. This is obviously only a theoretical example.

So how is it done? KWin gained a new optional dependency on libcap and on installation sets the capability CAP_SYS_NICE on the kwin_wayland binary. @Distributions: please update your build deps. On startup kwin_wayland adjusts the scheduler policy to request SCHED_FIFO and drops the capability again before doing anything else.

As this change is in master it won’t be part of the Plasma 5.11 release which also means we have now the maximum time span till it ends up in a release. I would be happy about users testing it and especially report any negative influences of the change.

Announcing the XFree KWin project

Over the last weeks I concentrated my work on KWin on what I call the XFree KWin project. The idea is to be able to start KWin/Wayland without XWayland support. While most of the changes required for it are already in Plasma 5.11, not everything got ready in time, but now everything is under review on phabricator, so it’s a good point in time to talk about this project.

Why an XFree KWin?

You might wonder why we spend time on getting KWin to run without X11 support. After all we need to provide support for XWayland anyway to be able to support legacy applications which are not yet on Wayland. So what’s the advantage if in a normal session one needs XWayland anyway?

One aspect is that it shows that our porting efforts to Wayland are finished. Over the last years I tried to start KWin without XWayland support for a few times just to find areas which are not yet ported. By being able to run KWin without X11 support we know that everything is ported or at least does not hard depend on X11 any more.

Another aspect is obviously Plasma Mobile which does not really require XWayland and also XWayland not making much sense on e.g. the libhybris enabled systems as Xwayland doesn’t have OpenGL there. By not requiring XWayland we can reduce our runtime and memory costs.

Speaking of runtime costs: not requiring X11 means that we don’t have to wait for XWayland during KWin startup. Instead XWayland can be started in parallel. This means KWin and the complete Plasma session starts a little bit faster.

And most important this is an important prerequisite to be able to handle a crashing XWayland. So far when XWayland crashed KWin terminated gracefully as KWin depends on X11. The hope is that when XWayland crashes we can just restart it and keep the session running.

How it was done

The general idea behind getting KWin X11 free is “code that isn’t loaded, cannot interfere”. KWin uses platform plugins (not Qt QPA plugins) for the various platforms KWin can run on. There is also a platform plugin for KWin/X11, so code which is only required in the KWin/X11 case can be moved into the platform plugin. As KWin/Wayland does not load this plugin we are certain that the code will not be loaded and thus cannot interfere.

But how to find code which is only required on KWin/X11? After all KWin’s code base is about 150 kSloc (according to cloc) and that makes it rather difficult. A good help here was our CI system which performs code coverage. KWin’s tests mostly are mostly based on KWin/Wayland so an area which does not have any test coverage is a good candidate for being X11 specific. By looking at these areas it was possible to find patterns which also helped to find more usages. A good help is KWin’s internal X11 API such as displayWidth, displayHeight, rootWindow and connection. The usage of these functions is partially so few that one could just evaluate each usage. As a result of this work the functions displayWidth and displayHeight are not used at all any more.

Plugin based compositors

Another idea was to get our compositors into plugins. Especially the XRender based compositor is not of much use in a Wayland world and thus should not be loaded into the binary. Unfortunately various parts of KWin called directly into the concrete compositor implementations, so to solve this we had to extend the internal API. In Plasma 5.11 the XRender and QPainter compositor are now loaded as plugins, so on Wayland the not-compatible XRender compositor is no longer loaded into memory and on X11 the not-compatible QPainter compositor is no longer loaded into memory. But also on Wayland the QPainter compositor is only loaded into memory if it is going to be used.

The OpenGL compositor is still always loaded in Plasma 5.11, but the change to have it as a plugin is already prepared and will be merged into master soonish. This will bring great advantages to the stability of the system: currently we are not able to define which platform supports which compositor as the initialization code just didn’t support being more flexible. But now with the plugin based approach I’m very confident that we can make this work in a better way.

Outlook

Being able to start and run KWin/Wayland without X11 support is of course only the start. More changes will be required. For example to delay loading XWayland until an application tries to connect to it (c.f. Weston). This would not make much sense in the start of Plasma yet as we still have applications in our startup which require X11 (e.g. ksmserver).

Another area is to try to get KWin compile without X11 support and to move everything required for Xwayland into a plugin. This will be a larger project as it will require to move much code around and to add further abstractions in some areas of KWin. Hint: this could be a nice Google Summer of Code project. As a fast step for especially Plasma Mobile and the Purism Librem phone an ifdef around every X11 code area could be a solution.

But my personal main goal is to be able to handle a crashing XWayland. This will also be an interesting task as the X11 code in KWin also depends on other libraries (e.g. KWindowSystem) which do not expect to outlive the X server. So that will be a challenging task to find all areas where KWin holds X11 data structures and to clean them up properly without crashing due to some cleanup code calling into xcb.

Plasma Wayland and Qt 5.9 and beyond

As you might know Qt 5.8 created challenging problems for our Wayland session and threw our efforts back quite a bit. In this post I want to discuss the actual problems it created, how we are addressing them and looking into the future.

How our integration used to work

Our integration uses additional Wayland protocols. We have a protocol for server side window decorations which we use in our Plasma integration plugin to inform KWin whether the window should have a decoration or not. We have a protocol for client provided shadows which is e.g. used by our widget style Breeze to add shadows to the context menus. We have a protocol for the desktop shell, so that it can mark windows as desktop, panel, auto hiding panel, position the window, etc. Also we have a few protocols for interacting with our effect system, e.g. sliding popups, blur behind.

To use these protocols we need to interact with Qt in a low level way. We use the native interface in the Qt Platform Abstraction to get a wl_surface pointer for the QWindow. In order to not have to keep this simple for our applications our KWayland::Client API provides an API point for it: Surface::fromWindow(QWindow*) -> Surface*.

But when exactly to inject our own integration? We found a very handy way which worked much better than what we had used in the past for X11 (and based on that also transitioned X11 code to use it). Qt emits an event once it has created the native platform surface (in case of Wayland the wl_surface*) for a QWindow. Verbatim quote of the documentation:

The QPlatformSurfaceEvent class is used to notify about native platform surface events.

Platform window events are synchronously sent to windows and offscreen surfaces when their underlying native surfaces are created or are about to be destroyed.

Applications can respond to these events to know when the underlying platform surface exists.

Awesome! We get an event when the surface is created and when it gets destroyed. This made it very simple to create the integration and what is really important for us is that we get this event before the window is shown. So we can prepare everything so that KWin gets a good state.

And the KWin side was to a large part implemented on assumptions on how the sequence will work. We first get the surface, then the (xdg) shell surface, then the integration bits. Sure it would be nice if KWin handled also other sequences, but as the only implementation of this is Qt it doesn’t make sense to really care about it. We know that it was not perfect, we even had the test cases for it, which expect failed.

What broke with Qt 5.8?

In Qt 5.8 our complete integration broke. When the platform surface created event was emitted the wl_surface was not created, see QTBUG 58423. This is in my humble opinion a clear violation of the documented behavior and thus a breakage of the stability guarantees Qt provides, but others might disagree. After some discussion, trying out patches by those who had a Qt 5.8 build we had a patch for Qt which made things mostly work together with a patch to KWin. But at the time we had the patch ready Qt 5.8 was already declared end of life with no prospect of a Qt 5.8.1 bug fix release. For our Wayland session it was just impossible to get Qt 5.8 compatible again. All we could do was to advise distributions to not combine Qt 5.8 with the Wayland session. From our side it was fine for distros to ship Qt 5.8, but if they do they should make it impossible to install the Wayland session. The state was just too broken.

Qt 5.9

With Qt 5.9 the situation looks better. The required patch is merged, it’s an LTS release and we already had the first bug fix release. Qt wants to create more bug fix releases for it and this allows us to use it as a new target for integration. But still the situation is not as good as it used to be. If you currently use our Wayland session with Qt 5.9 you will still see quite some rough corners compared to where we were with Qt 5.7.

The main problem for us is that the platform surface regression was not the only change affecting us. Pre Qt 5.7 a wl_surface lived as long as a QWindow. Now the wl_surface gets destroyed whenever the window gets hidden and a new one created on every show. Unfortunately without a platform surface created event. This means our integration breaks as soon as a window gets hidden. E.g. after closing KRunner the integration for KRunner is broken.

We tried to address this problem in various places, but it is challenging. On the show event we don’t have the wl_surface yet (too early), on the expose event the window is already mapped (too late). This creates problems for KWin which is not prepared for the integration bits to hit when the window is mapped. Our protocols were designed with the platform surface created event semantics. For example in KRunner we face an issue with the integration. KRunner is a panel, which accepts focus and allows windows to go below, also it positions manually. Now when it gets re-shown KWin doesn’t know that this window is supposed to position itself and positions it. Now we get the request to put it as panel, KWin adjusts and moves maximized windows around. And then we get the request that the panel allows windows to go below. KWin shuffles the windows again as the maximized area changed again.

This is a rather tricky situation as we cannot really do something about it. If the window is mapped it’s too late. Even if we improve our API to handle the situation better it will be too late.

There are two possibilities to handle this: Qt stops to destroy the surface or sends a platform surface created event when it recreates the surface. The latter would be my personally preferred solution as this would match the documentation again and allow us to just use the one event handler.

Other regression

The situation around the changed behavior in Qt 5.8 caused a few steps back. Our code needed to be adjusted and that sometimes caused issues. We had a few regressions which also affected the compositor, so the stability of the whole system suffered. These issues are luckily investigated and fixed. But there are still bugs lurking in the system.

For me personally the most annoying bug is a crash in Qt which affects the auto completion of kdevelop. This makes hacking in a Wayland session rather difficult. I’m running currently a patched KWin which disables the virtual keyboard integration to not hit this issue.

A huge problem is that context menus are not marked as transient windows. This means that the Wayland compositor does not know that it is a menu and positions the menu anywhere on the screen. It gives the system a very unfinished touch.

If KRunner is closed through the escape key, the key starts to repeat on the window constantly and due to that it is not possible to open KRunner again. Similar if you start an application in Kickoff through the enter key, when opening Kickoff again it automatically launches the currently selected item. This again makes it very difficult to use the session and gives the whole system an unfinished look. We are working on a workaround for this issue in the server.

Towards the future

Qt 5.9 is here to stay and that’s what we have to use as integration target. Given that Qt 5.9 and Qt 5.7 behave very differently it will become difficult for us to maintain support for both. My suggestion is that we drop support for Qt 5.7 and require Qt 5.9 for the Wayland session. In addition there is hope that we can improve the integration. Marco and David have been working on adding support for XDG Shell unstable v6 which is already supported by Qt and makes it easier to integrate with. Once this landed in KWin Qt will be switched from wl_shell to xdg shell. This will improve the situation for us quite a bit as we then have one code path for both Qt and GTK applications.

How to prevent such regressions in the future

The change of behavior in Qt 5.8 threw our Wayland efforts back a few months. This is something we communicated to Qt quite early and it’s something which worries me a lot. We cannot spend time on changing our integration every time Qt releases a new version. Given that we need to look into how to prevent that such a situation happens again.

I hope that we can improve the integration on the testing front between Qt and KDE. We have a huge test suite which can find regressions in Qt. If Qt would run KDE’s tests during the integration phase Qt would notice regressions before they hit the code base. Given that all our tests are free software it should be possible for Qt to integrate them.

But also the other side would be interesting: if we could get the latest Qt into our CI system we could also discover breakages early. We have now a new docker based CI system which allows running multiple builds of the same change (e.g. Plasma gets build on openSUSE tumbleweed and on FreeBSD) – an image with a daily or weekly Qt snapshot could help us and Qt a lot to detect breakages early.

I also hope in openQA which allows to test the full operating system. This would spot regressions like the misplaced context menus even if KWin’s own auto tests would not spot them (KWin doesn’t care about Qt there, only about the Wayland and X11 protocols). There we might need to invest some work to make sure that KWin/Wayland can be properly run in the openQA tests.

I hope that our Plasma devs can discuss this in more detail with Qt devs during Akademy in person. Unfortunately I cannot be at Akademy this year 🙁 so I cannot discuss in person.

Last but not least it is important that developers test. It would help a lot if the developers working on QtWayland test their changes in a running Plasma Wayland session. We are now overall in a state that the session is suited for hacking on. I do all my Wayland hacking in a Wayland session, experiencing all the glitches like kdevelop crashing.

Of course you might wonder what about us? Shouldn’t we KDE devs also test against the latest Qt? For me personally that is not the case. I’m working on the server side and not on the client side. I’m also not testing the latest GTK for example. Nevertheless I tried to use Qt 5.9 before it got released. Used the installer, spent a day to compile everything on top of it it, just to notice that it doesn’t have QtWayland and won’t get it. I didn’t give up that easily. So I tried to compile QtWayland myself. But when I tried to use it, it turned out to not have any keyboard support, because qtbase was compiled without support for xbkcommon. At that point I gave up. Not having QtWayland is one thing, but not being able to use keyboard is another, it’s rather pointless. The only other option is to compile Qt, but that is hardly an option as it’s really difficult to compile an actually working Qt with all components. The last times I tried, I failed, wasting days compiling. If there were usable weekly images for Qt I would be happy to try it, but of course only with a properly compiled and included QtWayland.

How input works – touch screen edge swipe gestures

Continuing my series about how input works in KWin/Wayland I want to discuss a brand new feature we implemented for Plasma 5.10. This year we had a developer sprint in Stuttgart and discussed what kind of touchpad and touch screen gestures we want to support and how to implement it. Now the result of this discussion got merged into our master branch and we are currently discussing which actions to use by default.

Gesture Recognizer

Touchpad and touch screen gestures are kind of similar so the approach we took is able to handle both of them. We introduced a GestureRecognizer which is able to recognize gestures (surprise) in a very abstract way. It doesn’t know how the input events look like, whether a touch screen, touchpad, mouse or whatever input device generated the event.

To use the GestureRecognizer a Gesture needs to be registered. The Gesture describes the actual Gesture which needs to be recognized. E.g. how many fingers need to participate in the gesture (for our touch screen gestures it is one, for our touchpad gestures it is four), the direction of the gesture (leftwards, rightwards, downwards, upwards), the minimum distance, the trigger position, etc. etc.

Now input events can be fed into the GestureRecognizer and the GestureRecognizer decides whether a Gesture is active or becomes non-active. As said this is absolutely generic, it doesn’t care how the events are triggered.

This alone does not yet allow to do anything with it, we don’t have any way to use the GestureRecognizer yet.
At this point our implementations for touchpad and touch screen divide. We have different existing implementations which are more suited than trying to have something shared for both.

Touchpad gestures

For touchpad gestures our global shortcuts handling is used. GlobalShortcutsManager is a KWin internal (!) mechanism to register some internal actions to tigger in a global way through input events. The GlobalShortcutsManager gained a GestureRecognizer and various areas in KWin can now register a QAction as a global touchpad gesture.

So far we still haven’t reached the elements we discussed in the previous posts like InputEventFilter. There is of course an InputEventFilter which feeds events into the GlobalShortcutsManager. This filter got extended to support touchpad gestures and now we have the full stack together:

  • libinput reports a touchpad gesture event
  • InputEventFilter passes the touchpad gesture event to the GlobalShortcutsManager
  • The GlobalShortcutsManager passes the information to the GestureRecognizer
  • The GestureRecognizer triggers the QAction
  • something happens

By default the following gestures are supported:

  • 4 finger swipe down: Present Windows
  • 4 finger swipe up: Desktop Grid
  • 4 finger swipe left/right: desktop next/previous

Screen edge support for touch

For touch screen gestures we used a different area in KWin which already provides a fairly similar functionality: Screen edge activation for mouse events.

It’s similar in the way that it activates an action when an input event happens at a screen edge. The main difference is the direction of the input event: for mouse it’s towards the edge, for touch it is from the edge.

The ScreenEdges gained a GestureRecognizer (you see there are two different, independently acting GestureRecognizers) and every Edge defines a Gesture. The Gesture is only passed to the GestureRecognizer if the Edge is reserved for a touch action. Each Edge can have a different action configured and of course you can configure different (or same) action for touch and mouse on the same Edge. When a Gesture for an Edge gets started we provide the same visual feedback as for the edges started through mouse events.

For ScreenEdges there is also a dedicated InputEventFilter which now gained support for touch events and feeds the touch events into the GestureRecognizer.

But there is more to it. This feature got backported to X11. Our X11-standalone platform plugin gained support for XInput 2.2 and touch events. KWin now listens for touch events on X11 and passes these events into the GestureRecognizer of the ScreenEdges. Thus the feature which we implemented on Wayland for Wayland is also available on X11 and directly usable for all users – be it on X11 or on Wayland.

Touchpad gestures are unfortunately only available on Wayland.

How input works – touch input

Last year I started a blog post series about how input works in KWin/Wayland. This blog post resumes this series by talking about touch input.

Several people wondered why it took so long for this blog post. After all it’s more than a month since the last one. Of course there is a good reason for it. I was reworking parts of the input stack and wanted to discuss the changes with the next post of the input blog post series. Unfortunately there are still a few changes missing, so I decided to nevertheless do the touch input post first.

Touch input is the new kid in the block concerning input events. It’s a technology which was created after X11 got created and thus it is not part of the X11 core protocol. On X11 this makes touch a weird beast. E.g. there is always an emulation to a pointer event. Applications which do not support touch can still be used as the touch events generate pointer events. Now this is actually a huge sacrifice for the API and means that touch feels – at least to me – as a second class citizen in X11.

On Wayland the situation is way better. Touch is part of the core input protocol and does not emulate pointer events. Applications need to support touch in order to get touch events. If an application does not support touch, the touch events won’t trigger any actions. This is a good thing as it means applications need to do something sensible with touch events.

Like with the other events touch events are reported to KWin by libinput. Touch events are quite straight forward. We get touch down events (when a finger touches the screen), touch up events (when a finger gets lifted again) and touch motion events (when the finger moves on a screen). This is fully multi-touch aware, meaning we can follow multiple touch points individually.

The events are sent through KWin’s internal filter architecture like all other events. Currently KWin does not really intercept events yet. We do support touch events on window decoration and KWin’s own internal windows. But in those cases we emulate mouse events. We don’t have any UI elements which would benefit from multi touch events, thus emulating mouse events internally is sufficient for the time being. If in future we add multi touch aware UI elements that would require changes.

In case KWin does not intercept the touch sequence the events are passed on to the KWayland Server component which forwards the events to the Wayland window which is currently receiving touch events. KWin determines the window by using the window at the first touch down of the sequence. While a sequence is in progress the window cannot change.

The touch events are then processed by the application and can provide sensible functionality. E.g. our Plasma calendar supports a pinch-zoom gesture to switch to an overview of all months. This was developed under X11 and just works on Wayland without any adjustments. Good job, Qt devs!

Last week at the Plasma sprint touch gestures were an important discussion point during the last days. We decided which global gestures we want to support in Plasma. We hope to be able to deliver this for Plasma 5.10 on Wayland and will also look to get the same on X11 by reusing the architecture written for Wayland. But this might land in a later release.

Global touch gestures have an interesting and useful feature. When a sequence starts KWin does not know whether that will be a global gesture or a gesture which needs to be forwarded to the applications. Thus all events must be sent to the applications. Once KWin knows that this is a global gesture it can send a cancel event to the application. This informs the application that the touch sequence got canceled. This prevents conflicts between the global and application touch gestures. On X11 this is not so comfortable, so we will have to see how we can support this.

How input works – pointer input

In the last blog post I discussed keyboard input. This blog post will be all about pointer devices – mostly known as “mouse”. Like my other posts in this series, this post only discusses the situation on KWin/Wayland.

Different hardware types

There are different kind of devices which are recognized as a pointer device. We have the classic mouse/trackball like devices and on notebooks we find touchpads. Furthermore there are also absolute positioning pointer devices, which are sometimes found on touch screens.

Given the differences of the devices there are quite a few configuration options available in libinput for pointer devices. There is for example pointer acceleration and many options for touchpads defining how it should behave. We are currently working on a touchpad KCM for Wayland, so it looks like this will return with Plasma 5.9. As explained in the first blog post of this series the configuration options are set as soon as the device is created.

Pointer motion

The pointer devices generate various events and one of them is the motion event. In general there are two kind of motion events: absolute and relative. Most devices like a mouse generates relative motion events which is the reason why this blog post will only focus on them.

Determining new position

A relative motion is a distance vector with an x and y coordinate. It describes how the cursor position should be moved.

So once the event is read from the queue inside KWin the new position needs to be determined. Now it’s not as simple as taking the last position and then adding the motion vector. This could result in the cursor leaving the visual area.

Instead the pointer motion gets validated and constrained. We ensure that the cursor doesn’t leave the visual area and also apply pointer constraints an application window set.

This is a new protocol KWin supports in Plasma 5.9. It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area. In the first case the pointer doesn’t move at all, in the second case it’s only allowed to move in a certain region of the window.

Processing new position

Even in case the pointer motion is constrained in a way that the cursor doesn’t move, the event is further processed. An application might be interested in the relative motion and react to it, even if the cursor doesn’t move.

For the further processing a QMouseEvent is generated and sent through KWin’s input filters just like described for the keyboard case. The pointer motion might be handled inside KWin, e.g. the active screen edges need to know the current position. Or the pointer motion might be forwarded to a window through KWayland.

Updating the focused window

If the pointer moves it might be that the cursor moves from one window to another or from a window to it’s server side decoration. This means that for every pointer position change KWin needs to evaluate the window at the current position.

Compared to keyboard input where KWin only needs to consider the active window this is a rather complicated task. We need to consider input transformation applied to the screen or window, we need to apply input masking on the window, consider the window decoration, check whether the screen is locked, workaround issues in Xwayland prior 1.18, etc. etc.

In the end the method might have determined a new Surface which gained pointer focus. KWin uses KWayland to update the focused pointer surface which ensures that the surface leave and enter events are emitted.

Of course not always when you move the mouse it should update. If a grab is active (pointer button pressed) it won’t update.

Updating the cursor image

If the pointer moved to a new position it might be that the cursor icon changed. This unfortunately might require a roundtrip. One doesn’t know which icon a window wants to use till the motion was sent to the window. A window might react in two ways: it updates the cursor image, or it doesn’t. In the first case KWin gets notified through KWayland that the image changed, in the second case there is no notification at all.

This means KWin doesn’t know what cursor icon is really valid when moving the cursor. So on pointer motion KWin updates the position of the cursor with the current cursor icon, but it might be that a frame later the client updates it. This is so far the only element in Wayland where I have to say that not every frame is perfect. The cursor could show the wrong icon.

When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. Something we have to improve on. Currently we don’t detect hung applications yet as I think we cannot detect them at all due to clients using a dedicated Wayland event thread and thus always happily replying that everything is ok, even if not.

But it might also possible that KWin needs to set a cursor image. E.g. when hovering a window decoration or entering a special mode for selecting a window the cursor image is provide by KWin. KWin loads the cursor image from the theme. Internally KWin tracks the source from where the cursor image should be used. Whether it’s a Wayland window, or the window selection, or an effect setting a specific image.

Updating the actual cursor

The actual update of the cursor position and icon happens through KWin’s internal platform API. Every platform sets the cursor image in a different way. For our primary platform we use the DRM api to up update the position and to update the image – if a new one is available.

For the nested platforms like X11 and Wayland this happens through the windowing system specific calls. The nested platforms don’t allow to update the cursor position – this happens by the windowing system through the pointer motion. The cursor image, though, can be updated.

The virtual platform only knows the concept of a software cursor. That is the cursor gets rendered in the compositor rendering pass. Currently that is only implemented in the QPainter compositor and not yet available in the OpenGL compositor.

Button events

The next event supported by libinput for pointer devices are pointer button press/release events. These events carry the pointer button they triggered for.

Compared to pointer motion the event processing is way more straight forward. The pointer event is either intercepted by one of our event filters (e.g. Alt+Tab) or forwarded to the window currently having pointer focus.

This is a huge improvement over X11. On X11 if KWin wants to exclusively process pointer events it needs to grab the pointer. That implies that the screen cannot be locked. So if e.g. Present Windows is active the screen doesn’t lock. On Wayland this doesn’t matter any more. If Present Windows is active, the screen will lock and get all pointer events. Once unlocked Present Windows is still active. I was quite happy when I was able to add an auto test for that situation.

Axis events

Many pointer devices have one or two axis. On X11 the core protocol implemented axis events as pointer buttons. With Wayland and libinput we now have dedicated axis events telling us which axis got scrolled and the delta. This is a big improvement compared to X11 as it means that “smooth scrolling” is part of the standard and not something added later on through an extension.

The handling is of course very similar to the other events. KWin creates a QWheelEvent and passes it through the various event filters and if no filter intercepted the event, it will be forwarded to KWayland::Server which in turn sends it to the focused pointer surface.

Touchpad gestures

For touchpads we have further events. Libinput does not only recognize motion and press events on touchpads, but is also able to recognize a multi finger gesture. It supports two kind of gestures: swipe and pinch/rotate gesture. KWin gets the gesture events and forwards them through the normal event system. There is a special Wayland protocol which we added support in Plasma 5.9. This allows forwarding the pointer gesture to Wayland applications as the following video demonstrates.

Unfortunately we do not really use these gestures yet. QtWayland doesn’t implement the protocol, so the forwarding doesn’t reach any application and we don’t make use for it yet internally for e.g. global gestures. We are still working on defining the gestures we want to support. I hope we have something for Plasma 5.9, but no promise.

Happy holidays

This is my last blog post for this year. Next year I will continue this series with a blog about touch screen events and maybe also wacom tablet.

I wish everyone happy holidays and a great start into 2017.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

How input works – Keyboard input

In the last blog post I explained how input devices are opened and handled in KWin. In this blog post I’ll have a closer look on keyboard devices and events.

Keyboard are not keyboards

Keyboards on Linux are weird. You don’t have one keyboard but many of them. Many devices also announce to be a keyboard and just support one key. A good example for this is the power button or an external headset which provides mute, volume up/down keys. From an input perspective such devices are also keyboards.

For us in KWin it is important to figure out what the keyboard really supports. If there is no “real” keyboard attached (or enabled), our virtual keyboard should get activated automatically. E.g. if you detach the keyboard from a convertable it should turn into tablet mode by having a virtual keyboard. When attaching the keyboard, the virtual keyboard should be disabled as the primary text input device. libinput provides a function to test which keys are supported. We use that to differentiate the classes of keyboards.

Keyboard events

Keyboards are the most simple input devices out there. Libinput only emits one event of type LIBINPUT_EVENT_KEYBOARD_KEY and that only contains the key which was either pressed or released. KWin reads events from libinput in a dedicated thread, so each event only gets queued and our main thread is notified about the new event. Once the main thread processes the event, the event gets translated into our input redirection classes. All input events go through the input redirection, no matter from which source the events are delivered. KWin does not only support events from libinput, but also the nested setups (KWin running on top of X11 or on top of another Wayland server) and fake events used in our integration tests. This means once the event reaches the input redirection we in general lose the information which device created the event. Though recently we extended the internal API to optionally include the device in the event handling. This is used by the Debug Console to show on which device an event was generated. But more on that later.

xkbcommon

Now the key press/release event has reached our central dispatching method KeyboardInputRedirection::processKey. The first (and most important) task is to update the keyboard state in xkbcommon. Xkbcommon is used to translate a hardware key with a layout to the actual key symbol depending on the state of the keyboard (e.g. active modifier). To explain: if I press the “y” (key code 21) key and have the “Shift” key pressed, it will create a “Z” with the German keyboard layout, but a “Y” with the English layout. Simplified that’s the job of xkbcommon.

In KWin we have wrapped all functionality for xkbcommon in a dedicated class called Xkb. This class tracks for us the active layout and performs the layout switching (including showing the OSD when the layout changes). It knows the last composed key symbols, the currently active modifiers and the modifiers relevant for shortcut activation.

When updating the state of xkb we also check what changed. Did the user activate the num lock? If yes we need to announce that the LEDs changed, so that our libinput code can update the LEDs on the physical keyboard. Did a modifier change? If yes we need to inform our Wayland windows about the new modifier set. In Wayland this is tracked on the server, although the actual translation from key to symbol happens on the client. So why does KWin also do the translation? KWin also needs the keysym in various places, e.g. the filter in Present Windows or in general for triggering global shortcuts.

Keyboard state in Debug Console
Keyboard state in Debug Console

Our Xkb state updating functionality is also responsible for handling modifier only shortcuts. Actually it’s the wrong place for it, but our input filtering code does not guarantee that a filter sees all input events. For the modifier only shortcuts it’s essential to see all events, though, and the only place is directly in Xkb. Not the most elegant solution, but it works. This functionality is also used by X11 as I explained in an older blog post.

Filtering through KWin

Now KWin has enough information to process the key event. For that it creates a customized QKeyEvent and sends it through an input filter chain. KWin’s input processing is using a chain of input filters. Each filter can perform an operation based on an event and decide whether the event should be further processed or whether event processing should end.

For example pretty early in the chain we have the lock screen filter. If the screen is locked this filter intercepts the event processing and ensures that the event is only sent to the screen locker and not to any window. Or there is a filter ensuring that ctrl+alt+f1 works even if the screen is locked. Another filter is responsible for handling global shortcuts, one for passing events to our effects system (such as Present Windows).

One of the input event filters shows information about key presses in the debug console
One of the input event filters shows information about key presses in the debug console

The last filter in the chain is our forwarding filter. The task of this filter is to forward the events to a window. It passes the event to KWayland::Server from where it is sent to the currently focused Wayland surface.

Focused Keyboard surface

The Wayland server needs the focused keyboard surface for that. In case of keyboard focus that is relatively trivial in KWin. KWin has a concept of an “active” window. Before forwarding the event KWin verifies which is the focused keyboard window. If there is an active window the surface of that window is marked as the focused keyboard surface in KWayland::Server.

Our KWayland::Server library takes care of sending a keyboard leave and keyboard enter event to the respective windows, so that KWin doesn’t have to care about this. This is one of our advantages by having an abstraction with KWayland::Server – everything that is not of relevance to the compositor is handled directly in the library.

Key event processing in Wayland

The forwarding input filter updated the keyboard surface and sends now the key event to the Wayland client. For that all the processing into keysymbol is no longer needed, the key code is sent to the client.

The client gets the key event through a callback and now also sends it through xkbcommon. In Wayland the keymap is sent from the server to the client, so that both server and client have the same keymap. The client can now do a translation from key code to key symbol, just like KWin did before.

The further event processing is handled inside the client. E.g. in Qt this will generate a QKeyEvent which is then sent to the focused widget.

Key Repeat

Keyboard input has also a special mode: repeating keys. When a key is pressed, some of them should generating repeating keys. KWin uses the configuration from the keyboard module to decide when and how often a key should repeat. A repeating key is not forwarded to the Wayland clients. Instead KWin tells through the Wayland Keyboard protocol the settings for key repeat and this is than handled directly in the client.

Unfortunately in Qt this is broken and a hardcoded value is used. So currently in a Plasma Wayland session key repeat is rather broken as it’s handled differently depending on the used application. KWin is correct, X11 applications are correct, GTK applications are correct, Qt applications are incorrect, if run on Wayland.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

How input works – creating a Device

Recently I did some work on the input stack in KWin/Wayland, e.g. implemented pointer gesture and pointer constraints protocol, and thought about writing a blog series about how input events get from the device to the application. In the first blog post I focus on creating and configuring an input device and everything that’s related to get this setup.

evdev

Input events are provided by the Linux kernel through the evdev API. If you are interested in how evdev works, I recommend to read the excellent post on that topic by Peter Hutterer. For all we care about the input event API is too low level and we want to use an abstraction for that.

libinput and device files

This abstraction exists and is called libinput. It allows us to get notified whenever an input device gets added or removed and when an input event is generated. But not so fast. First of all we need to open the input devices. And that’s a challenge.

The device files are normally not readable by the user. That’s a good thing as otherwise every application would be able to read all key events. Getting a key logger would be very trivial in that case.

But if KWin runs as a normal user and the user is not able to read from the device files, how can KWin read them? For this we need some support. Libinput is prepared for the situation and doesn’t try to open the files itself, but invokes an open_restricted function the library user has to provide. KWin does so and outsources the task to open the file to logind. Logind allows one process to take control over the current session. And this session controller is allowed to open some device files. So KWin interacts with logind’s dbus API to become the session controller and then opens the device files through the logind API and passes them back to libinput.

This is the reason why for a full Wayland session KWin has a runtime dependency on logind’s DBus interface. Please note that this does not mean that you need to use logind or systemd. It only means that one process is required which speaks logind’s DBus interface.

Devices in KWin

Now libinput is ready to open the device files and emits an LIBINPUT_EVENT_DEVICE_ADDED event for each device. KWin creates a small facade class for each device type and applies configuration options for it. KWin supports reading the configuration options set by Plasma’s mouse configuration module and has an own device specific configuration file which will soon allow the touchpad configuration module to configure the touchpad on Wayland. Also as part of setting up the device KWin enables LEDs – if the device supports them – for Num Lock and Caps Lock.

Input Devices

All the input devices created by KWin can be investigated in the Debug console (open KRunner, enter “KWin”). KWin reads a lot of information about the device from libinput and shows those in the Debug console. In the input event tab each of the events include the information which device generated the event.

Input Devices exported to DBus

All devices are also exported to DBus with the same properties as shown in the Debug console. This means the configuration can be changed at runtime through DBus. KWin saves the configuration after successful apply and thus ensures that your settings are restored correctly when you restart your system or replug your external device. This is also an important feature to support the touchpad configuration module.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

Following in the blog series are:

From window killing to screenshot

Last week I concentrated most of my development work on screenshot support through spectacle in a KWin Wayland session. Now I am happy to announce that we merged support for capturing a screenshot of a window with the help of an external application like spectacle.

To explain why this is a great achievement we first need to look at X11. On X11 taking a screenshot of a window is easy. It’s part of the X protocol to read the pixmap data of the root window and you get the position and size of each window. Thus one is able to cut out the window and have it as a screenshot. That’s the most simple variant to do it, spectacle and previously ksnapshot do it differently. More on that later on.

This is the way to screenshot the active window. If one wants to screenshot any window the user needs to select it. Also for that the X protocol contains everything one needs: grab the mouse cursor, get the click and query the x window tree to figure out which window got clicked. Afterwards screenshot it the same way as explained.

For bringing that to Wayland we see some “problems”. Wayland is designed in a sane and secure way matching the security requirements of 2016 and not the security requirements of the 1980s. An application taking a screenshot of another window or of the complete system is nowadays not acceptable any more. And there is no built in way to take a screenshot – neither of fullscreen nor of a window.

And even if there were, it wouldn’t help much. Information about other windows are not available. One cannot get the information of which is the active window or what is the window under the current mouse position. Also an application is not able to grab the mouse and get the click anywhere on the screen. Mouse is always only available on the current window.

So overall quite some obstacles to take a screenshot and we see that it will need support from the Wayland compositor:

  • Selection of the window
  • Taking the screenshot
  • Communication protocol with the application

Luckily KWin is partially already prepared for it. Even on X11 KWin provides a screenshot functionally to spectacle. A few years ago we wanted to have something better than the standard X screenshots. We wanted to have a window completely taken without overlapping windows and the decoration with the shadows included. Shadows are in case of KWin rendered by the compositor and not part of the X windows. So in order to screenshot it needed support from the compositor, just like on Wayland.

Unfortunately we didn’t think of something like “a successor for X11” back then and designed the interaction more in a way suited for KSnapshot than for not X. We used a DBus API which passed the window id of the window to screenshot as argument and as a result sent an XPixmap to ksnapshot.

Overall not suited for Wayland, but a very good starting point as we already have the screenshot functionality available. So what we needed is a an X free DBus protocol and a way to select the window from within KWin.

Just like spectacle also KWin has a functionality to select a window through mouse interaction: the kill window functionality triggered through Ctrl+Alt+Escape. So far this functionality was only available for X11 and X11 windows, we were not able to do the same on Wayland.

For taking the screenshot I wondered whether we could use this functionality in a more generic way: a feature to interactively select a window. This required a slight refactoring. Of course the X11 way to select a window doesn’t help much, but the ideas we have there. The X11 specific code got moved into the X11-standalone Platform plugin and is now invoked through the internal Platform API. It doesn’t directly kill the window any more, but only returns the window which got selected.

A similar interaction code got added for Wayland and now the kill window functionality can be triggered on both X11 and Wayland and can kill both X11 and Wayland windows.
kwin_screenshot_bn1249

Now all we needed was making this functionality aware to the screenshot functionality. And with that we could trigger an interactive way to trigger a screenshot. This keeps the user in control of the process: the user is informed that a screenshot is being taken and informed how to cancel this. This addresses the security concerns we had for taking screenshots. By making the user perform an explicit action we know that the user agreed to taking the screenshot.

Notification about screenshot taking

Now all that was needed is adjusting spectacle. Spectacle is a rather new and modern application which had multiple windowing systems in mind when the implementation started. So far I had not done any work on spectacle and the code base was new to me. Nevertheless in about an hour I had the screenshot selection working:

Spectacle on Wayland

Not everything is supported though. Fullscreen or screen area screenshots are not yet supported. But given that the primary problem is solved now, this will also be addressed soonish.

If you like the work we are doing for Wayland consider participating in our end of year fundraiser.