How input works – touch input

Last year I started a blog post series about how input works in KWin/Wayland. This blog post resumes this series by talking about touch input.

Several people wondered why it took so long for this blog post. After all it’s more than a month since the last one. Of course there is a good reason for it. I was reworking parts of the input stack and wanted to discuss the changes with the next post of the input blog post series. Unfortunately there are still a few changes missing, so I decided to nevertheless do the touch input post first.

Touch input is the new kid in the block concerning input events. It’s a technology which was created after X11 got created and thus it is not part of the X11 core protocol. On X11 this makes touch a weird beast. E.g. there is always an emulation to a pointer event. Applications which do not support touch can still be used as the touch events generate pointer events. Now this is actually a huge sacrifice for the API and means that touch feels – at least to me – as a second class citizen in X11.

On Wayland the situation is way better. Touch is part of the core input protocol and does not emulate pointer events. Applications need to support touch in order to get touch events. If an application does not support touch, the touch events won’t trigger any actions. This is a good thing as it means applications need to do something sensible with touch events.

Like with the other events touch events are reported to KWin by libinput. Touch events are quite straight forward. We get touch down events (when a finger touches the screen), touch up events (when a finger gets lifted again) and touch motion events (when the finger moves on a screen). This is fully multi-touch aware, meaning we can follow multiple touch points individually.

The events are sent through KWin’s internal filter architecture like all other events. Currently KWin does not really intercept events yet. We do support touch events on window decoration and KWin’s own internal windows. But in those cases we emulate mouse events. We don’t have any UI elements which would benefit from multi touch events, thus emulating mouse events internally is sufficient for the time being. If in future we add multi touch aware UI elements that would require changes.

In case KWin does not intercept the touch sequence the events are passed on to the KWayland Server component which forwards the events to the Wayland window which is currently receiving touch events. KWin determines the window by using the window at the first touch down of the sequence. While a sequence is in progress the window cannot change.

The touch events are then processed by the application and can provide sensible functionality. E.g. our Plasma calendar supports a pinch-zoom gesture to switch to an overview of all months. This was developed under X11 and just works on Wayland without any adjustments. Good job, Qt devs!

Last week at the Plasma sprint touch gestures were an important discussion point during the last days. We decided which global gestures we want to support in Plasma. We hope to be able to deliver this for Plasma 5.10 on Wayland and will also look to get the same on X11 by reusing the architecture written for Wayland. But this might land in a later release.

Global touch gestures have an interesting and useful feature. When a sequence starts KWin does not know whether that will be a global gesture or a gesture which needs to be forwarded to the applications. Thus all events must be sent to the applications. Once KWin knows that this is a global gesture it can send a cancel event to the application. This informs the application that the touch sequence got canceled. This prevents conflicts between the global and application touch gestures. On X11 this is not so comfortable, so we will have to see how we can support this.

How input works – pointer input

In the last blog post I discussed keyboard input. This blog post will be all about pointer devices – mostly known as “mouse”. Like my other posts in this series, this post only discusses the situation on KWin/Wayland.

Different hardware types

There are different kind of devices which are recognized as a pointer device. We have the classic mouse/trackball like devices and on notebooks we find touchpads. Furthermore there are also absolute positioning pointer devices, which are sometimes found on touch screens.

Given the differences of the devices there are quite a few configuration options available in libinput for pointer devices. There is for example pointer acceleration and many options for touchpads defining how it should behave. We are currently working on a touchpad KCM for Wayland, so it looks like this will return with Plasma 5.9. As explained in the first blog post of this series the configuration options are set as soon as the device is created.

Pointer motion

The pointer devices generate various events and one of them is the motion event. In general there are two kind of motion events: absolute and relative. Most devices like a mouse generates relative motion events which is the reason why this blog post will only focus on them.

Determining new position

A relative motion is a distance vector with an x and y coordinate. It describes how the cursor position should be moved.

So once the event is read from the queue inside KWin the new position needs to be determined. Now it’s not as simple as taking the last position and then adding the motion vector. This could result in the cursor leaving the visual area.

Instead the pointer motion gets validated and constrained. We ensure that the cursor doesn’t leave the visual area and also apply pointer constraints an application window set.

This is a new protocol KWin supports in Plasma 5.9. It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area. In the first case the pointer doesn’t move at all, in the second case it’s only allowed to move in a certain region of the window.

Processing new position

Even in case the pointer motion is constrained in a way that the cursor doesn’t move, the event is further processed. An application might be interested in the relative motion and react to it, even if the cursor doesn’t move.

For the further processing a QMouseEvent is generated and sent through KWin’s input filters just like described for the keyboard case. The pointer motion might be handled inside KWin, e.g. the active screen edges need to know the current position. Or the pointer motion might be forwarded to a window through KWayland.

Updating the focused window

If the pointer moves it might be that the cursor moves from one window to another or from a window to it’s server side decoration. This means that for every pointer position change KWin needs to evaluate the window at the current position.

Compared to keyboard input where KWin only needs to consider the active window this is a rather complicated task. We need to consider input transformation applied to the screen or window, we need to apply input masking on the window, consider the window decoration, check whether the screen is locked, workaround issues in Xwayland prior 1.18, etc. etc.

In the end the method might have determined a new Surface which gained pointer focus. KWin uses KWayland to update the focused pointer surface which ensures that the surface leave and enter events are emitted.

Of course not always when you move the mouse it should update. If a grab is active (pointer button pressed) it won’t update.

Updating the cursor image

If the pointer moved to a new position it might be that the cursor icon changed. This unfortunately might require a roundtrip. One doesn’t know which icon a window wants to use till the motion was sent to the window. A window might react in two ways: it updates the cursor image, or it doesn’t. In the first case KWin gets notified through KWayland that the image changed, in the second case there is no notification at all.

This means KWin doesn’t know what cursor icon is really valid when moving the cursor. So on pointer motion KWin updates the position of the cursor with the current cursor icon, but it might be that a frame later the client updates it. This is so far the only element in Wayland where I have to say that not every frame is perfect. The cursor could show the wrong icon.

When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. Something we have to improve on. Currently we don’t detect hung applications yet as I think we cannot detect them at all due to clients using a dedicated Wayland event thread and thus always happily replying that everything is ok, even if not.

But it might also possible that KWin needs to set a cursor image. E.g. when hovering a window decoration or entering a special mode for selecting a window the cursor image is provide by KWin. KWin loads the cursor image from the theme. Internally KWin tracks the source from where the cursor image should be used. Whether it’s a Wayland window, or the window selection, or an effect setting a specific image.

Updating the actual cursor

The actual update of the cursor position and icon happens through KWin’s internal platform API. Every platform sets the cursor image in a different way. For our primary platform we use the DRM api to up update the position and to update the image – if a new one is available.

For the nested platforms like X11 and Wayland this happens through the windowing system specific calls. The nested platforms don’t allow to update the cursor position – this happens by the windowing system through the pointer motion. The cursor image, though, can be updated.

The virtual platform only knows the concept of a software cursor. That is the cursor gets rendered in the compositor rendering pass. Currently that is only implemented in the QPainter compositor and not yet available in the OpenGL compositor.

Button events

The next event supported by libinput for pointer devices are pointer button press/release events. These events carry the pointer button they triggered for.

Compared to pointer motion the event processing is way more straight forward. The pointer event is either intercepted by one of our event filters (e.g. Alt+Tab) or forwarded to the window currently having pointer focus.

This is a huge improvement over X11. On X11 if KWin wants to exclusively process pointer events it needs to grab the pointer. That implies that the screen cannot be locked. So if e.g. Present Windows is active the screen doesn’t lock. On Wayland this doesn’t matter any more. If Present Windows is active, the screen will lock and get all pointer events. Once unlocked Present Windows is still active. I was quite happy when I was able to add an auto test for that situation.

Axis events

Many pointer devices have one or two axis. On X11 the core protocol implemented axis events as pointer buttons. With Wayland and libinput we now have dedicated axis events telling us which axis got scrolled and the delta. This is a big improvement compared to X11 as it means that “smooth scrolling” is part of the standard and not something added later on through an extension.

The handling is of course very similar to the other events. KWin creates a QWheelEvent and passes it through the various event filters and if no filter intercepted the event, it will be forwarded to KWayland::Server which in turn sends it to the focused pointer surface.

Touchpad gestures

For touchpads we have further events. Libinput does not only recognize motion and press events on touchpads, but is also able to recognize a multi finger gesture. It supports two kind of gestures: swipe and pinch/rotate gesture. KWin gets the gesture events and forwards them through the normal event system. There is a special Wayland protocol which we added support in Plasma 5.9. This allows forwarding the pointer gesture to Wayland applications as the following video demonstrates.

Unfortunately we do not really use these gestures yet. QtWayland doesn’t implement the protocol, so the forwarding doesn’t reach any application and we don’t make use for it yet internally for e.g. global gestures. We are still working on defining the gestures we want to support. I hope we have something for Plasma 5.9, but no promise.

Happy holidays

This is my last blog post for this year. Next year I will continue this series with a blog about touch screen events and maybe also wacom tablet.

I wish everyone happy holidays and a great start into 2017.

If you want to support our work, consider donating to our Make the World a Better Place! โ€“ KDE End of Year 2016 Fundraising campaing.

How input works – creating a Device

Recently I did some work on the input stack in KWin/Wayland, e.g. implemented pointer gesture and pointer constraints protocol, and thought about writing a blog series about how input events get from the device to the application. In the first blog post I focus on creating and configuring an input device and everything that’s related to get this setup.

evdev

Input events are provided by the Linux kernel through the evdev API. If you are interested in how evdev works, I recommend to read the excellent post on that topic by Peter Hutterer. For all we care about the input event API is too low level and we want to use an abstraction for that.

libinput and device files

This abstraction exists and is called libinput. It allows us to get notified whenever an input device gets added or removed and when an input event is generated. But not so fast. First of all we need to open the input devices. And that’s a challenge.

The device files are normally not readable by the user. That’s a good thing as otherwise every application would be able to read all key events. Getting a key logger would be very trivial in that case.

But if KWin runs as a normal user and the user is not able to read from the device files, how can KWin read them? For this we need some support. Libinput is prepared for the situation and doesn’t try to open the files itself, but invokes an open_restricted function the library user has to provide. KWin does so and outsources the task to open the file to logind. Logind allows one process to take control over the current session. And this session controller is allowed to open some device files. So KWin interacts with logind’s dbus API to become the session controller and then opens the device files through the logind API and passes them back to libinput.

This is the reason why for a full Wayland session KWin has a runtime dependency on logind’s DBus interface. Please note that this does not mean that you need to use logind or systemd. It only means that one process is required which speaks logind’s DBus interface.

Devices in KWin

Now libinput is ready to open the device files and emits an LIBINPUT_EVENT_DEVICE_ADDED event for each device. KWin creates a small facade class for each device type and applies configuration options for it. KWin supports reading the configuration options set by Plasma’s mouse configuration module and has an own device specific configuration file which will soon allow the touchpad configuration module to configure the touchpad on Wayland. Also as part of setting up the device KWin enables LEDs – if the device supports them – for Num Lock and Caps Lock.

Input Devices

All the input devices created by KWin can be investigated in the Debug console (open KRunner, enter “KWin”). KWin reads a lot of information about the device from libinput and shows those in the Debug console. In the input event tab each of the events include the information which device generated the event.

Input Devices exported to DBus

All devices are also exported to DBus with the same properties as shown in the Debug console. This means the configuration can be changed at runtime through DBus. KWin saves the configuration after successful apply and thus ensures that your settings are restored correctly when you restart your system or replug your external device. This is also an important feature to support the touchpad configuration module.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

Following in the blog series are:

Virtual keyboard support in KWin/Wayland 5.7

Over the last weeks I worked on improved input device support in KWin/Wayland and support for virtual keyboard. KWin 5.7 will integrate the new QtVirtualKeyboard module which is now available under GPLv3. For us this means that we have access to a high quality QML based keyboard. For Qt it means that the virtual keyboard is exposed to more users and thanks to the open source nature it means that we can upstream fixes.

The virtual keyboard is integrated into KWin/Wayland making it part of our platform. To support it we implemented the text input interface in KWayland. If an application supports it, then the virtual keyboard can input text from the compositor side.

Video on youtube:

As you can see in the video the virtual keyboard is only enabled by default if no hardware keyboard is available. If a hardware keyboard gets plugged in the virtual keyboard gets disabled automatically. In addition there is a Status Notifier Item registered which allows to enable/disable the virtual keyboard independently of the hardware keyboard state. So on a notebook with a touch screen it’s possible to get the virtual keyboard even if there is (of course) a hardware keyboard.

Implementing support for the virtual keyboard was an interesting journey which required many unexpected changes and improvements all over the place. First of all wl_text_input interface needed to be implemented as that’s what is used in QtWayland 5.6. At one point I hit a wall and saw that the implementation in QtWayland is incomplete. So I checked how it looks like in QtWayland 5.7 and alas the complete implementation changed, it’s using zwp_text_input_unstable_v2, which is not even in wayland-protocols. I’m not happy about QtWayland using not standardized interfaces as I think that’s very bad for compatibility. Nevertheless I also added support for it in KWayland as otherwise we would not be able to communicate with applications using Qt 5.7. As the QtVirtualKeyboard will only be available with Qt 5.7, we need to support Qt 5.7’s interface as well. In KWayland this is implemented in a transparent way and KWin announces support for both interfaces.

Another area which needed lots of work is the support for input devices in KWin. We need to properly detect whether a real keyboard is available. We needed to add further support for touch events in KWin. So thanks to the integration of virtual keyboard KWin now supports touch much better on Wayland.

The improvements for input devices are also useful in other areas. E.g. today I landed a change to not show the cursor image if no pointer device (mouse or touchpad) is connected.

Improved input device support in KWin/Wayland

One of the areas I’m currently working on is improving our support for input devices in KWin. While general usage is already quite good in Plasma 5.6, we are not yet able to configure the input devices. This is something I’m working on to improve for Plasma 5.7.

Input devices are provided by libinput and KWin integrates with that for quite some time. What it didn’t do yet is to configure them and keep track of them.

So as a first step we created a Device wrapper class which encapsulates a libinput device in a way that we can easily use it from inside KWin. The Device allows us to gain knowledge about what it supports and also to configure it.

The used Devices and their states can be queried through KWin’s new Debug Console:
Input Devices in Debug console

The Debug Console is also new in Plasma 5.7 and allows us to have a deeper introspection into KWin. It can be opened through KRunner (Alt+Space) and then with the command “KWin”. The functionality set is inspired by tools like xprop (one tab allows to read all window properties), xwininfo -tree (one tab shows a (sub)surface tree), xev (input events).

Back to input devices. Once we had the Device wrapper class in place we could start making use of it to configure devices. And today I landed a series of commits in KWin master which bring us back many features from X11. From the configuration point of view it’s really easy for us as we can just reuse our existing configuration tools. For example the mouse configuration module writes to a config file which KWin can read. When the settings change the configuration module sends out a DBus signal which KWin uses to trigger a reconfigure.

So KWin now supports setting pointer devices (mouse) to left-handed usage and modifying the pointer acceleration.
Toggle Touchpad

In addition KWin started to integrate changes for touchpads. KWin listens to the global shortcut for touchpad toggle and uses this to enable/disable the touchpad. Of course this comes with triggering the new on-screen-display notification for enabling the touchpad. Next step will be to integrate more options for touchpads, but this will require writing a kwin-backend for our touchpad control module.

Libinput integration in KWin/Wayland

Today I pushed my outstanding branch to get libinput support into kwin_wayland. Libinput is a very important part for the work to get a full Wayland session in Plasma which means we reached a very important milestone. As the name suggests it allows us to process input events directly. KWin needs to forward the input events to the currently active application(s) and also interpret them before any other application gets them. E.g. if there is a global shortcut KWin should intercept it and not send it to an application.

Why libinput integration in KWin?

KWin/Wayland already supported input handling by being a Wayland client and connecting to a Seat. But especially for pointer events this was not sufficient at all. We have quite some code where we warp the pointer and Wayland doesn’t support this (and shouldn’t). Warping the pointer is normally considered evil as it can introduce quite some problems if applications are allowed to warp the pointer. E.g. it can create security issues if you start typing your password and a malicious applications warps the pointer to trick you entering your password into a password field of the malicious application. Also from a usabililty perspective it can be problematic as it makes the system behave in an unpredictable way.

On the other hand if the application is a window manager/compositor the need for warping cursors arises. For example the screen edge handling pushes the cursor slightly back which needs cursor warping. Or you can move a window with the cursor keys (hold Control key to have very precise moving) and in these cases we need to warp the pointer. With libinput this is possible again as KWin gets put in control of the input events directly. It completely bypasses the currently used Wayland compositor.

Libinput is also an important piece in the puzzle for a full Wayland session which does not rely on another Wayland compositor. So far KWin/Wayland can only be used in a nested scenario – which is important for development and interesting new possibilities like the idea for a SoK project – but we also want full support without the need for a Wayland session. This means we need to handle input (which libinput does) and need to interact with DRM directly. DRM support is still missing. This could be an interesting GSoC project next year ๐Ÿ˜‰

The merged implementation does not support all of libinput yet. Most important touch screen support is omitted as I don’t have a touch enabled device. I plan to sit down with fellow KDE developers who have a touchscreen enabled device and implement that part together. Also I will contact the VDG to define some global touch gestures to interact with the system (I’m quite interested in having a touch gesture to activate Present Windows). There’s lots of work to be done here and I would welcome any helping hand.

Security challenges

Processing input events directly comes with a slight problem, though: one needs to be root to read the events. And that’s obviously an absolute no-go for KWin. KWin may never ever be executed with root privileges and also not an suid which drops privileges again (which also wouldn’t help in that case but so what). The application has not been designed for running as root. The same is the case for Weston and obviously I looked at how it’s solved there and there is a very neat solution to support the use case we have in logind. The session controller can ask logind to open devices and logind provides a file descriptor to the opened device. In addition logind automatically takes care to close the file descriptors when a VT switch happens, which is extremely convenient for the use cases of Wayland compositors. So obviously I went for this solution as all it needs is connecting to very few D-Bus calls. This means the libinput integration in kwin_wayland will have a runtime dependency to a logind D-Bus interface. Of course this does not affect kwin_x11, neither does it affect kwin_wayland without libinput integration, but of course without libinput you won’t get support for all features. There is one caveat though: logind will blank the VT when the session controller goes away. So make sure to not run kwin_wayland with libinput support on your primary X session. Because of that libinput support must be explicitly enabled with the –libinput command line switch of kwin_wayland.

Current state and downsides of libinput and logind

As libinput does not yet have a stable release, the dependency is still optional and it’s possible to build kwin_wayland without libinput support. This is currently very important for the non-Linux operating systems, which might want to provide kwin_wayland, as libinput only supports Linux.

I hope that libinput will become available on other platforms. At XDC during the BSD presentations I heard at least one presenter touch the topic. So I’m optimistic that in the long run this will happen as we also see that DRM and KMS is nowadays in quite a good shape on the BSDs. For KWin development it’s of course important that we have only one library to interact with. Otherwise it means platform dependent code which is hard to develop and extremely difficult to test for the main developers not using such a platform. So if you want to get kwin_wayland on non-Linux, please consider putting the energy into getting libinput working (challenge is udev) as that will help all systems and not just KWin. After all we want to stand on the shoulders of giants ๐Ÿ˜‰

Logind is in a similar situation. It is developed as a component in systemd, which isn’t available on all systems which run KWin. Luckily we don’t depend on logind directly but only use a subset of a well defined D-Bus interface and that interface can be provided by other tools as well. Something like that is already being worked on for the BSD’s.
Like with libinput, I would much prefer to keep KWin lean and efficient and not complicate the code base and development by including libraries for specific platforms or having security relevant code around. As written above: using suid wrappers is very much a no-no to me. But of course it would be possible to implement the subset of the D-Bus in an independent project and provide it. KWin would happily use it, it just needs someone to write the code. So if enough people care, I’m quite sure that there will be a developer stepping up and writing the code.

I decided to put out a small FAQ here for those who have questions about the implications of the above:

FAQ

Does that mean KWin (and Plasma) depend on systemd?

No.

But it depends on logind?

No. It uses one D-Bus interface provided by logind. It doesn’t care which program is providing this D-Bus interface. It can be logind or logind-shim or the implementation being worked on for the BSDs. Even a small binary just providing the used D-Bus interfaces would work.

You should not use logind, there must be a different solution!

I’m sorry I did not find any solution which was as efficient and secure as the one provided by logind. Of course there are solutions like weston-launch, but they introduce a lot of complexity – both on the coding side and on the installation side. As such a solution would need to be suid, I’m very reluctant to the idea. We shouldn’t introduce such possible security risks, if there are better solutions available. Logind is simply providing a feature which is needed by kwin_wayland.

Does that affect KWin on X11?

No, that only affects kwin_wayland.

But there is no logind for the BSDs! So I won’t be able to run kwin_wayland on BSD systems?

Unfortunately the fact that logind is missing is least of your problems on BSD. Logind support is only needed for libinput which right now is not available on BSD. The kwin_wayland binary on BSD will not try to interact with logind. I’m sorry I don’t have a solution for the input stack on BSDs. I really hope the BSD developers can come up with a solution for this as we don’t have the resources to build a separate input solution for one platform.

How can I change KWin to not use logind?

As I noted, it is important to me that KWin is secure and that the code base is as easy to understand as possible. I don’t like the idea of having ifdefs all over the place and multiple solutions as that results in bitrot. When I pushed the libinput change it directly failed to build on the CI system as the ifdefs introduced a variation which I couldn’t test on my system. Each ifdef and each platform-specific solution increases the development and maintenance costs significantly. This means that I will only accept patches which don’t introduce the above mentioned problems. Preferrable a small wrapper binary could provide the needed D-Bus interface for KWin and other applications which need this functionality. This would not need changes in KWin at all and would be from my perspective the perfect solution.

Why won’t you implement such a wrapper binary?

Honestly there are a million things I would do if I had time, but a day has only 24 h and I have to prioritize my work. Just check the Wayland TODO list for what we all need to do to get KWin/Wayland ready. Why don’t you open your editor and get some work done? ๐Ÿ˜‰

But if KWin uses logind, Slackware will drop all of KDE!

Yes, I have read that (see comments). Luckily the situation for Slackware is similar to the BSDs: it doesn’t matter right now. Slackware doesn’t provide Wayland packages yet, so the logind support won’t be used as there is no kwin_wayland binary which could be built. And if enough people care, as I said, one or more of them can write the wrapper binary and KWin and other compositors will work just fine.

How can i help?

Best by writing code ๐Ÿ™‚ See the TODO list I linked in an above answer. Also it would be good if someone documented the steps to get kwin_wayland running and how to develop on it cough.