Tag: Input

Last year I started a blog post series about how input works in KWin/Wayland. This blog post resumes this series by talking about touch input.

Several people wondered why it took so long for this blog post. After all it’s more than a month since the last one. Of course there is a good reason for it. I was reworking parts of the input stack and wanted to discuss the changes with the next post of the input blog post series. Unfortunately there are still a few changes missing, so I decided to nevertheless do the touch input post first.

Touch input is the new kid in the block concerning input events. It’s a technology which was created after X11 got created and thus it is not part of the X11 core protocol. On X11 this makes touch a weird beast. E.g. there is always an emulation to a pointer event. Applications which do not support touch can still be used as the touch events generate pointer events. Now this is actually a huge sacrifice for the API and means that touch feels – at least to me – as a second class citizen in X11.

On Wayland the situation is way better. Touch is part of the core input protocol and does not emulate pointer events. Applications need to support touch in order to get touch events. If an application does not support touch, the touch events won’t trigger any actions. This is a good thing as it means applications need to do something sensible with touch events.

Like with the other events touch events are reported to KWin by libinput. Touch events are quite straight forward. We get touch down events (when a finger touches the screen), touch up events (when a finger gets lifted again) and touch motion events (when the finger moves on a screen). This is fully multi-touch aware, meaning we can follow multiple touch points individually.

The events are sent through KWin’s internal filter architecture like all other events. Currently KWin does not really intercept events yet. We do support touch events on window decoration and KWin’s own internal windows. But in those cases we emulate mouse events. We don’t have any UI elements which would benefit from multi touch events, thus emulating mouse events internally is sufficient for the time being. If in future we add multi touch aware UI elements that would require changes.

In case KWin does not intercept the touch sequence the events are passed on to the KWayland Server component which forwards the events to the Wayland window which is currently receiving touch events. KWin determines the window by using the window at the first touch down of the sequence. While a sequence is in progress the window cannot change.

The touch events are then processed by the application and can provide sensible functionality. E.g. our Plasma calendar supports a pinch-zoom gesture to switch to an overview of all months. This was developed under X11 and just works on Wayland without any adjustments. Good job, Qt devs!

Last week at the Plasma sprint touch gestures were an important discussion point during the last days. We decided which global gestures we want to support in Plasma. We hope to be able to deliver this for Plasma 5.10 on Wayland and will also look to get the same on X11 by reusing the architecture written for Wayland. But this might land in a later release.

Global touch gestures have an interesting and useful feature. When a sequence starts KWin does not know whether that will be a global gesture or a gesture which needs to be forwarded to the applications. Thus all events must be sent to the applications. Once KWin knows that this is a global gesture it can send a cancel event to the application. This informs the application that the touch sequence got canceled. This prevents conflicts between the global and application touch gestures. On X11 this is not so comfortable, so we will have to see how we can support this.

In the last blog post I discussed keyboard input. This blog post will be all about pointer devices – mostly known as “mouse”. Like my other posts in this series, this post only discusses the situation on KWin/Wayland.

Different hardware types

There are different kind of devices which are recognized as a pointer device. We have the classic mouse/trackball like devices and on notebooks we find touchpads. Furthermore there are also absolute positioning pointer devices, which are sometimes found on touch screens.

Given the differences of the devices there are quite a few configuration options available in libinput for pointer devices. There is for example pointer acceleration and many options for touchpads defining how it should behave. We are currently working on a touchpad KCM for Wayland, so it looks like this will return with Plasma 5.9. As explained in the first blog post of this series the configuration options are set as soon as the device is created.

Pointer motion

The pointer devices generate various events and one of them is the motion event. In general there are two kind of motion events: absolute and relative. Most devices like a mouse generates relative motion events which is the reason why this blog post will only focus on them.

Determining new position

A relative motion is a distance vector with an x and y coordinate. It describes how the cursor position should be moved.

So once the event is read from the queue inside KWin the new position needs to be determined. Now it’s not as simple as taking the last position and then adding the motion vector. This could result in the cursor leaving the visual area.

Instead the pointer motion gets validated and constrained. We ensure that the cursor doesn’t leave the visual area and also apply pointer constraints an application window set.

This is a new protocol KWin supports in Plasma 5.9. It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area. In the first case the pointer doesn’t move at all, in the second case it’s only allowed to move in a certain region of the window.

Processing new position

Even in case the pointer motion is constrained in a way that the cursor doesn’t move, the event is further processed. An application might be interested in the relative motion and react to it, even if the cursor doesn’t move.

For the further processing a QMouseEvent is generated and sent through KWin’s input filters just like described for the keyboard case. The pointer motion might be handled inside KWin, e.g. the active screen edges need to know the current position. Or the pointer motion might be forwarded to a window through KWayland.

Updating the focused window

If the pointer moves it might be that the cursor moves from one window to another or from a window to it’s server side decoration. This means that for every pointer position change KWin needs to evaluate the window at the current position.

Compared to keyboard input where KWin only needs to consider the active window this is a rather complicated task. We need to consider input transformation applied to the screen or window, we need to apply input masking on the window, consider the window decoration, check whether the screen is locked, workaround issues in Xwayland prior 1.18, etc. etc.

In the end the method might have determined a new Surface which gained pointer focus. KWin uses KWayland to update the focused pointer surface which ensures that the surface leave and enter events are emitted.

Of course not always when you move the mouse it should update. If a grab is active (pointer button pressed) it won’t update.

Updating the cursor image

If the pointer moved to a new position it might be that the cursor icon changed. This unfortunately might require a roundtrip. One doesn’t know which icon a window wants to use till the motion was sent to the window. A window might react in two ways: it updates the cursor image, or it doesn’t. In the first case KWin gets notified through KWayland that the image changed, in the second case there is no notification at all.

This means KWin doesn’t know what cursor icon is really valid when moving the cursor. So on pointer motion KWin updates the position of the cursor with the current cursor icon, but it might be that a frame later the client updates it. This is so far the only element in Wayland where I have to say that not every frame is perfect. The cursor could show the wrong icon.

When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. Something we have to improve on. Currently we don’t detect hung applications yet as I think we cannot detect them at all due to clients using a dedicated Wayland event thread and thus always happily replying that everything is ok, even if not.

But it might also possible that KWin needs to set a cursor image. E.g. when hovering a window decoration or entering a special mode for selecting a window the cursor image is provide by KWin. KWin loads the cursor image from the theme. Internally KWin tracks the source from where the cursor image should be used. Whether it’s a Wayland window, or the window selection, or an effect setting a specific image.

Updating the actual cursor

The actual update of the cursor position and icon happens through KWin’s internal platform API. Every platform sets the cursor image in a different way. For our primary platform we use the DRM api to up update the position and to update the image – if a new one is available.

For the nested platforms like X11 and Wayland this happens through the windowing system specific calls. The nested platforms don’t allow to update the cursor position – this happens by the windowing system through the pointer motion. The cursor image, though, can be updated.

The virtual platform only knows the concept of a software cursor. That is the cursor gets rendered in the compositor rendering pass. Currently that is only implemented in the QPainter compositor and not yet available in the OpenGL compositor.

Button events

The next event supported by libinput for pointer devices are pointer button press/release events. These events carry the pointer button they triggered for.

Compared to pointer motion the event processing is way more straight forward. The pointer event is either intercepted by one of our event filters (e.g. Alt+Tab) or forwarded to the window currently having pointer focus.

This is a huge improvement over X11. On X11 if KWin wants to exclusively process pointer events it needs to grab the pointer. That implies that the screen cannot be locked. So if e.g. Present Windows is active the screen doesn’t lock. On Wayland this doesn’t matter any more. If Present Windows is active, the screen will lock and get all pointer events. Once unlocked Present Windows is still active. I was quite happy when I was able to add an auto test for that situation.

Axis events

Many pointer devices have one or two axis. On X11 the core protocol implemented axis events as pointer buttons. With Wayland and libinput we now have dedicated axis events telling us which axis got scrolled and the delta. This is a big improvement compared to X11 as it means that “smooth scrolling” is part of the standard and not something added later on through an extension.

The handling is of course very similar to the other events. KWin creates a QWheelEvent and passes it through the various event filters and if no filter intercepted the event, it will be forwarded to KWayland::Server which in turn sends it to the focused pointer surface.

Touchpad gestures

For touchpads we have further events. Libinput does not only recognize motion and press events on touchpads, but is also able to recognize a multi finger gesture. It supports two kind of gestures: swipe and pinch/rotate gesture. KWin gets the gesture events and forwards them through the normal event system. There is a special Wayland protocol which we added support in Plasma 5.9. This allows forwarding the pointer gesture to Wayland applications as the following video demonstrates.

Unfortunately we do not really use these gestures yet. QtWayland doesn’t implement the protocol, so the forwarding doesn’t reach any application and we don’t make use for it yet internally for e.g. global gestures. We are still working on defining the gestures we want to support. I hope we have something for Plasma 5.9, but no promise.

Happy holidays

This is my last blog post for this year. Next year I will continue this series with a blog about touch screen events and maybe also wacom tablet.

I wish everyone happy holidays and a great start into 2017.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

Recently I did some work on the input stack in KWin/Wayland, e.g. implemented pointer gesture and pointer constraints protocol, and thought about writing a blog series about how input events get from the device to the application. In the first blog post I focus on creating and configuring an input device and everything that’s related to get this setup.

evdev

Input events are provided by the Linux kernel through the evdev API. If you are interested in how evdev works, I recommend to read the excellent post on that topic by Peter Hutterer. For all we care about the input event API is too low level and we want to use an abstraction for that.

libinput and device files

This abstraction exists and is called libinput. It allows us to get notified whenever an input device gets added or removed and when an input event is generated. But not so fast. First of all we need to open the input devices. And that’s a challenge.

The device files are normally not readable by the user. That’s a good thing as otherwise every application would be able to read all key events. Getting a key logger would be very trivial in that case.

But if KWin runs as a normal user and the user is not able to read from the device files, how can KWin read them? For this we need some support. Libinput is prepared for the situation and doesn’t try to open the files itself, but invokes an open_restricted function the library user has to provide. KWin does so and outsources the task to open the file to logind. Logind allows one process to take control over the current session. And this session controller is allowed to open some device files. So KWin interacts with logind’s dbus API to become the session controller and then opens the device files through the logind API and passes them back to libinput.

This is the reason why for a full Wayland session KWin has a runtime dependency on logind’s DBus interface. Please note that this does not mean that you need to use logind or systemd. It only means that one process is required which speaks logind’s DBus interface.

Devices in KWin

Now libinput is ready to open the device files and emits an LIBINPUT_EVENT_DEVICE_ADDED event for each device. KWin creates a small facade class for each device type and applies configuration options for it. KWin supports reading the configuration options set by Plasma’s mouse configuration module and has an own device specific configuration file which will soon allow the touchpad configuration module to configure the touchpad on Wayland. Also as part of setting up the device KWin enables LEDs – if the device supports them – for Num Lock and Caps Lock.

All the input devices created by KWin can be investigated in the Debug console (open KRunner, enter “KWin”). KWin reads a lot of information about the device from libinput and shows those in the Debug console. In the input event tab each of the events include the information which device generated the event.

All devices are also exported to DBus with the same properties as shown in the Debug console. This means the configuration can be changed at runtime through DBus. KWin saves the configuration after successful apply and thus ensures that your settings are restored correctly when you restart your system or replug your external device. This is also an important feature to support the touchpad configuration module.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

Following in the blog series are:

• Keyboard input