How input works – pointer input

In the last blog post I discussed keyboard input. This blog post will be all about pointer devices – mostly known as “mouse”. Like my other posts in this series, this post only discusses the situation on KWin/Wayland.

Different hardware types

There are different kind of devices which are recognized as a pointer device. We have the classic mouse/trackball like devices and on notebooks we find touchpads. Furthermore there are also absolute positioning pointer devices, which are sometimes found on touch screens.

Given the differences of the devices there are quite a few configuration options available in libinput for pointer devices. There is for example pointer acceleration and many options for touchpads defining how it should behave. We are currently working on a touchpad KCM for Wayland, so it looks like this will return with Plasma 5.9. As explained in the first blog post of this series the configuration options are set as soon as the device is created.

Pointer motion

The pointer devices generate various events and one of them is the motion event. In general there are two kind of motion events: absolute and relative. Most devices like a mouse generates relative motion events which is the reason why this blog post will only focus on them.

Determining new position

A relative motion is a distance vector with an x and y coordinate. It describes how the cursor position should be moved.

So once the event is read from the queue inside KWin the new position needs to be determined. Now it’s not as simple as taking the last position and then adding the motion vector. This could result in the cursor leaving the visual area.

Instead the pointer motion gets validated and constrained. We ensure that the cursor doesn’t leave the visual area and also apply pointer constraints an application window set.

This is a new protocol KWin supports in Plasma 5.9. It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area. In the first case the pointer doesn’t move at all, in the second case it’s only allowed to move in a certain region of the window.

Processing new position

Even in case the pointer motion is constrained in a way that the cursor doesn’t move, the event is further processed. An application might be interested in the relative motion and react to it, even if the cursor doesn’t move.

For the further processing a QMouseEvent is generated and sent through KWin’s input filters just like described for the keyboard case. The pointer motion might be handled inside KWin, e.g. the active screen edges need to know the current position. Or the pointer motion might be forwarded to a window through KWayland.

Updating the focused window

If the pointer moves it might be that the cursor moves from one window to another or from a window to it’s server side decoration. This means that for every pointer position change KWin needs to evaluate the window at the current position.

Compared to keyboard input where KWin only needs to consider the active window this is a rather complicated task. We need to consider input transformation applied to the screen or window, we need to apply input masking on the window, consider the window decoration, check whether the screen is locked, workaround issues in Xwayland prior 1.18, etc. etc.

In the end the method might have determined a new Surface which gained pointer focus. KWin uses KWayland to update the focused pointer surface which ensures that the surface leave and enter events are emitted.

Of course not always when you move the mouse it should update. If a grab is active (pointer button pressed) it won’t update.

Updating the cursor image

If the pointer moved to a new position it might be that the cursor icon changed. This unfortunately might require a roundtrip. One doesn’t know which icon a window wants to use till the motion was sent to the window. A window might react in two ways: it updates the cursor image, or it doesn’t. In the first case KWin gets notified through KWayland that the image changed, in the second case there is no notification at all.

This means KWin doesn’t know what cursor icon is really valid when moving the cursor. So on pointer motion KWin updates the position of the cursor with the current cursor icon, but it might be that a frame later the client updates it. This is so far the only element in Wayland where I have to say that not every frame is perfect. The cursor could show the wrong icon.

When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. Something we have to improve on. Currently we don’t detect hung applications yet as I think we cannot detect them at all due to clients using a dedicated Wayland event thread and thus always happily replying that everything is ok, even if not.

But it might also possible that KWin needs to set a cursor image. E.g. when hovering a window decoration or entering a special mode for selecting a window the cursor image is provide by KWin. KWin loads the cursor image from the theme. Internally KWin tracks the source from where the cursor image should be used. Whether it’s a Wayland window, or the window selection, or an effect setting a specific image.

Updating the actual cursor

The actual update of the cursor position and icon happens through KWin’s internal platform API. Every platform sets the cursor image in a different way. For our primary platform we use the DRM api to up update the position and to update the image – if a new one is available.

For the nested platforms like X11 and Wayland this happens through the windowing system specific calls. The nested platforms don’t allow to update the cursor position – this happens by the windowing system through the pointer motion. The cursor image, though, can be updated.

The virtual platform only knows the concept of a software cursor. That is the cursor gets rendered in the compositor rendering pass. Currently that is only implemented in the QPainter compositor and not yet available in the OpenGL compositor.

Button events

The next event supported by libinput for pointer devices are pointer button press/release events. These events carry the pointer button they triggered for.

Compared to pointer motion the event processing is way more straight forward. The pointer event is either intercepted by one of our event filters (e.g. Alt+Tab) or forwarded to the window currently having pointer focus.

This is a huge improvement over X11. On X11 if KWin wants to exclusively process pointer events it needs to grab the pointer. That implies that the screen cannot be locked. So if e.g. Present Windows is active the screen doesn’t lock. On Wayland this doesn’t matter any more. If Present Windows is active, the screen will lock and get all pointer events. Once unlocked Present Windows is still active. I was quite happy when I was able to add an auto test for that situation.

Axis events

Many pointer devices have one or two axis. On X11 the core protocol implemented axis events as pointer buttons. With Wayland and libinput we now have dedicated axis events telling us which axis got scrolled and the delta. This is a big improvement compared to X11 as it means that “smooth scrolling” is part of the standard and not something added later on through an extension.

The handling is of course very similar to the other events. KWin creates a QWheelEvent and passes it through the various event filters and if no filter intercepted the event, it will be forwarded to KWayland::Server which in turn sends it to the focused pointer surface.

Touchpad gestures

For touchpads we have further events. Libinput does not only recognize motion and press events on touchpads, but is also able to recognize a multi finger gesture. It supports two kind of gestures: swipe and pinch/rotate gesture. KWin gets the gesture events and forwards them through the normal event system. There is a special Wayland protocol which we added support in Plasma 5.9. This allows forwarding the pointer gesture to Wayland applications as the following video demonstrates.

Unfortunately we do not really use these gestures yet. QtWayland doesn’t implement the protocol, so the forwarding doesn’t reach any application and we don’t make use for it yet internally for e.g. global gestures. We are still working on defining the gestures we want to support. I hope we have something for Plasma 5.9, but no promise.

Happy holidays

This is my last blog post for this year. Next year I will continue this series with a blog about touch screen events and maybe also wacom tablet.

I wish everyone happy holidays and a great start into 2017.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.