How input works – pointer input

In the last blog post I discussed keyboard input. This blog post will be all about pointer devices – mostly known as “mouse”. Like my other posts in this series, this post only discusses the situation on KWin/Wayland.

Different hardware types

There are different kind of devices which are recognized as a pointer device. We have the classic mouse/trackball like devices and on notebooks we find touchpads. Furthermore there are also absolute positioning pointer devices, which are sometimes found on touch screens.

Given the differences of the devices there are quite a few configuration options available in libinput for pointer devices. There is for example pointer acceleration and many options for touchpads defining how it should behave. We are currently working on a touchpad KCM for Wayland, so it looks like this will return with Plasma 5.9. As explained in the first blog post of this series the configuration options are set as soon as the device is created.

Pointer motion

The pointer devices generate various events and one of them is the motion event. In general there are two kind of motion events: absolute and relative. Most devices like a mouse generates relative motion events which is the reason why this blog post will only focus on them.

Determining new position

A relative motion is a distance vector with an x and y coordinate. It describes how the cursor position should be moved.

So once the event is read from the queue inside KWin the new position needs to be determined. Now it’s not as simple as taking the last position and then adding the motion vector. This could result in the cursor leaving the visual area.

Instead the pointer motion gets validated and constrained. We ensure that the cursor doesn’t leave the visual area and also apply pointer constraints an application window set.

This is a new protocol KWin supports in Plasma 5.9. It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area. In the first case the pointer doesn’t move at all, in the second case it’s only allowed to move in a certain region of the window.

Processing new position

Even in case the pointer motion is constrained in a way that the cursor doesn’t move, the event is further processed. An application might be interested in the relative motion and react to it, even if the cursor doesn’t move.

For the further processing a QMouseEvent is generated and sent through KWin’s input filters just like described for the keyboard case. The pointer motion might be handled inside KWin, e.g. the active screen edges need to know the current position. Or the pointer motion might be forwarded to a window through KWayland.

Updating the focused window

If the pointer moves it might be that the cursor moves from one window to another or from a window to it’s server side decoration. This means that for every pointer position change KWin needs to evaluate the window at the current position.

Compared to keyboard input where KWin only needs to consider the active window this is a rather complicated task. We need to consider input transformation applied to the screen or window, we need to apply input masking on the window, consider the window decoration, check whether the screen is locked, workaround issues in Xwayland prior 1.18, etc. etc.

In the end the method might have determined a new Surface which gained pointer focus. KWin uses KWayland to update the focused pointer surface which ensures that the surface leave and enter events are emitted.

Of course not always when you move the mouse it should update. If a grab is active (pointer button pressed) it won’t update.

Updating the cursor image

If the pointer moved to a new position it might be that the cursor icon changed. This unfortunately might require a roundtrip. One doesn’t know which icon a window wants to use till the motion was sent to the window. A window might react in two ways: it updates the cursor image, or it doesn’t. In the first case KWin gets notified through KWayland that the image changed, in the second case there is no notification at all.

This means KWin doesn’t know what cursor icon is really valid when moving the cursor. So on pointer motion KWin updates the position of the cursor with the current cursor icon, but it might be that a frame later the client updates it. This is so far the only element in Wayland where I have to say that not every frame is perfect. The cursor could show the wrong icon.

When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. Something we have to improve on. Currently we don’t detect hung applications yet as I think we cannot detect them at all due to clients using a dedicated Wayland event thread and thus always happily replying that everything is ok, even if not.

But it might also possible that KWin needs to set a cursor image. E.g. when hovering a window decoration or entering a special mode for selecting a window the cursor image is provide by KWin. KWin loads the cursor image from the theme. Internally KWin tracks the source from where the cursor image should be used. Whether it’s a Wayland window, or the window selection, or an effect setting a specific image.

Updating the actual cursor

The actual update of the cursor position and icon happens through KWin’s internal platform API. Every platform sets the cursor image in a different way. For our primary platform we use the DRM api to up update the position and to update the image – if a new one is available.

For the nested platforms like X11 and Wayland this happens through the windowing system specific calls. The nested platforms don’t allow to update the cursor position – this happens by the windowing system through the pointer motion. The cursor image, though, can be updated.

The virtual platform only knows the concept of a software cursor. That is the cursor gets rendered in the compositor rendering pass. Currently that is only implemented in the QPainter compositor and not yet available in the OpenGL compositor.

Button events

The next event supported by libinput for pointer devices are pointer button press/release events. These events carry the pointer button they triggered for.

Compared to pointer motion the event processing is way more straight forward. The pointer event is either intercepted by one of our event filters (e.g. Alt+Tab) or forwarded to the window currently having pointer focus.

This is a huge improvement over X11. On X11 if KWin wants to exclusively process pointer events it needs to grab the pointer. That implies that the screen cannot be locked. So if e.g. Present Windows is active the screen doesn’t lock. On Wayland this doesn’t matter any more. If Present Windows is active, the screen will lock and get all pointer events. Once unlocked Present Windows is still active. I was quite happy when I was able to add an auto test for that situation.

Axis events

Many pointer devices have one or two axis. On X11 the core protocol implemented axis events as pointer buttons. With Wayland and libinput we now have dedicated axis events telling us which axis got scrolled and the delta. This is a big improvement compared to X11 as it means that “smooth scrolling” is part of the standard and not something added later on through an extension.

The handling is of course very similar to the other events. KWin creates a QWheelEvent and passes it through the various event filters and if no filter intercepted the event, it will be forwarded to KWayland::Server which in turn sends it to the focused pointer surface.

Touchpad gestures

For touchpads we have further events. Libinput does not only recognize motion and press events on touchpads, but is also able to recognize a multi finger gesture. It supports two kind of gestures: swipe and pinch/rotate gesture. KWin gets the gesture events and forwards them through the normal event system. There is a special Wayland protocol which we added support in Plasma 5.9. This allows forwarding the pointer gesture to Wayland applications as the following video demonstrates.

Unfortunately we do not really use these gestures yet. QtWayland doesn’t implement the protocol, so the forwarding doesn’t reach any application and we don’t make use for it yet internally for e.g. global gestures. We are still working on defining the gestures we want to support. I hope we have something for Plasma 5.9, but no promise.

Happy holidays

This is my last blog post for this year. Next year I will continue this series with a blog about touch screen events and maybe also wacom tablet.

I wish everyone happy holidays and a great start into 2017.

If you want to support our work, consider donating to our Make the World a Better Place! – KDE End of Year 2016 Fundraising campaing.

17 Replies to “How input works – pointer input”

  1. “When entering a window the cursor is not defined. Till the window sends a cursor image it is not set. KWin doesn’t render the cursor then and this means that when entering a frozen window we don’t have a cursor. ”

    Can’t KWin just use the default cursor image until gets an event from the application? Or does the specification explicitly says “if the surface doesn’t send a cursor image, no cursor image shall be used”?

    I wish you a happy new year Martin!

    1. What if the application does not want to set a cursor image? Then we would have a wrong cursor all the time

      1. So does that mean that the cursor will flicker every time it hovers an application window?

        Move cursor to window -> KWin sets the cursor invisible -> vblank -> The app sets the cursor back on -> vblank -> the user sees the cursor flicker.

        I’m not an expert, but that doesn’t sound right to me. I had cursor flickering when I was using a VGA monitor on the integrated Intel GPU while having my Radeon doing the rendering, using PRIME. It was very distracting and enough to stop using PRIME when I didn’t have to.

        Maybe I didn’t understand it properly, but very few apps need to make the cursor disappear, so defaulting to show a cursor for a frame or two when it wasn’t supposed to appear (or the wrong one), it’s better than flickering cursors all the time. Or maybe not, you know more about this stuff in the end 🙂

      2. The issue with the courser image is the lack of information at the time you leave one surface and enter the next. I think, this lack cannot be fixed in an economic way. However, …

        Right now under X, the courser image changes instantaneously. If this would be replaced by an animation, you could start the animation in a generic fashion, and from the second frame on, the animation could be direction towards the target image.

        I think, such an animation should be fast and not distracting. I am not an usability expert, but KDE has a great usability team to iron out the details. If done right, such an animation could even make the desktop feel more natural.

        Thanks for all your work and your nice explanations thereof. I wish everyone a nice winter break.

  2. Looking at how the current state of touchscreens is a bit disappointing on Linux, I am looking forward to the article on how they will be handled in the future (and maybe understand why they are poorly supported now).

    1. Same here, specially since touchscreen enabled PC’s tablets and in between are showing up more and more. I do own one of those said devices and I was quite disappointed when I launched a few live USBs.

  3. Love this blog series :-).

    A question arises for me:
    “It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area”
    I’m a bit confused about this, the only usecase coming to my mind would be games. But why would the window manager handle this? Isn’t that something which would actually need handling in the wayland protocol?

    1. Well, the window manager is the Wayland compositor, so it’s actually handling the Wayland protocol. KWin is the process that takes care of the cursor moving around and all that.

  4. I have a different question relating to:

    “It allows a Wayland window to either lock the pointer to position or to confine the pointer to an area”

    A situation where an application is allowed to lock the pointer might actually be very bothersome… In case it is also allowed to lock the keyboard it would make the computer unusable. I wonder if there should be some notification to the user which optionally allows him to cancel the lock or tells him how to exit it…

    1. of course we considered that. Holding escape cancels the pointer constraint and we will show a notification (in work)

  5. Especially the Touchpad gestures are really exciting. Lots of things can be done with that. Since the Release anouncement for 5.9 has this specifically called out, I guess you cam up with something?

    Half-OT: Maybe you can write a blogpost in the near future about the current situation of multiple screen support in Plasma and the upcoming developments for this year.
    I will definitely get a new machine in the second half of it and some more detailed info about how projectors and dockingstations work might be useful for my choice of hardware. Also, I read some really depressing stories about the current situation which I cannot judge myself due to the lack of the software. Is Multi screen support in Plasma/qt as bad as many people say?

Comments are closed.