Creating a Photo-Box with the help of KWin

For a family celebration I wanted to create a “Photo-Box” or “Selfie-Box”: a place where the guests can trigger a photo of themselves without having to use tools like a selfie-stick.

The requirements for the setup were:

  • Trigger should be remote controlled
  • The remote control should not be visible or at max hardly visible
  • The guests should see themselves before taking the photo
  • All already taken photos should be presented in a slide show to the guests

The camera in question supported some but not all of the requirements. Especially the last two were tricky. While it supported showing a slide show of all taken photos, the slide show ended as soon as a new photo was taken. But the camera also has an usb-connector so the whole power of a computer could be taken in.

A short investigation showed that gphoto2 could be the solution. It allows to completely remote control the camera and download photos. With that all requirements can be fulfilled. But by using a computer a new requirement got added: the screen should be locked.

This requirement created a challenge. As the maintainer of Plasma’s lock screen infrastructure I know what it is good at and that is blocking input and preventing other applications to be visible. Thus we cannot just use e.g. Digikam with gphoto2 integration to take the photo – the lock screen would prevent the Digikam window to be visible. Also there is no way to have a slide show in the lock screen.

Which means all requirements must be fulfilled through the lock screen infrastructure. A result of that is that I spent some time on adding wallpaper plugin support to the lock screen. This allowed to reuse Plasma’s wallpaper plugins and thus also the slide show plugin. One problem solved and all Plasma users can benefit from it.

But how to trigger gphoto2? The idea I came up with is using KWin/Wayland. KWin has full control over the input stack (even before the lock screen intercepts) and also knows which input devices it is interacting with. As a remote control I decided to use a Logitech Presenter and accept any clicked button on that device as the trigger. The code looks like the following:

class PhotoBoxFilter : public InputEventFilter {
public:
    PhotoBoxFilter()
        : InputEventFilter()
        , m_tetheredProcess(new Process)
    {
        m_tetheredProcess->setProcessChannelMode(QProcess::ForwardedErrorChannel);
        m_tetheredProcess->setWorkingDirectory(QStringLiteral("/path/to/storage"));
        m_tetheredProcess->setArguments(QStringList{
                                                   QStringLiteral("--set-config"),
                                                   QStringLiteral("output=TFT"),
                                                   QStringLiteral("--capture-image-and-download"),
                                                   QStringLiteral("--filename"),
                                                   QStringLiteral("%Y%m%d-%H%M%S.jpg"),
                                                   QStringLiteral("-I"),
                                                   QStringLiteral("-1"),
                                                   QStringLiteral("--reset-interval")
        });
        m_tetheredProcess->setProgram(QStringLiteral("gphoto2"));
    }
    virtual ~PhotoBoxFilter() {
        m_tetheredProcess->terminate();
        m_tetheredProcess->waitForFinished();
    }
    bool keyEvent(QKeyEvent *event) override {
        if (!waylandServer()->isScreenLocked()) {
            return false;
        }
        auto key = static_cast(event);
        if (key->device() && key->device()->vendor() == 1133u && key->device()->product() == 50453u) {
            if (event->type() == QEvent::KeyRelease) {
                if (m_tetheredProcess->state() != QProcess::Running) {
                    m_tetheredProcess->start();
                    m_tetheredProcess->waitForStarted(5000);
                } else {
                    ::kill(m_tetheredProcess->pid(), SIGUSR1);
                }
            }
            return true;
        }
        return false;
    }

private:
    QScopedPointer m_tetheredProcess;
};

And in addition the method InputRedirection::setupInputFilters needed an adjustment to install this new InputFilter just before installing the LockScreenFilter.

photobox

The final setup:

  • Camera on tripod
  • Connected to an external screen showing the live capture
  • Connected to a notebook through USB
  • Notebook connected to an external TV
  • Notebook locked and lock screen configured to show slide show of photos
  • Logitech Presenter used as remote control

The last detail which needed adjustments was on the lock screen theme. The text input is destroying the experience of the slide show. Thus a small hack to the QML code was needed to hide it and reveal again after pointer motion.

What I want to show with this blog post is one of the advantage of open source software: you can adjust the software to your needs and turn it into something completely different which fits your needs.

Modifier only shortcuts available in Plasma 5.8

Today I’m happy to announce that for the first time we were able to backport a new feature from our Wayland offerings to X11: modifier only shortcut support. This is one of the most often requested features in Plasma and I’m very happy that we finally have an implementation for it.

Modifier only shortcuts mean that an action is triggered if one clicks a modifier key without any other key. By default the Meta (also known as super and windows) key triggers the main application launcher of your Plasma session. But the implementation is way more flexible and allows to use any modifier (ctrl, alt or shift) as a trigger for an action (currently configuration is only possible through manual modification of kwinrc).

The feature was initially implemented for Wayland inside KWin and has been available for a year already. The tricky part is to recognize when only a modifier is pressed. On Wayland KWin gets all key events and passes them through xkbcommon. Given that KWin knows when a key is pressed and knows when only the modifier is pressed. Thus it was possible to implement modifier only shortcuts.

On X11 our global shortcut system does not have such a deep knowledge about all key states. It only registers to the shortcuts one has configured and in general it triggers on the press event, for modifier only we need trigger on the release event. As our X11 shortcut system relies on key grabbing we couldn’t use it for modifier only support: the chances to break applications are too high.

The implementation we have now on X11 reuses the infrastructure setup for Wayland. It uses XInput 2 to listen to all raw key events (which are also delivered if an application grabs the device) and sends those events through our xkbcommon infrastructure just like on Wayland to recognize the pressed keys.

In the past my main concern was that we mis-detect the state and that we trigger the shortcut when that is not wanted, for the wrong key, etc. etc. Also with the new implementation I have these concerns, although I’m very confident that it works correctly. Nevertheless I want to ask you to give a try to this feature and report if you notice any problems. I’m especially interested in setups which change the meta-modifier position through xmodmap and layout changes. If anything is not working as expected please tell us so that we can have an awesome modifier only experience in Plasma 5.8.

In case you use KSuperKey I recommend to disable that. Otherwise it might happen that shortcuts are triggered twice. At that point I want to thank KSuperKey for having filled this gap in our default feature set and providing a solution for our users who wanted to use this feature.

OpenGL changes in KWin compositing

In Plasma 5.8 we will see a few changes in the OpenGL compositor which are worth a blog post.

Debug information

The new KWin debug console gained a new tab for OpenGL information. This shows information comparable to what we know from glxinfo/es2_info. It shows the information gathered from the OpenGL library (version, renderer, etc.) and all the available extensions. This is intended to be useful when for whatever reason glxinfo is not working and one needs to know exactly which extensions are available.

OpenGL information in Debug Console

As you can see in the screenshot: KWin also learned to add decorations around “internal” windows on Wayland. Which means the debug console can now be freely resized and moved around.

Support for llvmpipe

With Plasma 5.8 we finally allow OpenGL compositing through the llvmpipe driver. So far KWin fall back to XRender based compositing when the OpenGL driver uses software emulation. We thought that with XRender based compositing one gets better results than with software emulation. In this release we re-evaluated the situation and came to the conclusion that it doesn’t make sense to continue blocking llvmpipe driver. Due to QtQuick KWin will render parts of its user interface through llvmpipe anyway and also Plasma will render completely through llvmpipe. Thus most of the system is going through that driver. If the system is usable enough for Plasma, it will also be sufficient for KWin’s compositing.

Related to that many of the reasons to not use llvmpipe by default are going away. For example we didn’t want virtual machines to render through llvmpipe, but today also KVM can do accelerated OpenGL through virgl driver. Similar the embedded systems also provide working drivers nowadays, if we think of raspberry pi or odroid – they all can do OpenGL and we don’t have the risk of them going on llvmpipe which would result in a very bad experience.

Last but not least there is the question whether XRender or QPainter compositing is a better solution than llvmpipe. And there I’m not so sure anymore. For XRender it’s possible that the xorg modesetting driver is used in combination with glamor. In that case it would also use llvmpipe. So nothing gained by forcing to XRender.

But still it’s possible that using llvmpipe compositing will result in high CPU usage if KWin has to use it. The solution to this problem is to deactivate what we know to be expensive – all these effects which are not available on XRender because we know it to be too expensive. We don’t want blur effect on software emulation and neither wobbly windows. So we are now working on disabling the effects if we are on software emulation. For this we introduced a way to disable all animations in KWin and a lot of effects are already adjusted. This might also be a handy new features for users who don’t want any animations at all, but still want the compositor enabled.

Last but not least using llvmpipe also exposes performance problems which we normally don’t see with a fast GPU. And as a result we were already able to improve various parts of our rendering stack which will benefit all users – independently of whether llvmpipe is used or not.

Removal of GLX/EGL selection

A change which also got backported to a 5.7 bugfix release is the removal of the selection of GLX or EGL in the compositor settings. Unfortunately EGL is still not a good enough option on X11. Way too often we saw bug reports about rendering being broken and it was caused by using EGL. We decided that it doesn’t make sense to expose a config option which in most cases will result in the users systems being broken. So on X11 the default will stay GLX, while on Wayland the only option is EGL. Of course on X11 the option can still be set manually be editing the config file or by specifying the environment variable to pick EGL or to pick OpenGL ES 2.

Removal of unredirect fullscreen windows

Unredirection of fullscreen windows has been a kind of blue-headed step child in KWin’s compositing infrastructure for a long time. It’s a feature not loved by the developers, not properly integrated, but you have to support it. For those not knowing the feature: it excludes an area from compositing and let’s the fullscreen window be rendered the normal way in X11 (unredirect). The idea is that you get slightly better performance if you bypass the compositor.

The functionality was never fully integrated into the compositor. It was way too easy to break out of the condition (e.g. a tooltip), but at the same time effects which should break it, had no way to do it (e.g. Present Windows should either not activate or end it). The weirdest oddity of the feature is that we had to hard disable it for all Intel drivers due to crashes. We don’t know whether this is still the case but after having had such a bad experience with it in the past, we decided to never turn it on again. Which means it’s a feature not even supported by all devices.

We developers did not spent much time on the feature as we think it doesn’t make much sense as KWin has a better infrastructure in place: blocking compositing. Applications are allowed to specify that compositing should be blocked. This results in KWin shutting down the compositor, freeing all resource related to it (e.g. destroying the OpenGL context), so all power to the running game. As the compositor is shutting down, you don’t have weird interactions like tooltips jumping out or effects not working properly.

There is a standardized way for applications to request this and we see that many games and applications (e.g. Kodi) make use of it. This is the preferred way in our opinion. Given that this mode is fully supported, we decided to remove unredirect fullscreen windows from KWin’s compositor. This streamlines our implementation and gives us one feature to concentrate on and make sure that it works exactly as our users need it. On Wayland the architecture looks different: there is no such thing like unredirect fullscreen, but we can ideally just pass the buffer to the DRM device. The idea is that we do the best optimized way whenever possible.

Support for render devices

Our virtual rendering platform in KWin/Wayland gained a new feature. Like the normal DRM device it now uses udev to find all possible devices and will default to a render device (e.g. /dev/dri/renderD128) if present. If not present it will look for a card device created through the vgem driver. This is a change intended mostly for the use on build.kde.org so that we can properly test OpenGL compositing, but is also a handy feature for running KWin in the cloud on systems with “real” GPUs. If no GPU is present one can easily make KWin pick the virtual device.

Support for restarting the compositor on Wayland

Initially KWin did not support restarting the compositor or even switching the backend on Wayland. But of course we need to support that as well. The OpenGL driver might do a graphics reset after which our infrastructure wants to reinit the complete setup. Thus KWin needed to learn how to restart the OpenGL compositor on Wayland, which will be fully supported in Plasma 5.8.

Why does kwin_wayland not start?

From time to time I get contacted because kwin_wayland or startplasmacompositor doesn’t work. With this blog post I want to show some of the most common problems and how to diagnose correctly what’s going wrong.

First test nested setup

If you want to try Wayland please always first try the nested setup. This is less complex and if things go wrong easier to diagnose than a maybe frozen tty. So start your normal X session and run a nested KWin:
export $(dbus-launch)
kwin_wayland --xwayland

This should create a black window. If it works you can send windows there, e.g.:
kwrite --platform wayland

When things go wrong

Xwayland missing

KWin terminates and you get the error message:

FATAL ERROR: failed to start Xwayland

This means you don’t have Xwayland installed. It’s a runtime dependency of KWin. Please get in contact with your distribution, they need to fix the packaging 😉

Application doesn’t start

KWin starts fine but the application doesn’t show because of an error message like:

This application failed to start because it could not find or load the Qt platform plugin “wayland”in “”.

Available platform plugins are: wayland-org.kde.kwin.qpa, eglfs, linuxfb, minimal, minimalegl, offscreen, xcb.

Reinstalling the application may fix this problem.

This means QtWayland is not installed. Please install it and try again.

Platform plugin missing

KWin terminates and you get the error message:

FATAL ERROR: could not find a backend

This means that the platform/backend plugin is not installed. KWin supports multiple platforms and distributions put them in multiple packages. For X11 you need e.g. on Debian based systems the package kwin-wayland-backend-x11. For the “real thing” you need: kwin-wayland-backend-drm.

XDG_RUNTIME_DIR not set

KWin terminates and you get the error message:

FATAL ERROR: could not create Wayland server

This means that KWin failed to create the Wayland server socket. The most common reason for this is that your environment does not contain the XDG_RUNTIME_DIR environment variable. This should be created and set by your login system. Please get in contact with your distribution. The debug output should also say something about XDG_RUNTIME_DIR.

Platform fails to load

KWin terminates and you get the error message:

FATAL ERROR: could not instantiate a backend

You hit the jackpot. Something somewhere went horribly wrong. Please activate all KWin related debug categories, run it again and report a bug against KWin – ideally for the platform you used. The debug output hopefully contains more information on why it failed. If not we need to try to look into your specific setup.

Trying on the tty

KWin works fine in the nested setup: awesome. In most cases this means that KWin will also work on the DRM device. Before trying: make sure that you don’t use the NVIDIA proprietary driver, if you do: sorry that’s not yet supported. If you are on Mesa drivers everything should be fine.

Log in to a tty and setup similar to nested setup – I recommend the exit-with-session command line option to have a nice defined setup to exit again:

export $(dbus-launch)
export QT_QPA_PLATFORM=wayland
kwin_wayland --xwayland --exit-with-session=kwrite

Ideally this should turn your screen black, put the mouse cursor into the center of the screen and after a short time show kwrite. You should be able to interact with it and when closing kwrite KWin should terminate. If that all works you are ready to run startplasmacompositor. It will work.

But what if not. This is the tricky situation. The most often problem I have heard is that KWin freezes at this point and gdb shows KWin is in the main event loop. This most likely means that KWin tries to interact with logind DBus API, but your system does not provide this DBus interface. Please get in contact with your distribution on how to get the logind DBus interface installed (this does neither require using systemd nor logind). I would like to handle this better, but I don’t have a system without logind to test. Patches welcome.

Running KWin on the weird systems

Yes one can go crazy and try running KWin on devices like the Nexus 5, Virtual Machines or NVIDIA based systems. Here the experience differs and I myself don’t know exactly what is supported on which hardware and in which setting combination. Best get in contact with us to check what works and if you are interested: please help in adding support for it.

Animating auto-hiding panels

With Plasma 5 a change regarding auto-hiding panels was introduced. The complete interaction was moved from Plasma to KWin. This was an idea which we had in mind for a long time. The main idea is to have only one process to reserve the interaction with the screen edges (which is needed to show the panel when hidden) and to prevent conflicts there. Also it allowed to have only one place for providing the hint that there is the panel.

On an implementation level that uses an x11-property protocol between KWin and Plasma to indicate when the panel should be hidden. KWin then does the interaction to hide it and to show on screen edge activation.

Unfortunately from a visual perspective this created a regression. In Plasma 4 the auto-hiding panel was animated with our Sliding-Popups effect, but in Plasma 5 this doesn’t work any more. The reason for that is that our effect system can only animate when a window gets mapped and unmapped. With the new protocol the window doesn’t get unmapped when hidden and not mapped when shown, so our effects are not able to react on it. Technically our Client object is kept instead of being released on unmapped.

Thanks to Wayland this will be working again starting with Plasma 5.8. For Wayland windows KWin keeps the object around when a window gets unmapped and uses the same object when it gets shown again. KWin keeps more state for Wayland windows than for X11 windows. But this also means that our animations are not working for Wayland windows. Last week I addressed this and extended the internal effects API to also support hiding/showing again of Wayland windows. As the effects API does not differentiate between Wayland and X11 windows this change also enables the auto-hiding panel animation. All that was needed was emitting two signals at the right place.

Here Wayland shows another strength: not only does it help to bring features to X11 it also allows us to automate the testing. With a pure X11 system we would have had a hard time to properly auto-test this. But for Wayland we have a nice isolated integration test-framework which allowed to add a test-case for auto-hiding panels.

Synchronizing the X11 and Wayland clipboard

One of the greatest annoyances after switching my work systems to Wayland is the lack of clipboard synchronization between the X11 and Wayland windowing system. X11 clipboard is only available to X11 windows and the Wayland clipboard is only available to Wayland windows. It’s the useability problem of the two X11 clipboard (primary and secondary) just exposed on a much higher and more annoying way. Instead of “just” remembering whether you copied or selected text, you need to remember which windowing system you copied from and which windowing system the current window is in. And if they are not in the same, there is just no way.

So getting this problem resolved got quite a priority. It was something I had never really spent any thought on and incorrectly assumed it’s something Xwayland would take care of. But after thinking about the problem I realized that the X11 clipboard is part of ICCCM and not a core feature of the X-Server. It’s a communication protocol between clients.

This means that KWin had to learn to communicate with the X11 clipboard in order to provide the synchronization. Not an easy task as ICCCM is quite complex on this topic and implementing an X11 clipboard is nothing I’m looking forward to. Luckily I remembered that we have a clipboard manager for X11 (klipper) and that this one works without having X11 specific code thanks to Qt’s abstraction.

And this makes the whole thing way easier: by using Qt’s X11 clipboard implementation we only need to interact with Wayland’s clipboard which is thanks to KWayland quite straight forward. The solution involves a small helper binary which gets started by KWin. It’s forced to the xcb windowing system to be able to use Qt’s X11 clipboard and manually connects with KWin through KWayland. The task of the binary is quite simple: whenever the X11 clipboard changes it gets the data and sets it on the Wayland clipboard. Similar whenever the Wayland clipboard changes it gets set on the X11 clipboard.

Unlike the X11 clipboard on Wayland clients are normally not informed about clipboard changes. Only the application currently having keyboard focus gets informed by the Wayland server about what kind of data is available on the clipboard. This means we don’t need to inform all Wayland clients about changes in the X11 clipboard and similar we don’t need to inform X11 about clipboard changes in Wayland all the time. Instead KWin helps the process: when an X11 window gains keyboard focus our helper process gets informed about the current Wayland clipboard content, so that it can be available for the focused X11 window and while X11 windows are focused clipboard changes from the helper application are forwarded to Wayland. So when the next Wayland window gets focused it has the clipboard content set on X11. A small and simple solution which works quite well in practice. At the moment only the “real” clipboard is synchronized. There is not yet the concept of the selection based clipboard on Wayland. A solution might be to just synchronize it to the normal clipboard. But this would again introduce the useability problems of this non-explicit clipboard. So I’m not yet convinced what’s the proper way.

In my last blog post I pointed to our current donation campaign to support the Randa developer sprint. A lot of readers donated after reading this. I want to thank everybody who donated. It’s so important for a free software community to meet in person and we need donations for getting developers to sprints and conferences. Last week I was at openSUSE conference and had a talk about how Wayland improves the quality of Plasma. These are important things to talk about at a conference, but it’s only possible if we have the donations to send people there. Please help to make such developer sprints possible. Thank you!

Wayland in Plasma 5.7

Last week we released the beta version of Plasma 5.7 which means we know what this release will have for better Wayland support. First of all I need to mention what didn’t make it: unfortunately I missed the freeze of Frameworks 5.23 to land support for xdg-shell. I have a working implementation, but was not yet satisfied with the API. This is a difficult interface to provide an API for due to the unstable nature of the interface. Due to lack of xdg-shell support GTK applications are still going to use X11 on Wayland (like the Firefox window I’m just typing this blog post in).

In the past I already blogged about a few new features in 5.7 like the improved task manager for Wayland, the virtual keyboard support, sub-surface support and improved input device support. So in this blog post I want to focus on a different topic: quality.

For Plasma 5.7 my aim was to get the Plasma session into a state that I can use it as my primary system. And since last week I have not started into an X11 session any more. This means that we needed to get the whole system stable enough to have neither KWin nor applications crash due to Wayland. Given that our Wayland code is quite a fair amount of new code, changing lots of assumptions there are of course bugs to be expected. We still have code which calls into X11 unconditionally, we have things which are not implemented correctly and of course we do stupid mistakes. So for Plasma 5.7 the task was to find these issues and fix one for one.

For Wayland it’s much easier for us to test. KWayland – our framework for Wayland support – is developed in a test driven approach making it possible to create test cases for every problem. They expose the problems and verify that they are fixed and as regression tests ensure that they won’t hit us again. Over the last release cycle we added several thousand lines of test code in KWayland alone.

Finding those issues is not always easy. If KWin crashes we don’t have DrKonqi like normally, it doesn’t work for Wayland (tries to connect to a display server, but that just crashed). What I saw on my Wayland test device was that KWin sometimes randomly crashed – more often when I interacted with X11 windows. But when attaching gdb to KWin it didn’t crash. But once I caught it: it turned out to be an error in KWin in the handling of Xwayland windows. There are two possible code paths it can take and one was with a mistake. Due to running through a debugger it was more likely to take the correct one. So yeah it’s not always easy.

With that problem gone we were able to find a few more and fix also some bugs which caused windows to quit. Unfortunately some of this fixes had to go into KWayland after the 5.23 release. This means the frameworks version used with Plasma 5.7 is not going to have all fixes. If you want to give Plasma/Wayland a try I recommend to not just wait for Plasma 5.7 but also for frameworks 5.24.

This week I will be at the openSUSE conference in Nuremberg, where I will also give a talk about how Wayland helps us to improve our quality and our workflows. I’ll do another blog post about the content of that presentation – don’t want to spoil 😉 Though if you follow our development you are already aware of it. Thanks to openSUSE to give me the possibility to present at the conference and thanks to KDE e.V. for the support to go there.

I can hear you asking now the question of all questions: “When will it be ready?” I think that I am not objective enough to answer the question or to say that it is ready. I’m too close to the code and might just omit important problems because I don’t see them. Thus I cannot say that it is ready. It depends on your workflow and whether that workflow is already fully implemented. This is something only you can know.

Last week KDE had a very important developer sprint (where I did not participate) and is currently running a fundraising campaign for this sprint. We need the money to send our developers to such meetings or to a conference like openSUSE conf where I will be this week. At the moment just 107 people have participated and donated. This is something which makes me sad. I see the statistics for my blog posts and know that this one will have at least 1000 direct hits. In addition there are people reading planetkde and not directly my blog. We are trying to raise 24000 EUR and please do the math yourself to see how close we would be to it if everybody would donate just 10 EUR.

A task manager for the Plasma Wayland session

Plasma 5.7 will ship with a new taskmanager library. One of the reasons to implement a new taskmanager library is the port to Wayland. Our old code base was heavily relying on X11 and didn’t support the concept of multiple windowing systems. You can read more on that in Eike’s blog post about the new task manager. In this blog post I want to focus a little bit on the Wayland side of a task manager.

Architectural difference to X11

On X11 every application is able to read every aspect of every other window. This allows to implement a task manager. The task manager can get notified when a window is added, when a window is removed and install event filters for the changes of the window properties. That’s the basics for a task manager.

On Wayland the world is different. An application is not able to see the windows of other applications. This means by default Plasma is not able to see what windows exists, the taskmanager would be empty. The only application which knows about the existing windows is the Wayland compositor.

Merging Desktop Shell and Wayland compositor?

A possible solution to this problem could be to merge the desktop shell and the compositor into one process. This is the architecture used in e.g. Unity and GNOME Shell on X11.

But such an architecture has also disadvantages. It violates the concept of separating concerns and of “doing one thing and do it right”. We have an excellent task manager, an excellent window manager and an excellent desktop shell.

Also it would make code sharing between platforms more difficult. A task manager for Wayland would be very different from a task manager on X11. All the business logic and presentation should be the same. No matter which windowing system is used.

By merging the code bases together we would also make the code less reusable. One couldn’t use KWin/Wayland in another desktop environment (e.g. LXQt) nor could one use Plasma/Wayland with another compositor (e.g. sway). While we of course recommend to use KWin as the Wayland (and X11) compositor we don’t enforce it and also don’t want to enforce it.

Protocols, protocols

So a different solution is needed and that is done through a custom Wayland protocol which is called org_kde_plasma_windowmanagement. We have a few interfaces prefixed with Plasma. This does not mean that it can only be used by Plasma. It just means that the interface was first designed and developed for Plasma’s needs. We are happy to share the interface with other desktop environments.

The protocol is provided by KWin and it announces whenever a new window gets created. Once a new window gets created a bound client can bind an org_kde_plasma_window interface and get notified about all the states of the window.

The protocol exposes pretty much all the information KWin has about the window and it gets updated automatically whenever the state in KWin changes. In addition the protocol has requests the task manager needs, like “close the window”, “minimize it”, “maximize it”, etc. etc. KWin also listens to these requests and honors them.

Although the protocol is added to Wayland, it is windowing system agnostic. The created Plasma Window does not expose the actual windowing system resource (after all on Wayland another client cannot get access to it). KWin exposes a Plasma Window for both X11 managed windows and Wayland windows. By that the task manager is able to manage tasks from multiple windowing systems without knowing that it does so.

Evolution of the protocol

We added the Plasma window protocol initially for the work on the Plasma phone about a year ago. Also on the Plasma phone we had the use case of switching windows and using the X11 based task manager was just no option on a Wayland only device 😉

While the protocol worked fine for our initial needs, it was not yet sufficient for the use in the desktop task manager. So over the last weeks we introduced further support and improved the API to make it work more reliable. Also quite some changes went into KWin to ensure that only “interesting” windows are exposed. E.g. a context menu should not be shown in the task manager.

Our KWayland library of course has support for this protocol through the classes PlasmaWindowManagement and PlasmaWindow on client side. In addition there is a model exposed as PlasmaWindowModel.

Future work

The work on the interface is not done yet. There are still changes to the API needed and the upcoming KDE Frameworks 5.23 release will include new API calls in PlasmaWindow. As well we have already the first change scheduled for KDE Frameworks 5.24. And we also know that we need to expose a few more data elements in the protocol to get the task manager to the same level as on X11.

There is also an interesting idea floating around to use the – windowing system agnostic – Wayland implementation on both X11 and Wayland. For this KWin (on X11) would need to create a dummy Wayland server for the task manager to connect to. It’s an idea which we might or might not implement.

Security considerations

Currently every Wayland client is able to bind this interface, which means that some of the security improvements of Wayland are not available in Plasma/Wayland. We are fully aware of that and were also fully aware of the consequence when we added the interface. I do have ideas on how to address this and this will be implemented before we will recommend the Plasma/Wayland session for daily usage. The design allows to add security checks in these areas. Unfortunately my priority list did not allow me to implement this for Plasma 5.7, next chance 5.8.

Virtual keyboard support in KWin/Wayland 5.7

Over the last weeks I worked on improved input device support in KWin/Wayland and support for virtual keyboard. KWin 5.7 will integrate the new QtVirtualKeyboard module which is now available under GPLv3. For us this means that we have access to a high quality QML based keyboard. For Qt it means that the virtual keyboard is exposed to more users and thanks to the open source nature it means that we can upstream fixes.

The virtual keyboard is integrated into KWin/Wayland making it part of our platform. To support it we implemented the text input interface in KWayland. If an application supports it, then the virtual keyboard can input text from the compositor side.

Video on youtube:

As you can see in the video the virtual keyboard is only enabled by default if no hardware keyboard is available. If a hardware keyboard gets plugged in the virtual keyboard gets disabled automatically. In addition there is a Status Notifier Item registered which allows to enable/disable the virtual keyboard independently of the hardware keyboard state. So on a notebook with a touch screen it’s possible to get the virtual keyboard even if there is (of course) a hardware keyboard.

Implementing support for the virtual keyboard was an interesting journey which required many unexpected changes and improvements all over the place. First of all wl_text_input interface needed to be implemented as that’s what is used in QtWayland 5.6. At one point I hit a wall and saw that the implementation in QtWayland is incomplete. So I checked how it looks like in QtWayland 5.7 and alas the complete implementation changed, it’s using zwp_text_input_unstable_v2, which is not even in wayland-protocols. I’m not happy about QtWayland using not standardized interfaces as I think that’s very bad for compatibility. Nevertheless I also added support for it in KWayland as otherwise we would not be able to communicate with applications using Qt 5.7. As the QtVirtualKeyboard will only be available with Qt 5.7, we need to support Qt 5.7’s interface as well. In KWayland this is implemented in a transparent way and KWin announces support for both interfaces.

Another area which needed lots of work is the support for input devices in KWin. We need to properly detect whether a real keyboard is available. We needed to add further support for touch events in KWin. So thanks to the integration of virtual keyboard KWin now supports touch much better on Wayland.

The improvements for input devices are also useful in other areas. E.g. today I landed a change to not show the cursor image if no pointer device (mouse or touchpad) is connected.

Improved input device support in KWin/Wayland

One of the areas I’m currently working on is improving our support for input devices in KWin. While general usage is already quite good in Plasma 5.6, we are not yet able to configure the input devices. This is something I’m working on to improve for Plasma 5.7.

Input devices are provided by libinput and KWin integrates with that for quite some time. What it didn’t do yet is to configure them and keep track of them.

So as a first step we created a Device wrapper class which encapsulates a libinput device in a way that we can easily use it from inside KWin. The Device allows us to gain knowledge about what it supports and also to configure it.

The used Devices and their states can be queried through KWin’s new Debug Console:
Input Devices in Debug console

The Debug Console is also new in Plasma 5.7 and allows us to have a deeper introspection into KWin. It can be opened through KRunner (Alt+Space) and then with the command “KWin”. The functionality set is inspired by tools like xprop (one tab allows to read all window properties), xwininfo -tree (one tab shows a (sub)surface tree), xev (input events).

Back to input devices. Once we had the Device wrapper class in place we could start making use of it to configure devices. And today I landed a series of commits in KWin master which bring us back many features from X11. From the configuration point of view it’s really easy for us as we can just reuse our existing configuration tools. For example the mouse configuration module writes to a config file which KWin can read. When the settings change the configuration module sends out a DBus signal which KWin uses to trigger a reconfigure.

So KWin now supports setting pointer devices (mouse) to left-handed usage and modifying the pointer acceleration.
Toggle Touchpad

In addition KWin started to integrate changes for touchpads. KWin listens to the global shortcut for touchpad toggle and uses this to enable/disable the touchpad. Of course this comes with triggering the new on-screen-display notification for enabling the touchpad. Next step will be to integrate more options for touchpads, but this will require writing a kwin-backend for our touchpad control module.