Category Archives: planetkde

Porting Qt applications to Wayland

During Akademy I hold a session about porting applications to Wayland. I collected some of the general problems I saw in various KDE projects and want to highlight them in this blog post, too.

Starting Wayland

First the basics: how to start a Wayland compositor? Assuming one is using Plasma 5.4 (aka master) the way to start a nested KWin is:

kwin_wayland --xwayland

Important: ensure that you have Xwayland installed!

If you run nested this will open a window, if it turns black, everything works. If it doesn’t turn black: contact me (please check whether you use free drivers first).

As an alternative you can also start Weston:

weston

If you run into issues with your application running under Wayland, please always test whether the problem is also visible on Weston before blaming me :-P

Starting applications on Wayland

Now we have our black KWin window and we want to get a window to it. There are several ways to start it. As an example I’ll use kate.

Environment variable

Maybe the easiest way is to just use an environment variable to switch the QPA platform to Wayland:

export QT_QPA_PLATFORM=wayland
kate

And a Kate window will open on the nested KWin.

The environment variable is set to wayland in our startplasmacompositor script to start a full Plasma on Wayland session on tty.

Command line switch

Every Qt application also has a command line switch to select the windowing system:

kate --platform wayland

Starting applications together with KWin

KWin supports starting applications once the Wayland and Xwayland servers are started. You can add the programs to start to the command which starts KWin:

kwin_wayland --xwayland "kate --platform wayland"

Or to simplify with environment variables:

export QT_QPA_PLATFORM=wayland
kwin_wayland --xwayland kate

Of course it’s also possible to just start a Konsole and then launch applications from inside the Konsole. Please note that any key combination with ctrl-key is unfortunately broken when running Konsole as a Wayland application.

X11 Specific code

Many applications have X11 specific code. Being it implicitly or explicitly. So we need to move away from it and in most cases it means also improving the situation on X11 because the solution is wrong.

Opening X connection

A common mistake I saw in Qt applications is to open a connection to the X server. E.g. like that:

Display *dpy = XOpenDisplay(NULL);

Now this code is already wrong on X11 as it opens another connection which is not the Qt display connection. It might be on the wrong head, it might be on the wrong X server. From an integration point of view it’s broken.

So the solution is to always use QX11Info from QtX11Extras to integrate with the Display:

Display *dpy = QX11Info::display();

Runtime checks in addition to compile time checks

Before Qt5 the way to support multiple windowing systems was to use compile time switches, like:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    XFlush(QX11Info::display());
}
#endif

Now this will crash on Wayland. It will be compiled in, but crashes when running on Wayland. So you need to improve. Let’s try:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    Display *dpy = QX11Info::display();
    if (!dpy) {
        return;
    }
    XFlush(dpy);
}
#endif

Now this code will also crash! The problem is that QX11Info::display() goes through Qt’s native interface and uses the key “display”. QtWayland also provides the key “display” and returns a wl_display for it, which gets casted into a void in the native interface and then to Display* in QX11Info and boom at interesting places.

The proper way is to check the platform name:

#if HAVE_X11
#include <QX11Info>

void doSomethingWithX11()
{
    if (!QX11Info::isPlatformX11()) {
        return;
    }
    XFlush(QX11Info::display());
}
#endif

My recommendation is to keep the check in a local variable if the check is needed multiple times as in the end it’s a string comparison each time. Ideally you should restructure your code to have platform specific subclasses or similar refactored solutions.

General Changes with Wayland

Now here it becomes interesting: runtime changes. The most important information for application developers is probably that there are no more global window ids.

No global window ids

This has quite some implications for your application. E.g. QWindow::winId() returns an internal incremented number. Neither the compositor nor any other application knows anything about your internal window id counter. Similarly QWindow::fromWinId is not able to map a window id to a QWindow any more. This of course affects use cases like sending a window id through DBus to another process to use it for setting the transient hint. Also it means that KWindowSystem API has a hard time to support the calls on Wayland and in most cases it will just print a warning (this might improve, though).

Setting window icon

On X11 the applications exported the window icon as pixmap data. This is no longer possible on Wayland. Instead the compositor maps the window icon from the desktop file and the application announces the desktop file for a window. In Qt this is done by using the reverse domain name combined with the binary name. The domain name is set with KAboutData and defaults to “kde.org”. So for example in the case of kate the desktop file should be called “org.kde.kate.desktop”. So please fix the naming of your desktop files. If you want to test please be aware that this requires Qt 5.5 to work properly.

No longer working API

Quite some of the API calls of QWindow cannot be supported by Wayland on default (some of them might work on KWin, but please don’t implement against KWin!).
A short list of those API calls I know that they cannot work:

  • QWindow::setPosition
  • QWindow::setMouseGrabEnabled
  • QWindow::setKeyboardGrabEnabled
  • QWindow::alert
  • QWindow::lower
  • QWindow::raise
  • QWindow::requestActivate

In addition it’s no longer possible to grab the image of the screen using e.g. QScreen::grabWindow (most obvious for the case of “root” window). Also warping the pointer using QCursor::setPos is no longer supported. Please note that warping the pointer is also a bad idea on X11 and you shouldn’t do that.

In case your application set the Qt::BypassWindowManagerHint on your QWindow/QWidget you need to do some porting as QtWayland doesn’t show such windows. Unfortunately that needs a case by case evaluation of the use case and the solution I presented during my Akademy talk should not be applied for applications.

Porting to Wayland

Don’t port

The solution to port to Wayland is: DON’T PORT! Mostly your X11 specific code or behavior is wrong and could be solved in better ways. Instead of porting you might want to rethink your problem or change to existing abstracted API in e.g. Qt or KWindowSystem. I have seen many cases where X11 API was used as a workaround by e.g. raising windows, force activating them and so on. If one things about it one realizes that there seems to be an obvious problem. So don’t try to port that but rethink your problem and improve not just on Wayland, but also on X11.

But I really need platform specific code

Ok, ok, I get it! Now let’s start with the obvious: your code needs to be compile time and runtime checked. Let’s start with how to not do it:

#if HAVE_X11
    if (!QX11Info::isPlatformX11()) {
         return;
    }
    // here goes the X11 specific code
#else
    // here goes the Wayland specific code
#endif

The problem with this code snippet is that X11 support is probably also available when you use Wayland. Also the else part is not just Wayland, but also any other platform.

As I said the proper solution is to make it compile and runtime checked:

#if HAVE_X11
    if (QX11Info::isPlatformX11()) {
        callX11SpecificImplementation();
    }
#endif
#if HAVE_WAYLAND
    if (QGuiApplication::platformName().startsWith(
            QLatin1String("wayland"), Qt::CaseInsensitive)) {
        callWaylandSpecificImplementation();
    }
#endif

Be aware that your code might also run on other platforms on Wayland, e.g. eglfs or other custom windowing systems. So don’t crash if you don’t hit any of the code paths ;-)

Plasma Phone and KWin

As you are probably aware by now we announced the Plasma Phone project during Akademy this weekend. In this blog post I want to discuss the role of KWin in Plasma Phone.

Plasma Phone uses Wayland as the windowing system with KWin being the Wayland compositor. This is our first product which uses Wayland by default and also the first product which uses KWin as the Wayland compositor. The phone project pushed the Wayland efforts in Plasma a lot and is the only reason why we are able to make Wayland a technological preview with the upcoming Plasma 5.4 release.

The phone project gave KWin/Wayland into the hands of many developers who started to actively use it and to develop with it. This obviously helped to find lots of small and larger issues which then could be fixed. It triggered a stabilization process which reached a stage that I can use a Plasma Wayland session on my notebook with enough confidence that I won’t lose data due to crashes. So thanks to the whole team for pushing the system to the limits.

An area which saw lots of work thanks to the Phone is the interaction between Plasma as the desktop shell and KWin as the Wayland compositor. With Wayland we keep the architecture of having a dedicated shell process which is not inside the compositor. This architecture has served us well in the past and we don’t see a reason why we should change this. It means that KWin can serve as a compositor for other desktop projects, but it also means that Plasma can be used with other compositors. Now unlike X11, Wayland’s protocols are secure by default. This means that Plasma as the desktop shell does not know anything about windows from other processes. To solve this problem we designed a special window management protocol which exports the required information. The protocols are still under development, but we hope that they can be also useful for other projects with a similar architecture (LXQt being an obvious candidate). Of course such protocols should only be used by the “blessed” desktop shell process – this is something we still need to implement and one of the reasons why at the moment Plasma/Wayland is only a technological preview.

Window management is not the only area where the shell process needs to be “blessed”. Also for announcing information about its own windows, we need some more information. We need to know whether a window is a “panel” or the “desktop” view. So on the phone the shell background is just a desktop window, the panel on the bottom is just a normal dock. This allows us to share the general layering code with the X11 implementation of KWin. Even more having the panels marked as panels allows us to properly define the maximized area. And the windows on the phone open maximized by using the “Maximizing” placement strategy in KWin.

Working on the phone project also had some surprises. For example the virtual keyboard just didn’t want to show. It turned out that the reason for this was that Qt(Wayland) only requests the virtual keyboard if it has keyboard focus. Keyboard focus inside KWin is bound to the “active” window. So we had to implement activating Wayland clients. But the initial implementation didn’t really solve it, we still had areas where the keyboard didn’t come up. E.g. on the desktop in the KRunner search field we couldn’t get it to show. The reason was related to the described problem: we did not yet support activating Wayland clients when clicking on them. This got solved by implementing mouse action (and touch action) support inside KWin. So thanks to this change (done for the phone) we can properly switch between Wayland windows on the desktop with a mouse driven setup.

Another nice touch is that KWin requires a running Xwayland server. This gives us support for all “legacy” X11 applications such as Qt 4 or GTK2 on the phone. Just we hit a problem with them: they did not react on touch events. The reason for this is quite simple: touch support is not yet implemented in Xwayland. As a solution we implemented simulating pointer events in case a connected Wayland window does not bind the touch interface. Thus all Wayland and X11 applications can now be used with a touch screen – be it on the phone or on a notebook with a touch screen.

So far I have only spoken about progress made in KWin which is relevant for the desktop. So what about the phone specific adjustments? Well there are hardly any. In the core of KWin there is no phone specific code. The only phone specific code is the hwcomposer backend. This is a plugin used for creating an OpenGL context using libhybris and for reading input events (unfortunately libinput cannot read events on a libhybris enabled system). According to cloc this plugin is just about 800 lines of code and about 200 are just very straight forward mapping of Android key codes to “normal” Linux key codes. For comparison: KWin + kwayland/server currently have about 120,000 lines of code. And even this hwcomposer backend is not phone specific. It could also be used to run a normal KWin/Plasma session on any libhybris enabled device. There is one important difference of the plugin to the rest of KWin which is worth to mention: it is GPLv3+ licensed, while everything else is GPLv2+. The reason for this change is the fact that libhybris is Apache 2 licensed and this license requires a change to GPLv3.

Plugins, plugins, plugins – or how KDE Frameworks learned to Wayland

Several of KDE’s frameworks have windowing system specific code. This means that they need adjustments if they should be run on more than just X11. The frameworks were already adjusted to only call into X11 specific code if they are used on the X11 platform, but a proper Wayland port was still missing.

Over the last weeks I prepared several frameworks to support Wayland in a proper way. Especially I looked at KGlobalAccel, KIdleTime and KWindowSystem. For all three frameworks there is no “generic” Wayland solution available. That is there is no official interface available which could be used. So in order to add support we need to restrict the usage to KWin for the moment. This is especially the case for KGlobalAccel where the aim is to integrate directly into KWin as described in a previous blog post.

For KWindowSystem I wanted to reuse some Plasma-specific interfaces which we had been developing for the Plasma 5.4 release anyway. As those interfaces are part of the KWayland repository it was an obvious idea to reuse KWayland instead of duplicating the code. But using KWayland from a framework – especially a tier 1 framework is not trivial. KWayland is part of Workspace, thus depending on KWayland would create circular project dependencies. Turning KWayland into a framework would adjust the tier for pretty much all frameworks. Clearly neither a good idea.

Thinking about the problem one realizes there is another project with similar problems and they solved it: Qt. QtWayland provides plugins for Qt and by that qtbase itself does not need to depend on Wayland. So for our frameworks plugins can also be a good solution. We can provide plugins which depend on the respective frameworks and KWayland. As KWin would (for the time being) the only compositor to provide the interfaces anyway, it’s not a problem to depend on the KWayland libraries which are part of Workspace.

So as a first step the frameworks got restructured to have their platform dependent code provided by plugins which get loaded at runtime depending on the platform they are run in. In a second step the header files for the plugin interface needed to be installed. This is a requirement for the integration plugins to provide a plugin.

Now as a third step the development of the interfaces was required. This happened in a new repository called kwayland-integration which will see the initial release with Plasma 5.4. So far this repository provides two plugins: one for KIdleTime, one for KWindowSystem.

For KIdleTime I designed a specific Wayland protocol which allows us to get notified when a seat went idle for a specified interval. This is an important feature for many applications, like screen brightness, setting chat to away, etc. Given that I think it’s an interface which might be in general useful and I think it could be a candidate for inclusion in Wayland. This is something I will look into after Akademy when I have more time for it.

KWindowSystem is a little bit more Plasma/KWin specific, so I doubt that we can have anything which is in general useful for all Wayland compositors. KWindowSystem is also on X11 a little bit KWin/Plasma specific, so there is no general difference.

Nevertheless platforms should provide plugins for these frameworks if they want to support frameworks based applications properly. That’s especially a hint to all those distributions who decided to create their own windowing system.

For the long term I think the way to go is to make KWayland part of frameworks (targeting tier 1) and then KWayland-Integration also a framework (current dependencies would aim for tier 2, but also tier 4 could be a reasonable target). This would be especially important if the idle time interface becomes a proper Wayland interface and would be provided by more compositors than just KWin.

More information about the bigger picture of KWayland-Integration and Plasma on Wayland at Akademy next weekend :-) We have great Wayland things to show.

Global shortcut handling in a Plasma Wayland session

KDE Frameworks contain a framework called KGlobalAccel. This framework allows applications to register key bindings (e.g. Alt+Tab) for actions. When the key binding is triggered the action gets invoked. Internally this framework uses a DBus interface to communicate with a daemon (kglobalaccel5) to register the key bindings and for getting notified when the action triggered.

On X11 the daemon uses the X11 core functionality to get notified whenever key events it is interested in happen. Basically it is a global key logger. Such an architecture has the disadvantage that any process could have this infrastructure and it would be possible for multiple processes grabbing the same global shortcut. In such a case undefined behavior is triggered as either multiple actions are triggered at the same time or only one action is triggered while the others do not get informed at all.

In addition the X11 protocol and the X server do not know that kglobalaccel5 is a shortcut daemon. It doesn’t know that for example the shortcut to lock the screen must be forwarded even if there is an open context menu which grabbed the keyboard.

In Wayland the security around input handling got fixed. A global key logger is no longer possible. So our kglobalaccel5 just doesn’t get any input events (sad, sad kglobalaccel5 cannot do anything) and even when started on Xwayland with the xcb plugin it’s pretty much broken. Only if key events are sent to another Xwayland client it will be able to intercept the events.

This means a global shortcut handling needs support from the compositor. Now it doesn’t make much sense to keep the architecture with a separate daemon process as that would introduce a possible security vulnerability: it would mean that there is a way how to log the keys. One only needs to become the global shortcuts daemon and there you go. Also we don’t want to introduce a round trip to another application to decide where to deliver the key event to.

Therefore the only logical place is to integrate global shortcut handling directly into KWin. Now this is a little bit tricky. First of all kglobalaccel5 gets DBus activated when the first application tries to access the DBus interface. And of course KWin itself is using the DBus interface. So KWin starts up and has launched the useless kglobalaccel5. Which means one of our tasks is to prevent kglobalaccel5 from starting.

Of course we do not want to duplicate all the work which was done in kglobalaccel. We want to make use of as much work as possible. Because of that kglobalaccel5 got a little surgery and the platform specific parts got split out into loadable runtime plugins depending on the QGuiApplication::platformName(). This allows KWin to provide a plugin to perform the “platform specific” parts. But the plugin would still be loaded as part of kglobalaccel5 and not as part of KWin. So another change was to turn the functionality of kglobalaccel into a library and make the binary just a small wrapper around the library. This allows KWin to link the library and start kglobalaccel from within the KWin process and feed in its own plugin.

Starting the linked KGlobalAccel is one of the first things KWin needs to do during startup. It’s essential that KWin takes over the DBus interface before any process tries to access it (as a good part it’s done so early that the Wayland sockets do not accept connections yet and Xwayland is not even started). We will also try to make kglobalaccel5 a little bit more robust about it to not launch at all in a Plasma/Wayland session.

Now the reader might think: wait, that still gives me the possibility to install a stealth key logger, I just need to create shortcuts for all keys. Nope, doesn’t work. As key events get filtered out a user would pretty quickly notice that something is broken.

Integrating KGlobalAccel into KWin on Wayland brings an obvious disadvantage: it’s linked to KWin. If one wants to use applications using KGlobalAccel on other compositors some additional work might be needed to use their local global shortcut system – if there is some. For most applications this is no problem, though, as they are part of the Plasma workspace. Also for other global shortcut systems to work with KWin it’s needed to port them to use KGlobalAccel internally when running in a Plasma/Wayland session (that’s also a good idea for X11 sessions as KGlobalAccel can provide additional features like checking whether the key is already taken by another process).

Four years later

At beginning of June 2011 I made my first blog post about KWin support Wayland clients featuring a screenshot of Desktop Grid effect with a Wayland window shown on each desktop.

desktopgrid-with-wayland

Now it’s almost four years later and I show once again such a screenshot:

Desktop Grid effect in a Plasma on Wayland session
Desktop Grid effect in a Plasma on Wayland session

A few things have changed, for example the screenshot shows a KWin running on DRM. And there’s one huge difference: the KWin instance is using Wayland internally and no longer X11 (it still uses it for X11 applications, or course). Also KWin is able to properly integrate the Wayland windows. They are not rendered a top of the scene but take up normal spots like all other windows, too. Otherwise they are also more like “normal” windows. The plasmashell in this screenshot is also a Wayland client including all the panels and windows it creates (not visible as desktop grid effect hides panels).

A good question to ask is how much of the initial code written in 2011 ended in todays KWin and the honest answer is none. The branch got never merged and pretty quickly bitrotted. I got the rendering done pretty quickly, but at that time Wayland hadn’t had a stable release yet, so we simply couldn’t merge the code into master as that would have been rather inconvenient for development. With two moving targets (Wayland and KWin) the code diverged too quickly, broke too often and made development difficult. Thus when I heard a stable release of Wayland was planned it didn’t make much sense to continue the work on the branch.

But there was of course one thing which the branch provided: experience. The branch showed that our compositor is up to the task of integrating non-X11 windows, but the rest of KWin wasn’t. Which triggered a refactoring to make KWin more useable with multiple platforms. There were side-effects from that development which went already in some releases: e.g. the reworked scripting helped to identify the interface of a managed Client. Very recently this got split out into a new abstract base class which is now inherited by the good old X11 Client and the new Wayland ShellClient. A different area is screen edge handling which got reworked to not only allow multiple backends but also to have more fine grained per screen edges.

The work to get Wayland integrated into KWin took much longer than expected, part of it was the problem described above of the missing stable Wayland releases. Of course there was also a Qt5 port which I didn’t expect. The Qt5 port required a port to xcb – again something I didn’t expect. But most of all I underestimated the problem scope of how much work would be needed to get the window manager ported to Wayland. Still it will be quite some time till all features which are provided by KWin will be available for Wayland. Too much code is too X11 specific to be directly reusable.

And to get a complete session up and running there is much more to be ported than just KWin. We have Plasma which needs to access window management information provided by KWin (a taskbar should be able to show tasks after all). Some parts need to be moved into KWin for better security: screen locker and global shortcut handling (almost finished) are obvious candidates. We have functionality provided in our libraries which need adjustments: examples are KWindowSystem and KIdleTime. Our power management code need to learn how to talk with KWin to turn the screen off. KScreen needs to learn to interact with KWin to configure screens. And much, much more.

Also many of our applications need to be adjusted to work properly on Wayland. Obviously if one uses low-level X11 calls those won’t work any more. But also when you use higher level abstractions it might be that your feature doesn’t work any more, there are so many things which are X11 specific and you might not even know off. I plan to host a session on “Applications on Wayland” on this years Akademy. Make sure to go there. I’ll show all the “don’t” (and most are don’t on X11, too ;-) ) and show how one can easily setup a Wayland session to test the application.

Now I know that this blog post sounds a little bit like one shouldn’t expect Wayland on Plasma anytime soon. That’s not the case. I’m quite optimistic that I will shift my systems to Wayland as the primary work system before Akademy to properly dog food. And I’m sure that many Plasma developers will soon follow. After all I’m just editing this blog post in a full Plasma on Wayland session (though my Firefox is using XWayland).

And of course there are many, many tasks to work on. Most are really easy and easy to get into. Every contribution matters and helps us to embrace Wayland faster. Setting up a KWin development environment to test the Wayland progress is easy. All you need is to build kwayland and kwin through kdesrc-build (all other dependencies can be provided by your distribution). Once it’s installed just run:

kwin_wayland --windowed --xwayland

and a nice nested Wayland server will be started in your X session.

And if you want to get all down to running KWin directly on DRM like the screenshot above:

kwin_wayland --drm --libinput --xwayland

In all cases you can specify applications to be launched once KWin has fully started. Those can be either X11 or Wayland applications, e.g.:

kwin_wayland --drm --libinput --xwayland "konsole --platform wayland"

Will start a konsole as a Wayland client once can has started.

A recent addition as of today is that you don’t have to specify the commands like windowed or drm any more. KWin will automatically pick the correct backend depending on the environment variables which are exported. But be careful: if you start a kwin_wayland on an X session without DISPLAY exported, it will start the drm backend and that might break your X session!

Turning the world upside down

This blog post is a rather important one for me. Not only is it the first blog post which I write in a nearby cafe sitting in the sun and enjoying spring, it is also the first blog post written in a Plasma session running inside kwin_wayland.

This marks an important step in the process of getting Plasma and KWin ready for Wayland as we have reached a state where I can dogfood kwin_wayland on one of my systems. It means that I’m confident enough to write a blog post without having to fear that the session crashes every few seconds (granted I press save a lot nevertheless).

So what do I mean when saying that I’m running a kwin_wayland session? Does it mean that everything is already using Wayland? No, unfortunately not, rather the opposite: all running applications are still using X11, but we use a rootless Xwayland server. The only Wayland application in this setup is Xwayland and KWin itself.

Nevertheless it’s a noteworthy achievement as we now have the architecture pulled upside down. No longer does KWin connect to either an X-server or a Wayland-server which does the rendering to the screen, it does it now all by itself. Over the last week I worked on the DRM (Direct Rendering Manager) backend for kwin_wayland. The DRM backend will become the primary backend for KWin, doing mode-setting, creating buffers for rendering and performing page flips of those buffers on the connected outputs and rendering the mouse cursor on the outputs. In the current state it’s able to get all connected outputs and perform mode setting on them. In addition it can power both compositors: OpenGL and QPainter. The QPainter one is the rather trivial: all we need are two buffers which we map into a QImage to render on. With each rendered frame the buffers are swapped and the last rendering buffer gets presented.

The OpenGL compositor is slightly more difficult. Here we need to introduce another component: GBM (generic buffer management). Gbm allows us to create an EGLDisplay for our DRM device, create an EGLSurface for each of the outputs and get the buffer from the surface to be passed to DRM for presenting. Unfortunately GBM is currently only available on Mesa and given NVIDIA’s presentation at last years XDC it looks like NVIDIA will come up with a different solution. My hope was that this would have settled at the time when we start implementing the solution, but unfortunately that’s not the case yet. So for the foreseeable future it looks like we will first have a Mesa specific backend, then later on can add a new NVIDIA-specific backend and hope that at some future point in time Mesa also supports NVIDIA’s solution (after all it was presented as a generic solution) and we can drop the GBM backend. The situation is none I like, but at the moment we can only ignore the proprietary drivers.

Another noteworthy detail is how KWin opens the DRM device. KWin runs as the normal non-privileged user – thus is not allowed to open the device. Just like libinput integration the DRM backend uses logind to open the device file. Given that we already have support for that in KWin it was a straight forward, elegant and secure way to implement it. I know, that not everybody will agree with this solution. But of course it’s just a DBus interface and anybody can implement it.

There is still quite some work needed before this backend, which got merged into master today, will be fully functional. For example all outputs are set to (0/0) that is render overlapped sections. Here we will in one way or another integrate with kscreen to get sane defaults from the start. I’m so looking forward to being responsible for screen layouts. If I look at all the terrible things one has to do to keep properly updated on XRandR… Now we won’t render with a wrong viewport because different parts of the stack are stuck in different states and no, we won’t freeze just because applications start and call an XRandR call. There are so many things which “just work” and which have been painful on X11. From my first tests on hardware I’m confident that we will finally have working multi-screen handling which is adequate for 2015.

Of course this is a wonderful time to start working on KWin. There are lots of small tasks to work on. I am constantly adding new tasks to our todo list. And if you ask, I can always provide some more small tasks.

There is no KEndorsements

You might have noticed that my yesterday’s blog post was an April Fool. With this post I want to explain a little bit about the idea of the article.

First of all I want to thank the VDG, especially Jens and Ken, for their support by providing the great mockups. That was kind of the first element to notice that it’s an April Fool: normally I do only integrate screenshots by myself and I would not have been able to show a drawing in Krita.

The idea for this April Fool is based on the reports about the inclusion of an adware tool called superfish on Lenovo notebooks. Especially the initial response by Lenovo claiming that this malicious software got installed “to improve the shopping experience”.

Nevertheless the article contains truth, half truth and thing we would never do. So let’s look at some of the ideas presented and evaluate them. The first idea presented was share buttons integrated in the browser’s window decoration. This is an idea which already got introduced similarily through Share-Like-Connect. Personally I think it’s a good idea if browsers would provide such global buttons and in return filter out all the user tracking share buttons on the web sites. A global area for share buttons directly integrated in the browser could increase the security of users a lot.

The next idea is about monitioring the usage behavior of the user. For this I must say that in a Wayland world KWin would have enough information to do that, but we don’t look at the information at all. We do not care what an application is: we don’t know that a given window is “Krita”, we don’t know that it should be used with a drawing tablet, we don’t analyze any input and just pass it through to the application. KWin is completely ignorant on what the application does with input events passed to them and as ignorant to what a window content looks like. To us a window is just a texture we render to the screen. We don’t know that there is a “drawing area” or a “video” element, it’s just one texture. Similarly we do not know (and care) about input events. All we do is determine which window should get the event and pass it to it. We are quite aware that by passing all input events through KWin we could do evil things. The good thing is that our software is open source and you can see what we do (relevant code file is called input.cpp, events come in from the files under the libinput/ directory).

Last but not least I want to point out that we as a window manager will not integrate advertisement into the window management. We do not care whether you watch a video, we might not even know it, because as explained: we are ignorant about what you do. I consider applications which integrate advertisement without the user’s explicit acknowledgement as malware.

The bad part of my blog post is that I expect these things to happen. And if we look around us we can see that it already happened: adware is unfortunately also available in FLOSS. And in the walled garden’s of the proprietary world we will see more attempts to gain more information about the user’s to provide even better matching advertisement. It will blur more and more.

Enhancing user experience through the Compositor

This article is an April Fool. For an explanation please see the update.

Some time ago Ken introduced the conecept of dynamic window decorations (DWD) and during the last Plasma sprint there was already some work on experimenting with an implementation. DWD are extremely promising to enhance the user experience by morphing the application and window manager scope together.

The basic concept of DWD is to get content of the application inside the scope of the window manager. But what works one way works also the other way: we can enhance the user experience by providing additional information relevant for the window inside the window decoration.

An example for this could be a volume applet for music and audio players. But of course it doesn’t stop there, today is all about the user content and integrating the web. For web browsers we can integrate a global “Like It on Facebook” button. And by analyzing the user behavior we can provide even more help. We can notice if you struggle using Krita and provide a link to where to get a better drawing tablet.

If you use digiKam you might be interested in some books about photogrophy. We know which music you listen to on Amarok and show you that the new album of your favorite music group just got released. Not every application integrates stores in a proper way. This is a shame as it does not provide the best possible user experience and KWin as a window manager can help there to add missing store functionality where it is needed.

Of course this also allows to show you better alternatives. If we see that you struggle with wine emulated applications we can provide the Linux alternative. We can show that new updates are available for your currently used application. If you use sub-par free software applications we can show you were to get a proper (slightly more expensive) application.

But DWD are not the only place where KWin can help to increase the user experience by offering useful content. We always went a great way to make the desktop more useable by for example allowing to click on a splash screen to hide it. I think we can all agree that splash screens are something really useless: look at a stupid window for a few seconds and having to wait till it hides again. Meh.

We can do better: we know which application starts and what it is used for. And as a compositor we can exchange the content. Wouldn’t it be way more awesome to show a short video giving you advice where to buy the latest tips on C++11 book when launching kdevelop?

And there is so much more room to integrate useful content. When starting a video player we can overlay the window with a nice trailer of the latest movie (of course that will be skipable after 5 sec). This is a feature which might also be handy in Present Windows and Alt+Tab. Of course such videos would be perfectly localized as we know your geolocation and we would take great efforts to only offer trailers which might interest you: after all we have baloo to scan your video data and can also look at your Facebook profile. Now trailers are only one part, of course also other areas could be covered. If you have many pictures of cars you might be rather interested in spots about the latest car of your favorite brand instead of move trailers. Of course we will make it possible to configure which areas you are interested in. As we all know KDE is all about configurability.

Why don’t you just…

Relatively regularly I’m asked why I don’t “just” integrate QtCompositor or libweston and call it a day. Well it should be obvious, if it were as simple as “just” we would do that ;-) So instead of replying again and again I thought to give a full blog post explaining the options and why they don’t really suit our needs.

How KWin’s compositor works

Let’s start with a look at the Compositors in KWin:

  • No Compositing (X11 only)
  • XRender (X11 only)
  • QPainter (Wayland only)
  • OpenGL (ES) (X11 and Wayland)

The task of the compositors is to render the windows into a global scene. The information about the windows is taken from the window manager (e.g. the stacking order defines which window is rendered on top) and get transformed by the effect system.

The compositors use platform specific API to get the content of the window and to render it to the screen. E.g. OpenGL can use GLX and use texture from pixmap to map the content of an X11 window to an OpenGL texture and use an XWindow to create the OpenGL context on. The actual compositor which renders doesn’t care whether it’s an X11 Window or a Wayland window or whatever, all it cares about it’s a texture (OpenGL), a pixmap (XRender) or a QImage (QPainter). The platform specific API is abstracted allowing to make the actual compositors completely windowing system agnostic. Designed as a class diagram it looks (simplified) like this:

Now with that much abstraction one could think that there a huge differences between the code and that there is not much sharing going on. But the opposite is the case. The majority of the code is shared. To give an overview I provide the line count of all the files relevant to the OpenGL compositor.

  • Shared code: 3733 (cloc scene_opengl.* abstract_egl_backend.* scene.*)
  • Platforms specific: 1424 (cloc egl* glxbackend.*)

From the platform specific code the glxbackend is about 50 %. The sharing between the egl backends is larger, for the new X11 mode for kwin_wayland the difference is just 20 lines. For QPainter the differences are also rather small – in fact the differences are so small that all is in one source file making something like 200 lines of code for the three backends. Last week I worked on integrating a framebuffer backend into QPainter – although I prior had no idea about it, it was integrated into KWin in less than two hours fully functional with no change to the SceneQPainter at all except writing 50 lines of the backend code.

Now the backends are only part of the whole story. By using our own existing compositor we ensure a similar feature set between X11 and Wayland and can reuse all our effects directly without any adjustments. According to CLOC we have 22000 lines of effect code.

What about QtCompositor

QtCompositor seems to be the natural choice for bringing Wayland support to KWin. After all it’s part of Qt which is what we use in KWin. Except it doesn’t. For a starter it’s not yet a released component which makes it difficult to use for us to develop against, for our packagers to distribute and for our users to test. “But Martin”, you say, “you could develop on it and get it into a release state!”

Yes I could, but would it help KWin? What is the strength of QtCompositor? Providing a QtQuick API to write a Wayland compositor. But KWin doesn’t use QtQuick in the compositor. “But Martin”, you say, “QtQuick is the new cool thing, everybody ports to it, so should you!”. Yes, of course, but it means:

  • Throwing away all our effects
  • Writing new integration code between QtCompositor and the window manager
  • having no code sharing between the Wayland and the X11 compositor
  • No longer being able to provide non-OpenGL compositors as QtQuick requires OpenGL

If I look at this I rather doubt that this would be a “just integrate it”. That looks like a lot of work and duplication to have the same basic window manager for X11 and Wayland. Code sharing would be kind of impossible as the technologies involved are too different.

“But Martin”, you say, “you could at least use QtCompositor for wrapping the Wayland API and use it to map a Wayland buffer to OpenGL!” Yes, that is true and I did consider it and even planned to do so. In the end I decided against it for multiple reasons. First of all this still shows the problem of the unreleased component (yes, I could work on it), but then I wanted to integrate Wayland in KWin with a Facade. Using a facade is something I consider here as rather important. After all we want to use Wayland for quite some time and not run into the situation that we need to port away from a library in a few years. Using a facade makes such a situation much easier. Now if we write a facade anyway what is the benefit of being a facade to QtWayland compared to being a facade to Wayland directly? It just adds another layer of abstraction which might hinder us (and compared we ported lots away from Qt to XCB during the Qt 5 porting removing the abstraction added by Qt). Oh and of course we will have to use Wayland directly, because we will have to use our own protocols for e.g. interfacing with Plasma. Which would mean that we have to interact with two technologies instead of one. Overall I didn’t see an advantage and that’s why KWayland is a facade over Wayland and not over QtCompositor.

What about (lib)weston

“But Martin”, you say, “you could integrate (lib)weston’s backend!”. Yes, weston uses plugins for it’s backends, so in theory it would be possible to load the plugins and interface with them. There were also patches to make this possible through a libweston, but I actually don’t know whether they got integrated (quick check looks like they didn’t).

I would love to have a library which abstracts e.g. kms or framebuffer for me, but honestly I doubt weston is that. Weston was developed as the reference compositor and not as a library to do that. Weston code is always open in my editor, so I know a little bit about the code and can see that it is – well – Weston-specific. I doubt that the code base could be shared in it’s current state. It would need an effort like what has been done for libinput to move this into a well designed API. As long as that doesn’t happen weston’s backends are not directly useable for us.

And even if we would get back to a different feature set. We use QPainter for non-GL rendering, Weston uses pixman. And that is nested into the compositor implementations of Weston. Again: that needs lots of work to get in a state which would allow integration, so overall it’s not a “just use it”.

And all of that does not consider what is more difficult: integrating the C API or writing the code yourself tailored in a way which makes sense in a C++/Qt world.

KWinception

Last week I merged in a few important changes for the upcoming KWin 5.3 release. The rootless Xwayland support is integrated, which means we are a huge step closer to managing Wayland clients. Rendering, input and cursor is already using the Wayland code paths and will be shared with proper Wayland clients. I have already started working on code for that and have it working quite nicely already but decided to delay the integration for Plasma 5.4.

This means that kwin_wayland starts a Wayland server, but except Xwayland there is no application which can properly connect to it. Nevertheless it’s an important step and allows to also test the code in a better way.

In addition I worked on a better support for nesting compositors. So far one had to start Weston in order to test kwin_wayland. This is of course not optimal as it makes the development setup more difficult than it has to be. So last week I looked into developing a new backend which can use an X11 window to render to. This is comparable to Weston which can be used nested on X11. The backend is relatively straight forward: it can render to the created window using either the OpenGL or QPainter compositor, accepts input events and delegates them and passes the cursor from Wayland windows to the X11 window. The tricky part is that we cannot use any of our X11 specific libraries to create the window: we don’t use the xcb QPA, so no support from Qt and we cannot use KWindowSystem as it only allows one xcb_connection and that’s already needed for the Xwayland window manager. So I had to do things myself and as I consider this just a development feature it’s probably the worst X11 client in existance ;-) (Long term KWindowSystem should be changed to support multiple connections, would be very useful for unit tests).

The new backend triggered a small refactoring which makes it easier to implement further backends like e.g. a plain framebuffer or a drm/kms. As a small reminder: there’s an open GSoC idea for implementing the drm/kms backend and application period is to close soon.

Anyway with the nested KWin it’s now possible to create a kwin_wayland instance on the X Server using kwin_x11 and on the kwin_wayland one can create yet another kwin_wayland so that we have a KWinception:

Aus 2015-03-19

To start such a nested compositor use:

kwin_wayland --windowed --xwayland

Please watch the debug output for:

X-Server started on display :1

This tells the id of the created X Server (normally the next free id) and allows to start an application on the nested KWin:
DISPLAY=:1 kwrite

KWin_Wayland picks the windowing system depending on the environment variables which are defined. For DISPLAY it uses X11, for WAYLAND_DISPLAY it uses Wayland. If both are defined Wayland is preferred. If one wants to specify explicitly one can use the command line arguments –x11-display or –wayland-display. E.g.


kwin_wayland --windowed --xwayland --x11-display=:0

For X11 there are also the options –width and –height to specify an initial size (default is currently 1024×768). On Wayland these options are currently not available, the window always opens fullscreen.

Please remember that this is pretty new code and the support is still incomplete. Feature might be missing or broken. This is at the current state expected and because of that we do not yet accept bug reports. If you find a bug which you think should be fixed, please open an editor ;-)