Posted on

A Qt Platform Abstraction plugin for KWin

In Qt we have the Platform Abstraction (QPA) which allows to better interact with the used windowing system through a plugin. In case of KWin we use the “xcb” plugin on X11 and on Wayland we used to use the “wayland” plugin provided by QtWayland. For quite some time I had been thinking about migrating away from those and use an own KWin-specific plugin at least for Wayland.

What’s wrong with QtWayland?

QtWayland provides a plugin for Wayland clients, but KWin is not a Wayland client: it is the Wayland server. This created some interesting “problems” which we had to workaround in KWin. KWin just doesn’t fit the usecase of QtWayland QPA plugin and it would not be a good idea to change QtWayland in a way that it supports KWin as a first-class citizen – it’s just a too special use case. So let’s look at some of the issues which motivated me to write an own QPA plugin.

Connecting to the Wayland server

The wayland plugin is meant for Wayland clients and tries to connect to the Wayland server during the creation of the QGuiApplication. As KWin is the Wayland server the plugin wants to connect to, we needed to make sure that the Wayland server and the event loop are up and running before constructing the QGuiApplication. This added a restriction that we have to monitor the server socket from the main thread.

Blocking roundtrips

The wayland plugin performs a blocking roundtrip to the Wayland server during startup to wait for all outputs to be announced. But this happens from the main gui thread, in which also the server lives. You can guess what happens: dead-lock. We fixed this one in QtWayland, but the problem was still there: any blocking call from the main gui thread to the Wayland server would freeze KWin. We experienced this with e.g. eglInitialize, startup of dbus services (e.g. kded, kamd), rendering of Qt internal windows in case the compositor didn’t send the frame rendered yet. Overall I found this aspect way to fragile and consider especially the fact that the server is in the main thread as a problem.

BypassWindowManagerHint

KWin uses the Qt::BypassWindowManagerHint on all it’s windows because this is needed on X11. The hint is X11 specific and doesn’t make much sense on other platforms, so the Wayland plugin doesn’t show such windows at all. That is a valid approach – there is nothing I could say against it. So in KWin we needed to remove the hint for all windows on Wayland.

OpenGL Context for QtQuick

QtQuick requires an OpenGL context and if it cannot create one it will abort. That is normally not a problem but can easily become one in the case of KWin: KWin creates a “low-level” OpenGL context using the specific platform (e.g. DRM/GBM, hwcomposer, whatever). Qt(Wayland) doesn’t know which one KWin uses and might not even have code for it. QtWayland tries to create an OpenGL context using the EGL platform Wayland. In the case of Mesa everything works fine, but we noticed that on libhybris the creation of a Wayland context fails if there is already one created for hwcomposer. One could consider this as a libhybris specific issue, but I wouldn’t say so – interacting with multiple EGL platforms from one process is kind of out of the specification.

Window decorations

This is a small issue: QtWayland creates window decorations around all Qt windows. But in the context of KWin we do not want any decorations. So we had to disable them by setting the QT_WAYLAND_DISABLE_WINDOWDECORATION environment variable.

And here is the plugin

So after Akademy I decided to work on it and see whether we can get a plugin working. Now it’s in a state where it works good enough to blog about it. The plugin does not support everything that might be used by Qt, but rather reduces to the feature set used by KWin. For example we don’t need to forward any input events, because KWin (mostly) handles them internally anyway and sends them directly to the respective internal Qt window.

The plugin brings the advantage that it no longer depends on the Wayland server being started before creating the QApplication. In fact it does not even try to connect to it, but can use KWin’s internal connection. Similar we can use this to shortcut e.g. creation of low level windows, etc.

On the OpenGL side the plugin approach shows it’s true strength. As it can access KWin internals we can use the EGLDisplay used by the compositing scene and create a sharing OpenGL context. So we can bypass the windowing system completely to render QtQuick windows and this will also allow us to make our window textures available to QtQuick. So a nice improvement.

In case our compositor doesn’t use OpenGL the plugin creates an OpenGL context for the platform Wayland. This is done in a way that it doesn’t hit the blocking issue I mentioned above.

So is the plugin something you should use in other applications?

The plugin interacts with KWin internals and thus is not in any way useable for non-KWin applications. In fact it even checks whether the running binary is KWin and does nothing if that’s not the case. It will also never be turned into a generic plugin as there is no need for that (just use QtWayland).

When will this plugin get merged?

The plugin can be found in branch “qpa” on my personal kwin clone on git.kde.org. It’s not yet in a state that I could push it to master. I still need to test more and make sure that everything works as intended.

Now to something else!

KDE is currently running a fundraiser for the Randa Meetings with the topic “Bring Touch to KDE!”. I personally will not participate in the Randa Meetings, but many other KDE developers will go there and we need your support for it. Please help and make the Randa Meetings possible.

Fundraiser

Posted on

Should we target EGL as the default?

When we started the compositing work in KWin the only way to initialize an OpenGL context was by using GLX. In fact GLX is even part of the OpenGL library on Linux. Being an X11 window manager and an X11 compositor it was not a big problem.

Years later we started the effort to get KWin using optionally OpenGL ES in addition to normal OpenGL. One of the differences is that OpenGL ES doesn’t use GLX, but rather EGL. So we gained code to setup compositing using EGL. But this was still a compile time switch and only supported together with using OpenGL ES. If I remember correctly the option to create on OpenGL context on top of EGL was not yet available in our drivers back then.

But there was a time when the FLOSS drivers started to support it and since the 4.10 release in beginning of 2013 we can provide a normal OpenGL context over EGL. This option was not really exposed and only bound to an environment variable. Very few users knew about it and this was also wanted. Using EGL was a good way to shoot yourself in the foot given that not all drivers support it and that we had much more testing on the GLX backend.

Now the situation has changed. With 5.x we expose EGL also through a config interface, but more importantly EGL is the only way to get OpenGL on Wayland. The assumption that every user is using GLX doesn’t hold any more. The assumption that we don’t need to spend so much time (as it’s only relevant in the reduced feature set of mobile devices) on the EGL backend is wrong. It must reach the same quality as the GLX backend, otherwise our Wayland experience suffers.

So this raises an important question: should we try targeting EGL as the default? In the past one of the main reasons to not even think about EGL as default is the reason that the NVIDIA driver didn’t support it. If we would have switched to EGL by default it would have meant dealing with the situation that EGL doesn’t work and we need to fall back to GLX. It would have complicated the startup significantly and would have resulted in our user base using different rendering paths.

Now this one road blocker seems to resolve itself. While I haven’t tested it yet, the latest NVIDIA beta driver supports OpenGL over EGL, so we should be able to have the majority of our user base on the same backend after all. Given that it is only a beta driver and the feature is marked as “experimental” it’s probably still some time till we can default to it.

Nevertheless it’s time to start thinking about it. I encourage NVIDIA users to give EGL now a try and to report issues. Also I encourage all other users to report issues they see with the EGL backend. If we get the EGL backend into a good state maybe 5.6 might be a good point in time to default to it and then start to slowly deprecate the GLX backend with the aim to get it removed in the distant future.

Posted on

Plasma Phone and KWin

As you are probably aware by now we announced the Plasma Phone project during Akademy this weekend. In this blog post I want to discuss the role of KWin in Plasma Phone.

Plasma Phone uses Wayland as the windowing system with KWin being the Wayland compositor. This is our first product which uses Wayland by default and also the first product which uses KWin as the Wayland compositor. The phone project pushed the Wayland efforts in Plasma a lot and is the only reason why we are able to make Wayland a technological preview with the upcoming Plasma 5.4 release.

The phone project gave KWin/Wayland into the hands of many developers who started to actively use it and to develop with it. This obviously helped to find lots of small and larger issues which then could be fixed. It triggered a stabilization process which reached a stage that I can use a Plasma Wayland session on my notebook with enough confidence that I won’t lose data due to crashes. So thanks to the whole team for pushing the system to the limits.

An area which saw lots of work thanks to the Phone is the interaction between Plasma as the desktop shell and KWin as the Wayland compositor. With Wayland we keep the architecture of having a dedicated shell process which is not inside the compositor. This architecture has served us well in the past and we don’t see a reason why we should change this. It means that KWin can serve as a compositor for other desktop projects, but it also means that Plasma can be used with other compositors. Now unlike X11, Wayland’s protocols are secure by default. This means that Plasma as the desktop shell does not know anything about windows from other processes. To solve this problem we designed a special window management protocol which exports the required information. The protocols are still under development, but we hope that they can be also useful for other projects with a similar architecture (LXQt being an obvious candidate). Of course such protocols should only be used by the “blessed” desktop shell process – this is something we still need to implement and one of the reasons why at the moment Plasma/Wayland is only a technological preview.

Window management is not the only area where the shell process needs to be “blessed”. Also for announcing information about its own windows, we need some more information. We need to know whether a window is a “panel” or the “desktop” view. So on the phone the shell background is just a desktop window, the panel on the bottom is just a normal dock. This allows us to share the general layering code with the X11 implementation of KWin. Even more having the panels marked as panels allows us to properly define the maximized area. And the windows on the phone open maximized by using the “Maximizing” placement strategy in KWin.

Working on the phone project also had some surprises. For example the virtual keyboard just didn’t want to show. It turned out that the reason for this was that Qt(Wayland) only requests the virtual keyboard if it has keyboard focus. Keyboard focus inside KWin is bound to the “active” window. So we had to implement activating Wayland clients. But the initial implementation didn’t really solve it, we still had areas where the keyboard didn’t come up. E.g. on the desktop in the KRunner search field we couldn’t get it to show. The reason was related to the described problem: we did not yet support activating Wayland clients when clicking on them. This got solved by implementing mouse action (and touch action) support inside KWin. So thanks to this change (done for the phone) we can properly switch between Wayland windows on the desktop with a mouse driven setup.

Another nice touch is that KWin requires a running Xwayland server. This gives us support for all “legacy” X11 applications such as Qt 4 or GTK2 on the phone. Just we hit a problem with them: they did not react on touch events. The reason for this is quite simple: touch support is not yet implemented in Xwayland. As a solution we implemented simulating pointer events in case a connected Wayland window does not bind the touch interface. Thus all Wayland and X11 applications can now be used with a touch screen – be it on the phone or on a notebook with a touch screen.

So far I have only spoken about progress made in KWin which is relevant for the desktop. So what about the phone specific adjustments? Well there are hardly any. In the core of KWin there is no phone specific code. The only phone specific code is the hwcomposer backend. This is a plugin used for creating an OpenGL context using libhybris and for reading input events (unfortunately libinput cannot read events on a libhybris enabled system). According to cloc this plugin is just about 800 lines of code and about 200 are just very straight forward mapping of Android key codes to “normal” Linux key codes. For comparison: KWin + kwayland/server currently have about 120,000 lines of code. And even this hwcomposer backend is not phone specific. It could also be used to run a normal KWin/Plasma session on any libhybris enabled device. There is one important difference of the plugin to the rest of KWin which is worth to mention: it is GPLv3+ licensed, while everything else is GPLv2+. The reason for this change is the fact that libhybris is Apache 2 licensed and this license requires a change to GPLv3.

Posted on

Global shortcut handling in a Plasma Wayland session

KDE Frameworks contain a framework called KGlobalAccel. This framework allows applications to register key bindings (e.g. Alt+Tab) for actions. When the key binding is triggered the action gets invoked. Internally this framework uses a DBus interface to communicate with a daemon (kglobalaccel5) to register the key bindings and for getting notified when the action triggered.

On X11 the daemon uses the X11 core functionality to get notified whenever key events it is interested in happen. Basically it is a global key logger. Such an architecture has the disadvantage that any process could have this infrastructure and it would be possible for multiple processes grabbing the same global shortcut. In such a case undefined behavior is triggered as either multiple actions are triggered at the same time or only one action is triggered while the others do not get informed at all.

In addition the X11 protocol and the X server do not know that kglobalaccel5 is a shortcut daemon. It doesn’t know that for example the shortcut to lock the screen must be forwarded even if there is an open context menu which grabbed the keyboard.

In Wayland the security around input handling got fixed. A global key logger is no longer possible. So our kglobalaccel5 just doesn’t get any input events (sad, sad kglobalaccel5 cannot do anything) and even when started on Xwayland with the xcb plugin it’s pretty much broken. Only if key events are sent to another Xwayland client it will be able to intercept the events.

This means a global shortcut handling needs support from the compositor. Now it doesn’t make much sense to keep the architecture with a separate daemon process as that would introduce a possible security vulnerability: it would mean that there is a way how to log the keys. One only needs to become the global shortcuts daemon and there you go. Also we don’t want to introduce a round trip to another application to decide where to deliver the key event to.

Therefore the only logical place is to integrate global shortcut handling directly into KWin. Now this is a little bit tricky. First of all kglobalaccel5 gets DBus activated when the first application tries to access the DBus interface. And of course KWin itself is using the DBus interface. So KWin starts up and has launched the useless kglobalaccel5. Which means one of our tasks is to prevent kglobalaccel5 from starting.

Of course we do not want to duplicate all the work which was done in kglobalaccel. We want to make use of as much work as possible. Because of that kglobalaccel5 got a little surgery and the platform specific parts got split out into loadable runtime plugins depending on the QGuiApplication::platformName(). This allows KWin to provide a plugin to perform the “platform specific” parts. But the plugin would still be loaded as part of kglobalaccel5 and not as part of KWin. So another change was to turn the functionality of kglobalaccel into a library and make the binary just a small wrapper around the library. This allows KWin to link the library and start kglobalaccel from within the KWin process and feed in its own plugin.

Starting the linked KGlobalAccel is one of the first things KWin needs to do during startup. It’s essential that KWin takes over the DBus interface before any process tries to access it (as a good part it’s done so early that the Wayland sockets do not accept connections yet and Xwayland is not even started). We will also try to make kglobalaccel5 a little bit more robust about it to not launch at all in a Plasma/Wayland session.

Now the reader might think: wait, that still gives me the possibility to install a stealth key logger, I just need to create shortcuts for all keys. Nope, doesn’t work. As key events get filtered out a user would pretty quickly notice that something is broken.

Integrating KGlobalAccel into KWin on Wayland brings an obvious disadvantage: it’s linked to KWin. If one wants to use applications using KGlobalAccel on other compositors some additional work might be needed to use their local global shortcut system – if there is some. For most applications this is no problem, though, as they are part of the Plasma workspace. Also for other global shortcut systems to work with KWin it’s needed to port them to use KGlobalAccel internally when running in a Plasma/Wayland session (that’s also a good idea for X11 sessions as KGlobalAccel can provide additional features like checking whether the key is already taken by another process).

Posted on

Four years later

At beginning of June 2011 I made my first blog post about KWin support Wayland clients featuring a screenshot of Desktop Grid effect with a Wayland window shown on each desktop.

desktopgrid-with-wayland

Now it’s almost four years later and I show once again such a screenshot:

Desktop Grid effect in a Plasma on Wayland session
Desktop Grid effect in a Plasma on Wayland session

A few things have changed, for example the screenshot shows a KWin running on DRM. And there’s one huge difference: the KWin instance is using Wayland internally and no longer X11 (it still uses it for X11 applications, or course). Also KWin is able to properly integrate the Wayland windows. They are not rendered a top of the scene but take up normal spots like all other windows, too. Otherwise they are also more like “normal” windows. The plasmashell in this screenshot is also a Wayland client including all the panels and windows it creates (not visible as desktop grid effect hides panels).

A good question to ask is how much of the initial code written in 2011 ended in todays KWin and the honest answer is none. The branch got never merged and pretty quickly bitrotted. I got the rendering done pretty quickly, but at that time Wayland hadn’t had a stable release yet, so we simply couldn’t merge the code into master as that would have been rather inconvenient for development. With two moving targets (Wayland and KWin) the code diverged too quickly, broke too often and made development difficult. Thus when I heard a stable release of Wayland was planned it didn’t make much sense to continue the work on the branch.

But there was of course one thing which the branch provided: experience. The branch showed that our compositor is up to the task of integrating non-X11 windows, but the rest of KWin wasn’t. Which triggered a refactoring to make KWin more useable with multiple platforms. There were side-effects from that development which went already in some releases: e.g. the reworked scripting helped to identify the interface of a managed Client. Very recently this got split out into a new abstract base class which is now inherited by the good old X11 Client and the new Wayland ShellClient. A different area is screen edge handling which got reworked to not only allow multiple backends but also to have more fine grained per screen edges.

The work to get Wayland integrated into KWin took much longer than expected, part of it was the problem described above of the missing stable Wayland releases. Of course there was also a Qt5 port which I didn’t expect. The Qt5 port required a port to xcb – again something I didn’t expect. But most of all I underestimated the problem scope of how much work would be needed to get the window manager ported to Wayland. Still it will be quite some time till all features which are provided by KWin will be available for Wayland. Too much code is too X11 specific to be directly reusable.

And to get a complete session up and running there is much more to be ported than just KWin. We have Plasma which needs to access window management information provided by KWin (a taskbar should be able to show tasks after all). Some parts need to be moved into KWin for better security: screen locker and global shortcut handling (almost finished) are obvious candidates. We have functionality provided in our libraries which need adjustments: examples are KWindowSystem and KIdleTime. Our power management code need to learn how to talk with KWin to turn the screen off. KScreen needs to learn to interact with KWin to configure screens. And much, much more.

Also many of our applications need to be adjusted to work properly on Wayland. Obviously if one uses low-level X11 calls those won’t work any more. But also when you use higher level abstractions it might be that your feature doesn’t work any more, there are so many things which are X11 specific and you might not even know off. I plan to host a session on “Applications on Wayland” on this years Akademy. Make sure to go there. I’ll show all the “don’t” (and most are don’t on X11, too 😉 ) and show how one can easily setup a Wayland session to test the application.

Now I know that this blog post sounds a little bit like one shouldn’t expect Wayland on Plasma anytime soon. That’s not the case. I’m quite optimistic that I will shift my systems to Wayland as the primary work system before Akademy to properly dog food. And I’m sure that many Plasma developers will soon follow. After all I’m just editing this blog post in a full Plasma on Wayland session (though my Firefox is using XWayland).

And of course there are many, many tasks to work on. Most are really easy and easy to get into. Every contribution matters and helps us to embrace Wayland faster. Setting up a KWin development environment to test the Wayland progress is easy. All you need is to build kwayland and kwin through kdesrc-build (all other dependencies can be provided by your distribution). Once it’s installed just run:

kwin_wayland --windowed --xwayland

and a nice nested Wayland server will be started in your X session.

And if you want to get all down to running KWin directly on DRM like the screenshot above:

kwin_wayland --drm --libinput --xwayland

In all cases you can specify applications to be launched once KWin has fully started. Those can be either X11 or Wayland applications, e.g.:

kwin_wayland --drm --libinput --xwayland "konsole --platform wayland"

Will start a konsole as a Wayland client once can has started.

A recent addition as of today is that you don’t have to specify the commands like windowed or drm any more. KWin will automatically pick the correct backend depending on the environment variables which are exported. But be careful: if you start a kwin_wayland on an X session without DISPLAY exported, it will start the drm backend and that might break your X session!

Posted on

Enhancing user experience through the Compositor

This article is an April Fool. For an explanation please see the update.

Some time ago Ken introduced the conecept of dynamic window decorations (DWD) and during the last Plasma sprint there was already some work on experimenting with an implementation. DWD are extremely promising to enhance the user experience by morphing the application and window manager scope together.

The basic concept of DWD is to get content of the application inside the scope of the window manager. But what works one way works also the other way: we can enhance the user experience by providing additional information relevant for the window inside the window decoration.

An example for this could be a volume applet for music and audio players. But of course it doesn’t stop there, today is all about the user content and integrating the web. For web browsers we can integrate a global “Like It on Facebook” button. And by analyzing the user behavior we can provide even more help. We can notice if you struggle using Krita and provide a link to where to get a better drawing tablet.

If you use digiKam you might be interested in some books about photogrophy. We know which music you listen to on Amarok and show you that the new album of your favorite music group just got released. Not every application integrates stores in a proper way. This is a shame as it does not provide the best possible user experience and KWin as a window manager can help there to add missing store functionality where it is needed.

Of course this also allows to show you better alternatives. If we see that you struggle with wine emulated applications we can provide the Linux alternative. We can show that new updates are available for your currently used application. If you use sub-par free software applications we can show you were to get a proper (slightly more expensive) application.

But DWD are not the only place where KWin can help to increase the user experience by offering useful content. We always went a great way to make the desktop more useable by for example allowing to click on a splash screen to hide it. I think we can all agree that splash screens are something really useless: look at a stupid window for a few seconds and having to wait till it hides again. Meh.

We can do better: we know which application starts and what it is used for. And as a compositor we can exchange the content. Wouldn’t it be way more awesome to show a short video giving you advice where to buy the latest tips on C++11 book when launching kdevelop?

And there is so much more room to integrate useful content. When starting a video player we can overlay the window with a nice trailer of the latest movie (of course that will be skipable after 5 sec). This is a feature which might also be handy in Present Windows and Alt+Tab. Of course such videos would be perfectly localized as we know your geolocation and we would take great efforts to only offer trailers which might interest you: after all we have baloo to scan your video data and can also look at your Facebook profile. Now trailers are only one part, of course also other areas could be covered. If you have many pictures of cars you might be rather interested in spots about the latest car of your favorite brand instead of move trailers. Of course we will make it possible to configure which areas you are interested in. As we all know KDE is all about configurability.

Posted on

Why don’t you just…

Relatively regularly I’m asked why I don’t “just” integrate QtCompositor or libweston and call it a day. Well it should be obvious, if it were as simple as “just” we would do that 😉 So instead of replying again and again I thought to give a full blog post explaining the options and why they don’t really suit our needs.

How KWin’s compositor works

Let’s start with a look at the Compositors in KWin:

  • No Compositing (X11 only)
  • XRender (X11 only)
  • QPainter (Wayland only)
  • OpenGL (ES) (X11 and Wayland)

The task of the compositors is to render the windows into a global scene. The information about the windows is taken from the window manager (e.g. the stacking order defines which window is rendered on top) and get transformed by the effect system.

The compositors use platform specific API to get the content of the window and to render it to the screen. E.g. OpenGL can use GLX and use texture from pixmap to map the content of an X11 window to an OpenGL texture and use an XWindow to create the OpenGL context on. The actual compositor which renders doesn’t care whether it’s an X11 Window or a Wayland window or whatever, all it cares about it’s a texture (OpenGL), a pixmap (XRender) or a QImage (QPainter). The platform specific API is abstracted allowing to make the actual compositors completely windowing system agnostic. Designed as a class diagram it looks (simplified) like this:

Now with that much abstraction one could think that there a huge differences between the code and that there is not much sharing going on. But the opposite is the case. The majority of the code is shared. To give an overview I provide the line count of all the files relevant to the OpenGL compositor.

  • Shared code: 3733 (cloc scene_opengl.* abstract_egl_backend.* scene.*)
  • Platforms specific: 1424 (cloc egl* glxbackend.*)

From the platform specific code the glxbackend is about 50 %. The sharing between the egl backends is larger, for the new X11 mode for kwin_wayland the difference is just 20 lines. For QPainter the differences are also rather small – in fact the differences are so small that all is in one source file making something like 200 lines of code for the three backends. Last week I worked on integrating a framebuffer backend into QPainter – although I prior had no idea about it, it was integrated into KWin in less than two hours fully functional with no change to the SceneQPainter at all except writing 50 lines of the backend code.

Now the backends are only part of the whole story. By using our own existing compositor we ensure a similar feature set between X11 and Wayland and can reuse all our effects directly without any adjustments. According to CLOC we have 22000 lines of effect code.

What about QtCompositor

QtCompositor seems to be the natural choice for bringing Wayland support to KWin. After all it’s part of Qt which is what we use in KWin. Except it doesn’t. For a starter it’s not yet a released component which makes it difficult to use for us to develop against, for our packagers to distribute and for our users to test. “But Martin”, you say, “you could develop on it and get it into a release state!”

Yes I could, but would it help KWin? What is the strength of QtCompositor? Providing a QtQuick API to write a Wayland compositor. But KWin doesn’t use QtQuick in the compositor. “But Martin”, you say, “QtQuick is the new cool thing, everybody ports to it, so should you!”. Yes, of course, but it means:

  • Throwing away all our effects
  • Writing new integration code between QtCompositor and the window manager
  • having no code sharing between the Wayland and the X11 compositor
  • No longer being able to provide non-OpenGL compositors as QtQuick requires OpenGL

If I look at this I rather doubt that this would be a “just integrate it”. That looks like a lot of work and duplication to have the same basic window manager for X11 and Wayland. Code sharing would be kind of impossible as the technologies involved are too different.

“But Martin”, you say, “you could at least use QtCompositor for wrapping the Wayland API and use it to map a Wayland buffer to OpenGL!” Yes, that is true and I did consider it and even planned to do so. In the end I decided against it for multiple reasons. First of all this still shows the problem of the unreleased component (yes, I could work on it), but then I wanted to integrate Wayland in KWin with a Facade. Using a facade is something I consider here as rather important. After all we want to use Wayland for quite some time and not run into the situation that we need to port away from a library in a few years. Using a facade makes such a situation much easier. Now if we write a facade anyway what is the benefit of being a facade to QtWayland compared to being a facade to Wayland directly? It just adds another layer of abstraction which might hinder us (and compared we ported lots away from Qt to XCB during the Qt 5 porting removing the abstraction added by Qt). Oh and of course we will have to use Wayland directly, because we will have to use our own protocols for e.g. interfacing with Plasma. Which would mean that we have to interact with two technologies instead of one. Overall I didn’t see an advantage and that’s why KWayland is a facade over Wayland and not over QtCompositor.

What about (lib)weston

“But Martin”, you say, “you could integrate (lib)weston’s backend!”. Yes, weston uses plugins for it’s backends, so in theory it would be possible to load the plugins and interface with them. There were also patches to make this possible through a libweston, but I actually don’t know whether they got integrated (quick check looks like they didn’t).

I would love to have a library which abstracts e.g. kms or framebuffer for me, but honestly I doubt weston is that. Weston was developed as the reference compositor and not as a library to do that. Weston code is always open in my editor, so I know a little bit about the code and can see that it is – well – Weston-specific. I doubt that the code base could be shared in it’s current state. It would need an effort like what has been done for libinput to move this into a well designed API. As long as that doesn’t happen weston’s backends are not directly useable for us.

And even if we would get back to a different feature set. We use QPainter for non-GL rendering, Weston uses pixman. And that is nested into the compositor implementations of Weston. Again: that needs lots of work to get in a state which would allow integration, so overall it’s not a “just use it”.

And all of that does not consider what is more difficult: integrating the C API or writing the code yourself tailored in a way which makes sense in a C++/Qt world.

Posted on

KWinception

Last week I merged in a few important changes for the upcoming KWin 5.3 release. The rootless Xwayland support is integrated, which means we are a huge step closer to managing Wayland clients. Rendering, input and cursor is already using the Wayland code paths and will be shared with proper Wayland clients. I have already started working on code for that and have it working quite nicely already but decided to delay the integration for Plasma 5.4.

This means that kwin_wayland starts a Wayland server, but except Xwayland there is no application which can properly connect to it. Nevertheless it’s an important step and allows to also test the code in a better way.

In addition I worked on a better support for nesting compositors. So far one had to start Weston in order to test kwin_wayland. This is of course not optimal as it makes the development setup more difficult than it has to be. So last week I looked into developing a new backend which can use an X11 window to render to. This is comparable to Weston which can be used nested on X11. The backend is relatively straight forward: it can render to the created window using either the OpenGL or QPainter compositor, accepts input events and delegates them and passes the cursor from Wayland windows to the X11 window. The tricky part is that we cannot use any of our X11 specific libraries to create the window: we don’t use the xcb QPA, so no support from Qt and we cannot use KWindowSystem as it only allows one xcb_connection and that’s already needed for the Xwayland window manager. So I had to do things myself and as I consider this just a development feature it’s probably the worst X11 client in existance 😉 (Long term KWindowSystem should be changed to support multiple connections, would be very useful for unit tests).

The new backend triggered a small refactoring which makes it easier to implement further backends like e.g. a plain framebuffer or a drm/kms. As a small reminder: there’s an open GSoC idea for implementing the drm/kms backend and application period is to close soon.

Anyway with the nested KWin it’s now possible to create a kwin_wayland instance on the X Server using kwin_x11 and on the kwin_wayland one can create yet another kwin_wayland so that we have a KWinception:

Aus 2015-03-19

To start such a nested compositor use:

kwin_wayland --windowed --xwayland

Please watch the debug output for:

X-Server started on display :1

This tells the id of the created X Server (normally the next free id) and allows to start an application on the nested KWin:
DISPLAY=:1 kwrite

KWin_Wayland picks the windowing system depending on the environment variables which are defined. For DISPLAY it uses X11, for WAYLAND_DISPLAY it uses Wayland. If both are defined Wayland is preferred. If one wants to specify explicitly one can use the command line arguments –x11-display or –wayland-display. E.g.


kwin_wayland --windowed --xwayland --x11-display=:0

For X11 there are also the options –width and –height to specify an initial size (default is currently 1024×768). On Wayland these options are currently not available, the window always opens fullscreen.

Please remember that this is pretty new code and the support is still incomplete. Feature might be missing or broken. This is at the current state expected and because of that we do not yet accept bug reports. If you find a bug which you think should be fixed, please open an editor 😉

Posted on

Rootless Xwayland server integration into KWin

Over the last weeks I concentrated my KWin related work on trying to integrate the Xwayland server properly. Xwayland is an interesting step on the way to Wayland as it maps X11 windows to Wayland surfaces. But it also needs an X11 window manager to manage the X11 windows. Overall it allows us to start integrating Wayland into the compositor without too much breakage. It’s still X11 after all, so our existing code base continues to work. And gruadually functionality can be replaced with the Wayland equivalent, so that we can afterwards start integrating proper Wayland clients.

Integrating Xwayland showed interesting challenges. KWin as an X11 window manager requires a running X server prior to start. It also can only communicate with one X server, the code base has many hard constraints on being only one X server. This means if we want to use Xwayland the Xwayland server must be the one KWin uses. Which means Xwayland must be started prior to KWin’s X11 usage. So far KWin also enforced the usage of the “xcb” QPA plugin, the windowing system abstraction plugin for the X world in Qt. This plugin gets loaded directly at application startup and aborts if there is no X server it can connect to.

As you can see KWin needs to ensure that Xwayland is running prior to the application startup. But Xwayland requires a running Wayland server, which is supposed to be provided by KWin and for processing Wayland events we need a running event loop, which is only present after the application started. Looks like we are caught in a dependency loop:
circular-dependencies

After some hacking and experimenting I found a solution which can start the Wayland server and Xwayland prior to the application startup, but it’s considerable fragile and it can only be a temporary solution. In the long run it would of course be better if KWin could use the Wayland QPA plugin provided by QtWayland to connect to it’s own Wayland server and start Xwayland later-on.

Nevertheless I succeeded in getting KWin connect to the Xwayland server and to start transitioning X clients to be rendered using Wayland buffers instead of performing texture from X11 pixmap:

Aus 2015-02-11

But it just hit another problem: it didn’t support accelerated OpenGL rendering on the X server. That’s quite a problem if no X11 client connected to the Xwayland server can use proper OpenGL and it’s even a problem for KWin as KWin uses QtQuick scenes which use the X server. So KWin itself fails to render accelerated UI. Compositing is not affected as we don’t use Qt for that.

The reason for the problem is that Xwayland expects the Wayland “wl_drm” interface to be present. This interface gets created when binding an EGLDisplay to a Wayland display. In case of KWin the EGLDisplay exists after the Compositor is fully initialized. And again we are in a dependency chain: the Compositor gets created and uses the Workspace class. This class controls the complete startup of the X11 window manager which means it’s highly X11 dependent and requires Xwayland to be present. Again we are in a dependency loop.

Breaking up this dependency loop is quite tricky. The Compositor is too deeply nested into the application to be considered started before creating the QApplication. This means we must be able to create the QApplication before we have an X Server running. This means no usage of xcb QPA plugin. As KWin is going to start a Wayland server anyway, it would be good to get KWin to use the wayland QPA plugin.

This was again quite a challenge. The QtWayland QPA plugin performs blocking roundtrips to the Wayland server in the main thread during startup. But the Wayland server is running in the same thread. So a blocking call to the Wayland server dead locks the server. It’s not possible to move the Wayland server at that point into a thread as one cannot start a QThread prior to having the QCoreApplication created.

The solution I developed for this problem involves creating an own event dispatcher prior to creating the QApplication. So we have the event dispatching for the Wayland server ready to use. Just the event loop is not yet running. This allowed to provide a small patch for QtWayland to run any event dispatcher set before creating the QApplication. A test application in the kwayland repository is also adjusted to make use of it (the test application is also able to start an Xwayland server prior to creating the QApplication).

Unfortunately there is still another issue: QtWayland might call eglInitialize in the main thread which again performs a blocking wayland call. This is a problem I haven’t solved yet and currently just hacked around by disabling OpenGL in Qt (which breaks the QtQuick views).

Being able to use the Wayland QPA just creates a new bunch of problems for KWin. KWin still needs to use X11 and thus needs to create an xcb connection. Just that this wouldn’t help much. KWin on X11 doesn’t init the xcb connection, it’s the xcb plugin in Qt which does it and we only access it via QX11Info. Thus KWin needs to be moved away from the usage. Luckily in many cases we already wrapped the functionality as going through the QPA interface (which QX11Info does internally) is too expensive for our use cases. So the changes are not that invasive. But KWin also uses frameworks like KF5WindowSystem which use QX11Info. Even more those frameworks were properly fixed to perform platform checks and don’t do the X11 specific code if it’s not on platform xcb. But it’s providing important X11 window manager functionality for KWin. For some classes like KWindowInfo and KWindowSystem the fix was trivial: don’t use in KWin. There’s also the more low-level NETRootInfo and NETWinInfo and that’s what KWin should use – usage of the KWindowInfo or KWindowSystem inside KWin can be considered a bug. For some other classes it was already partially possible to be used without the xcb plugin on X11. The classes are only used if we compile with xcb present, so it was possible to add more xcb specific methods which can then be used by KWin even if we do not use platform xcb. The required changes will be part of frameworks 5.8 release.

To summary where we are now: we can start kwin_wayland on platform wayland connecting to a Wayland server started by kwin_wayland, we do not require QX11Info in (most of) KF5WindowSystem and KWin. We are a good step closer to the aim, but still not there. The dependency loop is still in place: Workspace starts the Compositor, the Compositor creates the EGLDisplay which is needed to start Xwayland, which provides X11 which is needed for starting the Workspace.

This means: reorder the startup. We need to be able to start the Compositor prior to the Workspace (which could be interesting for kwin_x11, too, as it could improve persumed startup time). This task was easier than expected. Workspace got split into many modules over the last years and most modules which need to be created prior to creating the Compositor do not depend on Workspace and do not depend on X11. In the few cases where it actually does depend on X11 it was not difficult to delay the X11 specific code till after the X11 connection is created.

With all that in place I was able to delay starting Xwayland to after the Compositor is created and Xwayland can provide OpenGL to the connected clients:

This screenshot shows kwin_wayland running on top of Weston connected to an Xwayland server supporting proper OpenGL as can be seen by the output of glxinfo (in the konsole) and glxgears. Also plasmoidviewer just works on top of this X stack.

Of course there is still some work to be done till this is production ready code, but it looks really promising and I hope to have this ready for the KWin 5.3 release. The aim is also to get more and more features changed to use the Wayland functionality instead of the X11 functionality. E.g. for damage event handling it already uses the damage event of a wl_surface instead of creating an X11 damage handle. This makes supporting Wayland clients easier afterwards.

Posted on

Looking at the memory improvements of KDecoration2

This week I finally moved kdecoration2 to the kde/workspace project structure and merged in the required backend code in kwin. This means the upcoming 5.2 release will ship with the new Breeze window decoration by default.

Thanks to the usage of the new library we get a nice performance boost in KWin. Some of this I already explained in my blog post announcing KDecoration2. The huge advantage here is that we do no longer use a QWindow as a backend and only require to hold an image the same size as the decoration. But there are more nice improvements around it. When I started designing KDecoration2 one of my aims was to focus it more on the use cases of a composited setup and design the API in such a way that a default decoration could be more efficient. Our previous default decoration Oxygen was rather heavy and KWin was not able to provide features to Oxygen which would make it more efficient. From a memory point of view the main issue with Oxygen were the shadows. With the old decoration API the shadows were part of the decoration. So the renderer got one big image for the shadow and the decoration part. But the shadow is the same for all inactive windows and could be defined by a much smaller image (lots of information is redundant). For KDecoration2 I looked at our existing shadow sharing mechanismn on X11 level and adopted a similar API which allows to provide a compressed shadow to the compositor. That way the shadow is no longer part of the window decoration and does not need to be part of the OpenGL texture. Of course this comes at the cost of needing more rendering calls (I deliberately moved the Time-memory-tradeoff towards better memory usage).

The real improvement comes when the decoration plugin can share the shadow between the decorations. This is being done for Breeze: there is never more than one shadow being created. In Breeze the shadow always has a size of 60×60 pixels, so we save 3600 pixels per decorated window. This is in addition to the saved memory for the compressed shadow instead of having the shadow as part of the window.

Today I went a step further and also modified the code in our OpenGL compositor to share the textures being created for the decoration shadow. So if two decorations have the same shadow, they will also use the same texture. This does not only save us some texture memory, but also means less pixel transfers from CPU to GPU. So a rather nice improvement. For our default decoration Breeze this means that only one shadow is created in the decoration plugin and also only one OpenGL texture needs to be created.

Unfortunately not all decoration plugins can benefit from this change. Also with KDecoration2 the problems outlined for Aurorae still exist and the change in the shadow mechanismn doesn’t improve the situation. The hope to improve this is in Qt 5.4 which gives us new useful features like QQuickRenderControl which allows us to integrate the OpenGL context used for QtQuick with our compositor’s OpenGL context. The hope is that we can render the QtQuick based decoration into an FBO and share the texture with our compositor, so that we can just bypass the normal decoration rendering process which is too costly in the case of Aurorae. I already switched the normal rendering to QQuickRenderControl if kwin is compiled with Qt 5.4 and the result is looking rather promising. Still the memory usage and performance of Aurorae will never be as good as the memory usage and performance of a native decoration. This has never been the aim of Aurorae.