Evolving 3D desktop effects in Plasma

The latest Plasma release dropped a few desktop effects: the cube family, CoverSwitch and FlipSwitch. All of those effects were written back in 2008, the early days of KDE 4.x and the early days of desktop effects in KWin. The effects were implemented by me and when Vlad asked about removing them I saw the need for this and supported this step for technical reasons. With this blog post I want to share a little bit of why it was needed to remove them and why this means that they can come back in better ways than ever before.

State of CPU in 2008

To really understand this we need to time travel back to 2008 and the years before when desktop effects were introduced. This can help to understand how the hardware architecture changed and how that influenced design decisions in the effects API which are nowadays problematic. First of all CPUs. The Intel Core 2 Duo architecture was launched in 2006 as the brand new thing which had multiple (namely 2) cores which slowly replaced the NetBurst architecture which dominated Desktop computing for the beginning of that decade.

Also for developers these multi-core systems were a new thing. And that meant lots of software written before was single threaded. Especially KWin at that time was single threaded as it also depended on libraries which were not really thread safe such as the Xlib library and back at that time even OpenGL. Even years later when Qt introduced the threaded rendering in Qt 5 on many Mesa drivers this was disabled due to thread safety issues. Nobody would have thought of having any benefit of a threaded compositing approach back in 2007 with the state of hardware and the available libraries. Thread libraries were of course already available such as QtConcurrent or ThreadWeaver, but not of a useful use in KWin. This means the API written back then did not support ideas like doing rendering on a second thread or even rendering for each screen in a thread.

State of OpenGL in 2008

KWin’s compositing pipeline was written for OpenGL 1.4 which was the only supported version in the Mesa stack. KWin supported shaders written in OpenGL Shading Language which replaced parts of the fixed functionality pipeline in OpenGL 1.4 and emulated the fixed functionality pipeline. This was only supported on the proprietary NVIDIA driver. OpenGL 3 was released in 2008, but it was years till it became available on the Linux desktop (according to Wikipedia in 2012).

Thus the design of the OpenGL compositing scene and desktop effect system was written for the fixed functionality and only with this in mind. While KWin gained support for the programmable pipeline and is the only supported way nowadays the design is still in place and problematic.

State of multi screen support

Multiple screens were not that common back in 2008 and there existed multiple competing technology approaches. There was the old “one X screen per screen” approach with a different DISPLAY variable for every screen. There was Xinerama, with the NVIDIA implementation called TwinView and there was the “new kid on the block” XRandR. Using multiple screens still required hacking the xorg.conf file and then with luck it was working. Especially if you had an NVIDIA card which one needed for good desktop effect performance.

 

From https://xkcd.com/963/

From an X11 perspective there was (and even today) there is not such a thing as multi screen. For the compositor everything is one screen and we have to present all screens at the same time. So much for variable refresh rates (AMD FreeSync introduced in 2015), buffer age extension (implemented in KWin in 2013) and so on. From rendering point of view there was not much difference between rendering one screen or multiple screens. All we had in KWin was an integer variable telling us the count of known screens and the geometries.

State of Alt+Tab

Alt+Tab, which CoverSwitch and FlipSwitch provide, was still a hard coded implementation in QWidget. With the effect system an API was added to suppress the QWidget and use a desktop effect as replacement. This allowed to have an effect (BoxSwitch) which showed thumbnails for the windows. Overall this was rather a break with the other parts of the effects API as the effects are mostly used to influence position and drawing of the windows. With the Alt+Tab API an internal part of KWin was exposed and suppressed. It was it’s own API inside the API.

CoverSwitch and FlipSwitch even took this a step further by introducing 3D elements in the so far 2D world of desktop effects and completely intercepted the rendering. Most effects do not change the order how windows are rendered, but with those effects it was important to render the windows exactly in the order Alt+Tab wants them. So the effect intercepts the normal render calls and renders the windows in another order. To make this worse the CoverSwitch included reflections which meant the windows needed to be rendered twice. And the effects had to combine windows from multiple screens. The cube effect family did even more horrible things.

How Alt+Tab evolved

The biggest change to Alt+Tab happened thanks to QtQuick. While it was reworked before already and was no longer QWidget based, QtQuick allowed to easily define new layouts. The internal Alt+Tab API was one of the first parts to be exposed to QtQuick in KWin. Furthermore we introduced an interesting concept to make it possible to render window thumbnails in the QtQuick scene. This was done by telling the compositor to draw a thumbnail at a specific region of the QtQuick Alt+Tab window. As this was perfectly synced we finally had an easy to use API to make Alt+Tab layouts with all the fancy things only desktop effects could give us. Though we could not put other elements on top of the thumbnails (e.g. a close button) or transform them from within QtQuick. This was the end to the already mentioned BoxSwitch effect and the Alt+Tab mode in PresentWindows. Thus the only remaining effects for Alt+Tab are FlipSwitch and CoverSwitch.

With Qt 5 there was the hope to further improve that. Now also QtQuick used OpenGL for rendering which meant that in theory it would be possible to make our window textures available to QtQuick. Thanks to the work on the compositing engine lately this is now possible and gives even more flexibility to render Alt+Tab and makes it possible to implement FlipSwitch and CoverSwitch with QtQuick. This is really awesome as it means we have much better tools at hand to implement such fancy effects, we don’t have to develop our own toolkit and implement our own transition handling. Instead we can use all the great things like PathView in QtQuick. The C++ desktop effect implementation of CoverSwitch was 1000 lines of code, while the new emerging QtQuick based implementation is just about 200 lines of code.

How the effects evolved

Also the KWin effect system had a transition. We noticed that most effects are actually animations and added a dedicated implementation for it. This implementation is exposed to JavaScript and most of those effects are nowadays written in JavaScript. The effects in C++ are often the odd ones which do too much and use the wrong toolkit for the wrong thing. Such as I said years ago that PresentWindows and DesktopGrid need to be rewritten in QtQuick. The effect system should be for animating, not for developing a user interface. With QtQuick we have a way better toolkit.

Overall huge effects such as the cube family, CoverSwitch and FlipSwitch are standing in the way of evolving the effect system. While we have better tools to implement and maintain them thanks to QtQuick.

OpenGL changes in KWin compositing

In Plasma 5.8 we will see a few changes in the OpenGL compositor which are worth a blog post.

Debug information

The new KWin debug console gained a new tab for OpenGL information. This shows information comparable to what we know from glxinfo/es2_info. It shows the information gathered from the OpenGL library (version, renderer, etc.) and all the available extensions. This is intended to be useful when for whatever reason glxinfo is not working and one needs to know exactly which extensions are available.

OpenGL information in Debug Console

As you can see in the screenshot: KWin also learned to add decorations around “internal” windows on Wayland. Which means the debug console can now be freely resized and moved around.

Support for llvmpipe

With Plasma 5.8 we finally allow OpenGL compositing through the llvmpipe driver. So far KWin fall back to XRender based compositing when the OpenGL driver uses software emulation. We thought that with XRender based compositing one gets better results than with software emulation. In this release we re-evaluated the situation and came to the conclusion that it doesn’t make sense to continue blocking llvmpipe driver. Due to QtQuick KWin will render parts of its user interface through llvmpipe anyway and also Plasma will render completely through llvmpipe. Thus most of the system is going through that driver. If the system is usable enough for Plasma, it will also be sufficient for KWin’s compositing.

Related to that many of the reasons to not use llvmpipe by default are going away. For example we didn’t want virtual machines to render through llvmpipe, but today also KVM can do accelerated OpenGL through virgl driver. Similar the embedded systems also provide working drivers nowadays, if we think of raspberry pi or odroid – they all can do OpenGL and we don’t have the risk of them going on llvmpipe which would result in a very bad experience.

Last but not least there is the question whether XRender or QPainter compositing is a better solution than llvmpipe. And there I’m not so sure anymore. For XRender it’s possible that the xorg modesetting driver is used in combination with glamor. In that case it would also use llvmpipe. So nothing gained by forcing to XRender.

But still it’s possible that using llvmpipe compositing will result in high CPU usage if KWin has to use it. The solution to this problem is to deactivate what we know to be expensive – all these effects which are not available on XRender because we know it to be too expensive. We don’t want blur effect on software emulation and neither wobbly windows. So we are now working on disabling the effects if we are on software emulation. For this we introduced a way to disable all animations in KWin and a lot of effects are already adjusted. This might also be a handy new features for users who don’t want any animations at all, but still want the compositor enabled.

Last but not least using llvmpipe also exposes performance problems which we normally don’t see with a fast GPU. And as a result we were already able to improve various parts of our rendering stack which will benefit all users – independently of whether llvmpipe is used or not.

Removal of GLX/EGL selection

A change which also got backported to a 5.7 bugfix release is the removal of the selection of GLX or EGL in the compositor settings. Unfortunately EGL is still not a good enough option on X11. Way too often we saw bug reports about rendering being broken and it was caused by using EGL. We decided that it doesn’t make sense to expose a config option which in most cases will result in the users systems being broken. So on X11 the default will stay GLX, while on Wayland the only option is EGL. Of course on X11 the option can still be set manually be editing the config file or by specifying the environment variable to pick EGL or to pick OpenGL ES 2.

Removal of unredirect fullscreen windows

Unredirection of fullscreen windows has been a kind of blue-headed step child in KWin’s compositing infrastructure for a long time. It’s a feature not loved by the developers, not properly integrated, but you have to support it. For those not knowing the feature: it excludes an area from compositing and let’s the fullscreen window be rendered the normal way in X11 (unredirect). The idea is that you get slightly better performance if you bypass the compositor.

The functionality was never fully integrated into the compositor. It was way too easy to break out of the condition (e.g. a tooltip), but at the same time effects which should break it, had no way to do it (e.g. Present Windows should either not activate or end it). The weirdest oddity of the feature is that we had to hard disable it for all Intel drivers due to crashes. We don’t know whether this is still the case but after having had such a bad experience with it in the past, we decided to never turn it on again. Which means it’s a feature not even supported by all devices.

We developers did not spent much time on the feature as we think it doesn’t make much sense as KWin has a better infrastructure in place: blocking compositing. Applications are allowed to specify that compositing should be blocked. This results in KWin shutting down the compositor, freeing all resource related to it (e.g. destroying the OpenGL context), so all power to the running game. As the compositor is shutting down, you don’t have weird interactions like tooltips jumping out or effects not working properly.

There is a standardized way for applications to request this and we see that many games and applications (e.g. Kodi) make use of it. This is the preferred way in our opinion. Given that this mode is fully supported, we decided to remove unredirect fullscreen windows from KWin’s compositor. This streamlines our implementation and gives us one feature to concentrate on and make sure that it works exactly as our users need it. On Wayland the architecture looks different: there is no such thing like unredirect fullscreen, but we can ideally just pass the buffer to the DRM device. The idea is that we do the best optimized way whenever possible.

Support for render devices

Our virtual rendering platform in KWin/Wayland gained a new feature. Like the normal DRM device it now uses udev to find all possible devices and will default to a render device (e.g. /dev/dri/renderD128) if present. If not present it will look for a card device created through the vgem driver. This is a change intended mostly for the use on build.kde.org so that we can properly test OpenGL compositing, but is also a handy feature for running KWin in the cloud on systems with “real” GPUs. If no GPU is present one can easily make KWin pick the virtual device.

Support for restarting the compositor on Wayland

Initially KWin did not support restarting the compositor or even switching the backend on Wayland. But of course we need to support that as well. The OpenGL driver might do a graphics reset after which our infrastructure wants to reinit the complete setup. Thus KWin needed to learn how to restart the OpenGL compositor on Wayland, which will be fully supported in Plasma 5.8.

The return of kwin_gles

Back in 4.x we provided two binaries for KWin: one compiled against OpenGL (kwin) and one compiled against OpenGL ES (kwin_gles). The reason for that is that one can only reasonably link either OpenGL or OpenGL ES and OpenGL ES is only a subset of OpenGL, so one needs to hide the OpenGL calls (especially the OpenGL 1 calls).

With 5.x we were no longer able to provide these two binaries. The reason for that is that OpenGL got “upgraded” in Qt and QtGui itself links either OpenGL or OpenGL ES. To keep the system’s sanity we decided to follow how Qt is compiled. If Qt is compiled with OpenGL support KWin gets compiled with OpenGL support, otherwise with OpenGL ES support.

That’s of course a reasonable design, but it means that it becomes difficult to test the OpenGL ES code paths. One needs a dedicated Qt and all other dependencies compiled against OpenGL ES. Or one needs a nice device like a Nexus 5 with Plasma mobile. This had resulted in breakage already as we were not able to test enough. Such times belong to the past as I have a nice Plasma mobile device to compile and test my KWin on.

But since we introduced OpenGL ES support through a compile time switch, many things have changed. KWin dropped the OpenGL 1 support which means that most of the code which wouldn’t compile with OpenGL ES is just gone. Furthermore we switched to use libepoxy, so we don’t link OpenGL or OpenGL ES at all, but libepoxy which does the right thing for us. With that we are able to remove all the compile time checks. Of course we need runtime checks to ensure that we don’t call functionality which isn’t available on OpenGL ES.

Now with the upcoming 5.5 release we are able to have one binary which serves both OpenGL and OpenGL ES. Note to distributions: the artifact kwinglesutils.so is no longer compiled, please adjust your packaging rules. KWin will use either OpenGL or OpenGL ES depending on what Qt uses.

Given that nowadays it’s also possible to create both an OpenGL and OpenGL ES context at runtime we can also make use of that and introduced a new value for our KWIN_COMPOSE environment variable: O2ES. If that’s specified KWin will use the EGL backend and create an OpenGL ES context. Although it in general is also possible to create an OpenGL ES context through glx we do not support that for simplification. As proof debug output (qdbus org.kde.KWin /KWin supportInformation) from my running KWin instance:

<code>Compositing
===========
Compositing is active
Compositing Type: OpenGL ES 2.0
OpenGL vendor string: Intel Open Source Technology Center
OpenGL renderer string: Mesa DRI Intel(R) Ivybridge Desktop 
OpenGL version string: OpenGL ES 3.0 Mesa 10.6.8
OpenGL platform interface: EGL
OpenGL shading language version string: OpenGL ES GLSL ES 3.00
Driver: Intel
GPU class: IvyBridge
OpenGL version: 3.0
GLSL version: 3.0
Mesa version: 10.6.8
X server version: 1.17.3
Linux kernel version: 4.2
Direct rendering: Requires strict binding: no
GLSL shaders:  yes
Texture NPOT support:  yes
Virtual Machine:  no
OpenGL 2 Shaders are used
Painting blocks for vertical retrace:  yes
</code>

So in a way we have kwin_gles back, it’s different as it’s no longer a dedicated binary, but it’s runtime switchable. For the moment the only way will be the environment variable. I’m reluctant to add a config option as that sounds like quite some chance for breakage.

Thoughts on Vulkan in KWin

Lately I have been asked a lot about using Vulkan in KWin: in fact almost every blog post in the last few months has questions about it and that seems to me there is something to write about it.

So the quick tldr: I don’t have any plans on adding support for Vulkan. Over the last years I ported to so many new technologies, including at least 3 incompatible OpenGL versions. I am not looking forward to another incompatible OpenGL version (whether it’s called Vulkan or OpenGL doesn’t matter to me on that).

Now let’s look at the strength of Vulkan: going closer to the hardware and doing multi-threaded rendering. KWin still performs the rendering in the main GUI thread. Qt tried to do rendering in an off-thread for QtQuick’s scene-graph, in case of KWin we also do that in the main-gui thread. Reworking our compositor to use threading is a lot of work and would also probably improve the performance with OpenGL. Anyway as long as KWin doesn’t support threaded rendering this improvement by Vulkan is rather moot.

Now let’s look on the closer to hardware: yesterday I did a blog post on how I want to use e.g. KMS Planes to get closer to hardware and bypass the OpenGL compositing. Vulkan will allow us to perform rendering closer to hardware, but if we don’t render at all, we don’t benefit from it. Vulkan is probably more power saving, but not using the rendering will save even more.

So Vulkan can only improve when the scene needs to be rendered: e.g. when you wobble a window or spin the desktop cube. I don’t think that those effects are what we should optimize for and spend our time on and even if: there are low-hanging fruits to optimize using OpenGL, we do not yet use OpenGL 3 or 4 features to improve these effects.

Overall Vulkan looks to me like a lot of work as once again we would have to add a new compositing backend, write a new low level interaction, rewrite all shaders, etc. etc. Going to Vulkan early would mean introducing more complexity to KWin, more different code paths our users might use. It sounds like a mood exercise to do so. But I also doubt that there will be useable Vulkan drivers any time soon.

Vulkan is a great technology which will make graphics much better. But I doubt that an application like KWin is the use-case it was developed for.

Now to something else!

KDE is currently running a fundraiser for the Randa Meetings with the topic “Bring Touch to KDE!”. I personally will not participate in the Randa Meetings, but many other KDE developers will go there and we need your support for it. Please help and make the Randa Meetings possible.

Fundraiser

Should we target EGL as the default?

When we started the compositing work in KWin the only way to initialize an OpenGL context was by using GLX. In fact GLX is even part of the OpenGL library on Linux. Being an X11 window manager and an X11 compositor it was not a big problem.

Years later we started the effort to get KWin using optionally OpenGL ES in addition to normal OpenGL. One of the differences is that OpenGL ES doesn’t use GLX, but rather EGL. So we gained code to setup compositing using EGL. But this was still a compile time switch and only supported together with using OpenGL ES. If I remember correctly the option to create on OpenGL context on top of EGL was not yet available in our drivers back then.

But there was a time when the FLOSS drivers started to support it and since the 4.10 release in beginning of 2013 we can provide a normal OpenGL context over EGL. This option was not really exposed and only bound to an environment variable. Very few users knew about it and this was also wanted. Using EGL was a good way to shoot yourself in the foot given that not all drivers support it and that we had much more testing on the GLX backend.

Now the situation has changed. With 5.x we expose EGL also through a config interface, but more importantly EGL is the only way to get OpenGL on Wayland. The assumption that every user is using GLX doesn’t hold any more. The assumption that we don’t need to spend so much time (as it’s only relevant in the reduced feature set of mobile devices) on the EGL backend is wrong. It must reach the same quality as the GLX backend, otherwise our Wayland experience suffers.

So this raises an important question: should we try targeting EGL as the default? In the past one of the main reasons to not even think about EGL as default is the reason that the NVIDIA driver didn’t support it. If we would have switched to EGL by default it would have meant dealing with the situation that EGL doesn’t work and we need to fall back to GLX. It would have complicated the startup significantly and would have resulted in our user base using different rendering paths.

Now this one road blocker seems to resolve itself. While I haven’t tested it yet, the latest NVIDIA beta driver supports OpenGL over EGL, so we should be able to have the majority of our user base on the same backend after all. Given that it is only a beta driver and the feature is marked as “experimental” it’s probably still some time till we can default to it.

Nevertheless it’s time to start thinking about it. I encourage NVIDIA users to give EGL now a try and to report issues. Also I encourage all other users to report issues they see with the EGL backend. If we get the EGL backend into a good state maybe 5.6 might be a good point in time to default to it and then start to slowly deprecate the GLX backend with the aim to get it removed in the distant future.

KWin/5, QtQuick 2 and the caused changes for OpenGL

In the KWin/4 days our OpenGL implementation could be based on some assumptions. First of all only KWin creates an OpenGL context, none of the libraries, KWin depends on, uses OpenGL. From this it is possible to derive more assumptions, for example our compositing OpenGL context is always current. Which means that it’s possible to run any OpenGL code in any effect. No matter how the code path got triggered. Also we can link KWin against OpenGL or OpenGL ES. We don’t have to check with other libraries to prevent conflicts.

With KWin/5 these assumptions no longer hold. KWin uses QtQuick 2 and with that we pull in the Qt OpenGL module. One of the direct implications of this change is, that we are no longer able to provide kwin/OpenGL and kwin/OpenGLES at the same time. The compilation of KWin has to follow the compilation of the Qt OpenGL module. So compiling against OpenGLES is only possible if Qt is compiled for OpenGLES, which means that the option to run KWin on OpenGLES is probably non-existing on classic desktop computers and the option to run KWin on OpenGL is probably non-existing on ARM systems such as the Improv. Given that it’s no longer possible to compile both versions at the same time, the binary kwin_gles got removed. A kwin compiled against GLES is just called kwin.

With QtQuick also our assumption that only KWin creates an OpenGL context and makes it current doesn’t hold any more. In fact it could be that at any time Qt creates an OpenGL context and makes it current in the main GUI thread. Now people probably know that QtQuick 2 uses a rendering thread, so that should be fine, right? Well not quite. To decide whether QtQuick uses a rendering thread it creates a dummy context in the main gui thread and makes it current. So our assumption that our context is created once and then kept current, doesn’t hold any more. The first solution to this problem was to make our context current whenever we go into the repaint loop. That worked quite well, till I tested KWin/5 on my SandyBridge notebook.

The problem I stumbled upon is that Qt doesn’t use a rendering thread in all cases. For some hardware it prefers to use the main gui thread. One of them is SandyBridge Mobile, the hardware I am using on my notebook. This showed that the initial solution was not elaborated enough. With another context rendering in the same thread it showed that it’s quite likely that we hit conditions where our compositing context is not current. Resulting in e.g. not rendered window decorations, effects not able to load their textures, etc. etc.

These problems are hopefully solved, now. The effects API is extended by calls to make the context current and I tried to hunt down all effects which do OpenGL calls outside the repaint loop. Unfortunately given the large number of effects it’s still possible that some effects use it incorrectly. It will be difficult to track those down: so please test.

The case when QtQuick uses the main GUI thread for rendering illustrates that we in KWin are not the only ones having incorrect assumptions on OpenGL. QOpenGLContext assumes that every OpenGL context in a Qt application has been created through QOpenGLContext and that an OpenGL context is only made current on the current thread through the QOpenGLContext API. Especially if one uses glx or egl directly to make a context current QOpenGLContext doesn’t notice this and assumes that its own context is still current which can result in nastiness. This is circumvented in KWin now by making sure that QOpenGLContext has correct information once the compositing context is made current. Nevertheless we are still hitting a bug causing a crash. This one is also currently worked around in the development version by enforcing XRender based compositing on the hardware which uses the main GUI thread for rendering. On SandyBridge one can also use the environment variable QT_OPENGL_NO_SANITY_CHECK to force QtQuick to use a dedicated rendering thread as the problem why Qt uses the main gui thread is not relevant to KWin’s usage of QtQuick. KWin also checks for this environment variable and doesn’t force to XRender, if it is set.

Obviously one could question why we are not using QOpenGLContext given that this seems to conflict. We haven’t used Qt’s OpenGL module mostly for historic reasons. Of course I evaluated the option of using QOpenGLContext when investigating this issue and right now in Qt 5.2 it would not be an appropriate solution for the usage in KWin. It’s not possible to create a QOpenGLContext from a native context and even if it were possible our implementation is more flexible. KWin can be compiled against both egl and glx allowing to switch through an environment variable. Qt on the other hand supports either egl or glx, but not both at the same time. If I find time for this, I intend to improve the situation for Qt 5.3, so that we can consider the usage of QOpenGLContext once we depend on Qt 5.3. Switching to Qt’s OpenGL context wrapper would allow us to get rid of a considerable amount of code. I’m especially interested in the QOpenGLFunctions. Obviously that will only be a solution if KWin uses the same windowing system as Qt’s platform plugin. But that’s a problem for another day 😉

Compositing and “lightweight” desktops

In the general discussion about “lightweight” desktop environments I have read a few times that one should disable Compositing in KWin. That’s done in Kubuntu’s low-fat settings package and also something Jos talked about in the context of Klyde.

I have never seen an explanation on why Compositing should matter at all. It mostly boils down to “OpenGL is evil” and “I don’t want 3D”. So let’s leave the “educated guesses” behind us and have a proper look to the question whether Compositing matters for “lightweight”. (Remember: lightweight is a buzz-word without any meaning.)

Let’s start from one of the “lightweight” aspects: Memory. For this it’s important to know that KWin is a Window Manager and Compositor. You cannot just have the Window Manager – even if compositing is disabled. It’s not like xcompmgr which runs in another process. For the binary that gets loaded into memory it doesn’t matter whether compositing is enabled or not. The pure binary containing both window manager and compositor has a size of 1.3 M (on Debian testing, that should be KDE SC 4.8). But when you enable compositing an additional library gets loaded containing the Effects. They are all in one library to not have to open 40 different files. (Again on Debian testing) This library has a size of 783 K. The effects do not link any library which is not already linked by KWin core, so that’s what we are talking about from additional memory costs when using any desktop effect from binary size: 783 K!

Obviously the binary mapped into memory is just one aspect. There are also additional data structures which need to be created on the heap. I have no idea how much is really needed and cannot properly look at my system because I have a KWin with debugging symbols loaded. What’s important to know is that only Effects one uses will get loaded and use memory for their data structures. Also the Effects do not load everything they need but wait till they get activated. E.g. the cube effect will only load a background texture if the cube gets activated. But of course effects do not have anything to do with just using compositing. If you don’t want an effect, just disable it.

The biggest cost from compositing in the memory perspective is probably using the XComposite extension which redirects the rendering into an X pixmap. That’s something which will go away once we are on Wayland as application and compositor share the buffer the application is rendering to.

But even so the usage of the additional memory is not just something which is thrown out. It’s just another case of the time-memory tradeoff. Unrelated windows do not need to repaint if a window get’s unmapped, moved or resized. Just look at this comment from the non-composited switching of virtual desktops:

During virt. desktop switching, desktop areas covered by windows that are going to be hidden are first obscured by new windows with no background ( i.e. transparent ) placed right below the windows. These invisible windows are removed after the switch is complete. Reduces desktop ( wallpaper ) repaints during desktop switching

Without compositing we move the time-memory tradeoff towards using more CPU (create those windows) to prevent even more CPU usage. With compositing we don’t have to do anything of it. So disabling compositing is obviously not the silver bullet for being “lightweight”. It just means moving the time-memory tradeoff slider towards CPU.

What remains is the OpenGL question. I know that many people think it’s about having a “3D desktop”. But the desktop I’m using is quite 2D-ish. There are people saying that 2D is just a special case of 3D. Looking at OpenGL that is certainly the case. As long as you don’t use any effect which needs the “3D aspect” of OpenGL, KWin just does 2D with OpenGL. Using OpenGL means to make use of hardware which is designed to do these operations instead of doing it on the CPU which is not designed for it. But what if one doesn’t have the hardware for it? A valid question, but just not in the context of KWin. If your system only provides software rendering KWin automatically falls back to XRender based compositing.

The last aspect to mention in the context of OpenGL is texture from pixmap. I don’t know how it’s implemented in the drivers so I don’t know whether that needs additional memory or not. But if you are concerned about it: XRender does not need a texture from pixmap.

So as you can see compositing comes with a little bit more memory usage but reduces CPU usage. It’s nothing I could say which one is generally better, but less CPU usage means a sleeping CPU. Granted some effects do animations – that also requires CPU usage. If you are concerned about that it’s better to disable the effects or set the global animation speed to instant than to just disable compositing completely.

And if you try to be “lightweight” and play with the compositing settings: one unrelated change can ruin all your benefit. Just imagine you would disable compositing because you want to be “lightweight” and then you enable a window decoration which uses QML (that is interpreted code) and does animations?

Fallback mode in KDE Plasma Workspaces

Recently there has been a lot of buzz about non-composited fallback modes in various Desktop Shells and of course I have been asked several times about the fallback modes in KDE Plasma workspaces and whether they would be removed, too. Now instead of answering the same question again and again I decided to write a blog post to discuss the situation in more detail.

The first thing to notice is that KDE Plasma workspaces do not have a non-composited fallback mode in the way GNOME Shell or Unity used to have. The main difference is that our window manager (KWin) is able to act as a non-composited, XRender based compositor and OpenGL (ES) based compositor. This means that we do not have to maintain two window managers in order to provide non-composited setups.

The second major difference is that the Desktop Shell (either KDE Plasma Desktop, KDE Plasma Netbook or KDE Plasma Active) is not a plugin to the compositor but a separate application running in an own process. This allows to use a completely different window manager together with the Desktop Shell. Some years ago it was not that uncommon to use Compiz together with Plasma, though we see less and less such usage patterns (mostly caused by Compiz no longer being available in most major distributions except Ubuntu). This is a design decision which served us well and we do not plan to change it. Some time ago I brought up the topic, but more experienced developers (mainly Aaron) illustrated nicely the advantages of our design choices.

Of course we need some code to be able to adapt to non-composited mode inside the Plasma workspaces. But this code would be needed anyway. Why? To answer this question we have to go back in time to when the development of Plasma had started – that is around 2006/2007. Back then OpenGL based compositing had been a rather new topic (I think I first heard about Compiz in 2004 or 2005) and it had only been possible with the binary blobs and hacks such as Xgl. The Intel hardware back then was not powerful enough and drivers were lacking, too. With other words: going an OpenGL-based only path seemed not feasible at that time and consider that the fallback modes in Unity and GNOME Shell got only removed in late 2012/early 2013. This is more than half a decade later – an eternity in IT. Also KWin did not enable compositing support by default till 4.2, so the first releases of KDE Plasma were non-composited by default.

The solution inside Plasma was to make our themes aware of compositing. That is each theme contains an opaque element set which will be used when in non-composited mode. The fact whether compositing is used, is globally available through a standardized X11 manager selection. So whenever the compositing state is changed Plasma is notified and switches the used elements. Simple and elegant. This functionality served us quite well when we introduced the blur effect. Translucency is actually a very difficult topic. If you choose a too strong translucency level the text is no longer readable, if you choose a translucency fitted for text readability the translucency is hardly visible. The solution to this problem is the blur effect. It provides easily readable text even with strong translucency levels. But even today we do not enable the blur effect by default on all hardware as it is an expensive effect and we would not dare to enable it on embedded devices (personally I doubt that Windows 8 removed the blur because the designers disliked it). So we see we need themes with different levels of translucency depending on whether blur is available or not. The design decisions for the Plasma themes provided us the solution directly. As well it means that even if we would drop the non-composited mode we would have to keep our design to support the blur selection.

For most of the applications it hardly matters whether the desktop is composited or not. Only very few applications use the alpha channel in the first place and if they want to use it, everything’s nicely abstracted inside Qt.

Inside the window manager it also doesn’t really matter. Of course our window decorations are aware of whether compositing is used or not and especially Oxygen makes use of it by providing an adjusted look for the non-composited case, so there we have some overhead, but except that there are hardly any adjustments.

This also means that we would not gain anything from requiring compositing. The compositor is a separate module inside the window manager which just gets started by the window manager on start-up. It lives more or less for itself and is just notified when a new window is added to the window manager and so on. Nothing inside the window manager really depends on the compositor. That means it would be more work to get KWin requiring compositing than to keep supporting the non-composited mode. And having the non-composited mode around allows us to do things like turning compositing off when running games or heavy OpenGL based applications such as Blender. So if you want to get some of the now finally available games for Linux, KDE Plasma should be your primary choice to enjoy the game. I have also heard of users switching to KDE Plasma because we still provide non OpenGL based setups.

What remains is the question of what will change with Qt 5? Obviously not much. Plasma will continue to provide a non-composited and a composited theme, but Plasma itself will require OpenGL due to being based on QtQuick 2. I do hope that llvmpipe will be sufficient for Plasma in case of non available drivers. The same is true for KWin: our QtQuick based elements will require OpenGL 2, but that does not mean that we will require OpenGL based compositing. We will still provide non-composited mode and if the XRender based compositor get’s ported to XCB (we have lots of work there to get involved 😉 will continue to provide XRender based compositing as well as OpenGL 1 and OpenGL 2 based. Since the refactoring of the OpenGL based compositor I do not see much difficulties with keeping the OpenGL 1 based compositor around, though if it turns out to not be used anymore, it might be removed.

The last remaining question is what will change with Wayland? Obviously Wayland requires to have a compositor, but it does not require us to stop supporting X11. KWin will continue to be an X11 based window manager allowing you to not use Wayland at all. And even if you choose to use a Wayland based KWin nothing says that it needs to be a compositor on top of OpenGL. Whether we will provide non-OpenGL based compositors for Wayland is a question for the future, though.

A final remark: this blog post describes the situation inside the KDE Plasma workspaces and the design decisions we have done. This is not a critic at the design decisions other projects have done and I am not able to judge whether their decision is right or wrong and it is completely irrelevant what my opinion is on their decisions.

A journey through virtualization

This weekend I had a look on enabling the OpenGL compositor when running in a virtual machine. My assumption was that given that the next version of Ubuntu is going to require OpenGL, there should be no problem with getting OpenGL running in a virtual machine.

I normally use KVM when I have need for a virtual machine running a Linux guest. As far as I know you cannot get OpenGL with this technology, so I hadn’t looked into it for quite some time.

My first attempt was to just use the project neon weekly build. Unfortunately I failed to convert it to a VirtualBox image, nevertheless I decided to run it and verify that KVM does not support OpenGL. Well I was rather surprised once the machine was up to see that OpenGL was functional, but well it was slow. Restarting KWin showed me that there is a missing slot (which I am aware of and is already fixed in one of the pending reviews) in master which makes the fallback to XRender not work, but KWin just uses llvmpipe. OK, result incorrect, but at least validated that KVM is not working.

I then decided to try VirtualBox with an install of Kubuntu Natty. Install went well but no OpenGL available. The Xorg log file told me that VirtualBox does not know the XServer version. Installed the latest version of VirtualBox, installed the latest version of guest additions in the Kubuntu guest, but no success: driver does not work.

So I decided to test with an outdated system, aka Debian Testing as I’m also running this on my workstation. Here I was more successful. After installing the system in the virtual machine OpenGL worked out-of-the-box. And what surprised me even more was that OpenGL in KWin worked when using KWIN_DIRECT_GL=1 and KWIN_COMPOSE=O. This means OpenGL works in general and it’s only blocked by KWin not knowing the driver.

KWin with OpenGL in VirtualBox

The main reason is that KWin does not use direct rendering if it finds an unknown driver and VirtualBox only provides OpenGL in direct rendering mode. If LIBGL_ALWAYS_INDIRECT is set, it falls back to Mesa’s software rasterization and KWin sets this environment variable if it doesn’t know the driver.

This meant I had to perform some coding to recognize the driver and had to test it in the virtual machine. Thanks to host and guest running the same version of Debian I was able to setup a NFS share and just use my normal development version of KWin.

After a little bit of hacking and fighting with NFS I got the driver reported correctly and KWin running with OpenGL without using environment variable hacks. But VirtualBox’s driver is not complete. Some methods we (optionally) use from GLX 1.3 are not implemented and trying to use it resulted in warnings. That unfortunately needed a hack inside KWin (it’s small and well contained, so it’s not much of an issue).

With VirtualBox working I decided to also look into VMware and have installed VMware Player for the first time in years. Converted my Debian image into a VMware compatible format, enabled 3D acceleration and started the system, just to be presented with warnings that 3D is not working.

The Internet told me to add

mks.gl.allowBlacklistedDrivers = TRUE

to the vmx file. But that did not resolve the issue. The two warnings went down to one, but still a warning. Apparently one has to install libtxc-dxtn-s2tc0 in order to get 3D support.

Installed and no warnings: yeah. Started the system, but no OpenGL. Installed the guest additions, but it told me that the system already has X11 integration. Looked into the Xorg log and yes it complained that something is wrong with the module. So no OpenGL inside Debian guest.

KWin with OpenGL in VMWare player

Resolved to my previous plan of trying Kubuntu 12.10. Started it in a live session and KWin is not composited. But glxinfo reported a driver, so I restarted KWin with KWIN_COMPOSE=O and there it was: OpenGL running also in VMware.

Looking at KWin’s output I was quite happy to see that it is a Gallium3D driver, which means we don’t need any adjustments at all. KWin uses OpenGL out-of-the-box, it was only not running, because of the live session of Kubuntu. So all that was needed was adding some detection code, that KWin doesn’t print out “unknown” in the debug output. Unfortunately using my KWin from Debian did not work, so I could not yet verify the output, but that can be done once it hit project neon.

Once the patches are merged into master KWin will use OpenGL in virtual machines – if supported. But given the experience it looks like at least on Linux hosts OpenGL is basically not supported in default setups. I would like to also support Parallels and Windows Virtual PC, but am of course lacking both required host operating system and licenses.

To llvmpipe or Not?

Some days ago I prepared a patch which moves the decision which compositor to use inside the driver detection code. This allows us to remove hacks we currently have inside the compositor initialization code to move specific drivers (e.g. the software rasterizer) to a better suited compositor. The base idea is that we always give a user the compositor which is best suited for her system. It’s in my opinion not a good idea to push users on the OpenGL 2 compositor, just because the hardware supports the required extensions, if we know that the hardware is actually not capable of performing well. For such users the OpenGL 1 based compositor is better suited or maybe even the XRender based compositor.

As well for rather old hardware it’s a better idea to not provide any OpenGL based compositing even if theoretically the driver supports it.

But automatism is not always the best solution. Yes giving from experience the best suited compositor is a good idea, but still it makes sense to allow the user to overwrite it. In this case the “user” is mostly myself and other developers who have to be able to easily switch the compositor. For me this is most easily done using an environment variable as I am restarting KWin all the time anyway and is faster than going into a UI and changing the settings and afterward changing them back. Of course although this is controlled through an environment variable, there are also settings to control the chosen compositor. E.g. you can select XRender or disable the OpenGL 2 based compositor. But there is no UI option to enforce the OpenGL 2 compositor if our driver recommendation is to use OpenGL 1 or XRender (we don’t want to provide users the option to destroy their system).

One of the side-effects of this driver based recommendation is that the hack to switch to XRender for software based rasterizers has been removed in the patch. Instead these drivers now recommend to use the XRender based compositor. And in combination with the changes in the handling of the KWIN_COMPOSE environment variable this results in the possibility to run OpenGL based compositing with the llvmpipe driver.

But as you can see one has to manually enforce it through an environment variable and I would not recommend anybody (or any distribution) to use it. OpenGL based compositing over llvmpipe is considered as a fallback approach for other OpenGL based desktop environments like GNOME Shell and Unity. But I do not consider it as a solution for the KDE Plasma Workspaces. I think we have better solutions for fallback (XRender or no compositing).

Let’s have a look on the use cases for llvmpipe based compositing:

  • Outdated hardware not providing OpenGL
  • Too new hardware without driver yet
  • Embedded hardware without driver yet
  • Virtual machines
  • RMS not like binary drivers

If your hardware is too old to provide the requirements for OpenGL based compositing, it is quite likely that your CPU is also rather old. We can assume a single-core CPU of either the Pentium era or an early Atom. You don’t want your CPU which is not suited for this task to spend all the time rendering the desktop. If I enable llvmpipe for KWin on my system (quad core) it puts about one core under full steam.

The case that there is new hardware without drivers available on Linux has luckily become a very rare case. Basic modesetting support for NVIDIA’s latest hardware (Kepler) was available on the same day as the announcement of the hardware. And the blobs do support also their latest hardware. So in most cases all that is needed is installing the latest driver or upgrading the distribution. If you spend lots of money for new hardware you probably want to use it and not have the CPU emulate what your expensive hardware is supposed to do.

In the case of embedded hardware it’s unfortunately still rather common that there are no drivers available. But ARM based systems are also not known for having extreme power. If you don’t want your high-end system to use llvmpipe based OpenGL compositing you even less want your low-end system to do it. You bought your Raspberry Pi hopefully for more useful things than running at full-steam to render the desktop.

If you run a virtual machine your resources are rather limited. It needs to share the resources with other virtual machines and your host system. If you run in a cloud you probably don’t want to pay for rendering the desktop through llvmpipe. You have a task to solve and the more CPU that task gets the less you have to pay. In case of a local system you also don’t want to have the virtual system run at full-steam just to render the desktop. There are really better things to do with a VM.

Last but not least there are the users who don’t want to run binary drivers. Sorry I cannot help you there. If you really think you want to spend lots of money for hardware that you don’t use, that’s fine with me. But I would suggest to only buy hardware which has free drivers available and there are nowadays options from all of the three big GPU providers.

Overall we see that llvmpipe as a backend for OpenGL based compositing is actually not needed in the case of the KDE Plasma Workspaces. We have working fallbacks like XRender or no compositing at all and those do not put the system under full-steam.

Given that I am not considering to turn llvmpipe into an option for KWin. I think that most users who would need llvmpipe are better suited with XRender based compositing even if a few effects are missing then. For me as a developer on the other hand llvmpipe is a very interesting target as it can help to optimize our rendering stack.

Of course what I considered in this blog post is only relevant for OpenGL based compositing of the KDE Plasma Workspaces. We do have proper fallbacks like XRender or no compositing which is not available in other environments which decided to use llvmpipe as a fallback. They have to do a different evaluation and I cannot provide it for them as my evaluation result would be: use KWin 😉