Evolving 3D desktop effects in Plasma

The latest Plasma release dropped a few desktop effects: the cube family, CoverSwitch and FlipSwitch. All of those effects were written back in 2008, the early days of KDE 4.x and the early days of desktop effects in KWin. The effects were implemented by me and when Vlad asked about removing them I saw the need for this and supported this step for technical reasons. With this blog post I want to share a little bit of why it was needed to remove them and why this means that they can come back in better ways than ever before.

State of CPU in 2008

To really understand this we need to time travel back to 2008 and the years before when desktop effects were introduced. This can help to understand how the hardware architecture changed and how that influenced design decisions in the effects API which are nowadays problematic. First of all CPUs. The Intel Core 2 Duo architecture was launched in 2006 as the brand new thing which had multiple (namely 2) cores which slowly replaced the NetBurst architecture which dominated Desktop computing for the beginning of that decade.

Also for developers these multi-core systems were a new thing. And that meant lots of software written before was single threaded. Especially KWin at that time was single threaded as it also depended on libraries which were not really thread safe such as the Xlib library and back at that time even OpenGL. Even years later when Qt introduced the threaded rendering in Qt 5 on many Mesa drivers this was disabled due to thread safety issues. Nobody would have thought of having any benefit of a threaded compositing approach back in 2007 with the state of hardware and the available libraries. Thread libraries were of course already available such as QtConcurrent or ThreadWeaver, but not of a useful use in KWin. This means the API written back then did not support ideas like doing rendering on a second thread or even rendering for each screen in a thread.

State of OpenGL in 2008

KWin’s compositing pipeline was written for OpenGL 1.4 which was the only supported version in the Mesa stack. KWin supported shaders written in OpenGL Shading Language which replaced parts of the fixed functionality pipeline in OpenGL 1.4 and emulated the fixed functionality pipeline. This was only supported on the proprietary NVIDIA driver. OpenGL 3 was released in 2008, but it was years till it became available on the Linux desktop (according to Wikipedia in 2012).

Thus the design of the OpenGL compositing scene and desktop effect system was written for the fixed functionality and only with this in mind. While KWin gained support for the programmable pipeline and is the only supported way nowadays the design is still in place and problematic.

State of multi screen support

Multiple screens were not that common back in 2008 and there existed multiple competing technology approaches. There was the old “one X screen per screen” approach with a different DISPLAY variable for every screen. There was Xinerama, with the NVIDIA implementation called TwinView and there was the “new kid on the block” XRandR. Using multiple screens still required hacking the xorg.conf file and then with luck it was working. Especially if you had an NVIDIA card which one needed for good desktop effect performance.

 

From an X11 perspective there was (and even today) there is not such a thing as multi screen. For the compositor everything is one screen and we have to present all screens at the same time. So much for variable refresh rates (AMD FreeSync introduced in 2015), buffer age extension (implemented in KWin in 2013) and so on. From rendering point of view there was not much difference between rendering one screen or multiple screens. All we had in KWin was an integer variable telling us the count of known screens and the geometries.

State of Alt+Tab

Alt+Tab, which CoverSwitch and FlipSwitch provide, was still a hard coded implementation in QWidget. With the effect system an API was added to suppress the QWidget and use a desktop effect as replacement. This allowed to have an effect (BoxSwitch) which showed thumbnails for the windows. Overall this was rather a break with the other parts of the effects API as the effects are mostly used to influence position and drawing of the windows. With the Alt+Tab API an internal part of KWin was exposed and suppressed. It was it’s own API inside the API.

CoverSwitch and FlipSwitch even took this a step further by introducing 3D elements in the so far 2D world of desktop effects and completely intercepted the rendering. Most effects do not change the order how windows are rendered, but with those effects it was important to render the windows exactly in the order Alt+Tab wants them. So the effect intercepts the normal render calls and renders the windows in another order. To make this worse the CoverSwitch included reflections which meant the windows needed to be rendered twice. And the effects had to combine windows from multiple screens. The cube effect family did even more horrible things.

How Alt+Tab evolved

The biggest change to Alt+Tab happened thanks to QtQuick. While it was reworked before already and was no longer QWidget based, QtQuick allowed to easily define new layouts. The internal Alt+Tab API was one of the first parts to be exposed to QtQuick in KWin. Furthermore we introduced an interesting concept to make it possible to render window thumbnails in the QtQuick scene. This was done by telling the compositor to draw a thumbnail at a specific region of the QtQuick Alt+Tab window. As this was perfectly synced we finally had an easy to use API to make Alt+Tab layouts with all the fancy things only desktop effects could give us. Though we could not put other elements on top of the thumbnails (e.g. a close button) or transform them from within QtQuick. This was the end to the already mentioned BoxSwitch effect and the Alt+Tab mode in PresentWindows. Thus the only remaining effects for Alt+Tab are FlipSwitch and CoverSwitch.

With Qt 5 there was the hope to further improve that. Now also QtQuick used OpenGL for rendering which meant that in theory it would be possible to make our window textures available to QtQuick. Thanks to the work on the compositing engine lately this is now possible and gives even more flexibility to render Alt+Tab and makes it possible to implement FlipSwitch and CoverSwitch with QtQuick. This is really awesome as it means we have much better tools at hand to implement such fancy effects, we don’t have to develop our own toolkit and implement our own transition handling. Instead we can use all the great things like PathView in QtQuick. The C++ desktop effect implementation of CoverSwitch was 1000 lines of code, while the new emerging QtQuick based implementation is just about 200 lines of code.

How the effects evolved

Also the KWin effect system had a transition. We noticed that most effects are actually animations and added a dedicated implementation for it. This implementation is exposed to JavaScript and most of those effects are nowadays written in JavaScript. The effects in C++ are often the odd ones which do too much and use the wrong toolkit for the wrong thing. Such as I said years ago that PresentWindows and DesktopGrid need to be rewritten in QtQuick. The effect system should be for animating, not for developing a user interface. With QtQuick we have a way better toolkit.

Overall huge effects such as the cube family, CoverSwitch and FlipSwitch are standing in the way of evolving the effect system. While we have better tools to implement and maintain them thanks to QtQuick.