Posted on

Evolving 3D desktop effects in Plasma

The latest Plasma release dropped a few desktop effects: the cube family, CoverSwitch and FlipSwitch. All of those effects were written back in 2008, the early days of KDE 4.x and the early days of desktop effects in KWin. The effects were implemented by me and when Vlad asked about removing them I saw the need for this and supported this step for technical reasons. With this blog post I want to share a little bit of why it was needed to remove them and why this means that they can come back in better ways than ever before.

State of CPU in 2008

To really understand this we need to time travel back to 2008 and the years before when desktop effects were introduced. This can help to understand how the hardware architecture changed and how that influenced design decisions in the effects API which are nowadays problematic. First of all CPUs. The Intel Core 2 Duo architecture was launched in 2006 as the brand new thing which had multiple (namely 2) cores which slowly replaced the NetBurst architecture which dominated Desktop computing for the beginning of that decade.

Also for developers these multi-core systems were a new thing. And that meant lots of software written before was single threaded. Especially KWin at that time was single threaded as it also depended on libraries which were not really thread safe such as the Xlib library and back at that time even OpenGL. Even years later when Qt introduced the threaded rendering in Qt 5 on many Mesa drivers this was disabled due to thread safety issues. Nobody would have thought of having any benefit of a threaded compositing approach back in 2007 with the state of hardware and the available libraries. Thread libraries were of course already available such as QtConcurrent or ThreadWeaver, but not of a useful use in KWin. This means the API written back then did not support ideas like doing rendering on a second thread or even rendering for each screen in a thread.

State of OpenGL in 2008

KWin’s compositing pipeline was written for OpenGL 1.4 which was the only supported version in the Mesa stack. KWin supported shaders written in OpenGL Shading Language which replaced parts of the fixed functionality pipeline in OpenGL 1.4 and emulated the fixed functionality pipeline. This was only supported on the proprietary NVIDIA driver. OpenGL 3 was released in 2008, but it was years till it became available on the Linux desktop (according to Wikipedia in 2012).

Thus the design of the OpenGL compositing scene and desktop effect system was written for the fixed functionality and only with this in mind. While KWin gained support for the programmable pipeline and is the only supported way nowadays the design is still in place and problematic.

State of multi screen support

Multiple screens were not that common back in 2008 and there existed multiple competing technology approaches. There was the old “one X screen per screen” approach with a different DISPLAY variable for every screen. There was Xinerama, with the NVIDIA implementation called TwinView and there was the “new kid on the block” XRandR. Using multiple screens still required hacking the xorg.conf file and then with luck it was working. Especially if you had an NVIDIA card which one needed for good desktop effect performance.

 

From https://xkcd.com/963/

From an X11 perspective there was (and even today) there is not such a thing as multi screen. For the compositor everything is one screen and we have to present all screens at the same time. So much for variable refresh rates (AMD FreeSync introduced in 2015), buffer age extension (implemented in KWin in 2013) and so on. From rendering point of view there was not much difference between rendering one screen or multiple screens. All we had in KWin was an integer variable telling us the count of known screens and the geometries.

State of Alt+Tab

Alt+Tab, which CoverSwitch and FlipSwitch provide, was still a hard coded implementation in QWidget. With the effect system an API was added to suppress the QWidget and use a desktop effect as replacement. This allowed to have an effect (BoxSwitch) which showed thumbnails for the windows. Overall this was rather a break with the other parts of the effects API as the effects are mostly used to influence position and drawing of the windows. With the Alt+Tab API an internal part of KWin was exposed and suppressed. It was it’s own API inside the API.

CoverSwitch and FlipSwitch even took this a step further by introducing 3D elements in the so far 2D world of desktop effects and completely intercepted the rendering. Most effects do not change the order how windows are rendered, but with those effects it was important to render the windows exactly in the order Alt+Tab wants them. So the effect intercepts the normal render calls and renders the windows in another order. To make this worse the CoverSwitch included reflections which meant the windows needed to be rendered twice. And the effects had to combine windows from multiple screens. The cube effect family did even more horrible things.

How Alt+Tab evolved

The biggest change to Alt+Tab happened thanks to QtQuick. While it was reworked before already and was no longer QWidget based, QtQuick allowed to easily define new layouts. The internal Alt+Tab API was one of the first parts to be exposed to QtQuick in KWin. Furthermore we introduced an interesting concept to make it possible to render window thumbnails in the QtQuick scene. This was done by telling the compositor to draw a thumbnail at a specific region of the QtQuick Alt+Tab window. As this was perfectly synced we finally had an easy to use API to make Alt+Tab layouts with all the fancy things only desktop effects could give us. Though we could not put other elements on top of the thumbnails (e.g. a close button) or transform them from within QtQuick. This was the end to the already mentioned BoxSwitch effect and the Alt+Tab mode in PresentWindows. Thus the only remaining effects for Alt+Tab are FlipSwitch and CoverSwitch.

With Qt 5 there was the hope to further improve that. Now also QtQuick used OpenGL for rendering which meant that in theory it would be possible to make our window textures available to QtQuick. Thanks to the work on the compositing engine lately this is now possible and gives even more flexibility to render Alt+Tab and makes it possible to implement FlipSwitch and CoverSwitch with QtQuick. This is really awesome as it means we have much better tools at hand to implement such fancy effects, we don’t have to develop our own toolkit and implement our own transition handling. Instead we can use all the great things like PathView in QtQuick. The C++ desktop effect implementation of CoverSwitch was 1000 lines of code, while the new emerging QtQuick based implementation is just about 200 lines of code.

How the effects evolved

Also the KWin effect system had a transition. We noticed that most effects are actually animations and added a dedicated implementation for it. This implementation is exposed to JavaScript and most of those effects are nowadays written in JavaScript. The effects in C++ are often the odd ones which do too much and use the wrong toolkit for the wrong thing. Such as I said years ago that PresentWindows and DesktopGrid need to be rewritten in QtQuick. The effect system should be for animating, not for developing a user interface. With QtQuick we have a way better toolkit.

Overall huge effects such as the cube family, CoverSwitch and FlipSwitch are standing in the way of evolving the effect system. While we have better tools to implement and maintain them thanks to QtQuick.

Posted on

Window and Desktop Switcher moved to Look’n’Feel Package

Today we did an important change in how KWin will distribute its assets in the upcoming 5.2 release. When we started our thoughts about the Look’n’Feel Package and how we want to have meta themes for the complete Plasma workspace we also wanted to have this for the Window and Desktop switcher provided by KWin. So the structure of the Look’n’Feel Package already has all the pieces for including the Window and Desktop Switcher, but it was not used. Now we finally addressed this for the 5.2 release and moved the default switcher into the Look’n’Feel Package and KWin can locate the switchers from the Look’n’Feel Package.

At the same time we want to follow in a better way the “Simple by default, powerful when needed” approach. Our configuration should be simple and thus not offer an overwhelming amount of switchers. But it also should be powerful and thus one can install additional switchers either through GHNS or through your package manager. The result is that all the additional switchers so far shipped with KWin were moved to kdeplasma-addons repository. So all switchers are still available (powerful when needed), just not shown when opening the configuration menu (simple by default).

This also opens the door for including more switchers in the plasma addons repository. So far we had been very reluctant to add more switchers to the KWin repository. There were already too many switchers installed by default and it gives a feeling of we don’t know what we want. By only installing one switcher by default this improves significantly and allows to add high quality switchers with enough differentiation to other switchers to the plasma addons repository.

The change has also some implications for users of non Plasma desktop environments wanting to use KWin as their window manager. By moving the switchers out, KWin removes some of the Plasma dependencies. All switchers provided by KWin are using Plasma components, the default switcher is part of the design concept for Plasma 5 following the same idea as other similar components. Thus KWin had a direct dependency on Plasma with the window switchers. This is now kind of solved by not offering any switcher at all.

My suggestion for desktop environment projects wanting to use KWin is to provide their own default Look’n’Feel package with a Window and Desktop switcher specific for their environment. On the other hand I don’t see a problem with providing a simple fallback theme in KWin. And of course there are still the Desktop Effects for switching between Windows (CoverSwitch and FlipSwitch) installed by default and just need to be enabled.

Posted on

Quick Window Switchers in KWin 5

One of the oldest usages of QtQuick in KWin are the Window Switchers. During the design it was aimed for having them Plasma-styled just like the previous implementation. This was not as easy as one could hope, because back then we couldn’t use the Dialog component provided by PlasmaCore. So all the various Switcher styles implemented the Plasma styling themselves and the C++ side did some additional adjustments like guessing the blur region. As the Plasma theme recognized that the blur effect is active the FrameSvg used as a background was rendered in the blur-available translucency state. To have it readable it needs to be blurred.

For this we would the correct blur mask of the FrameSvg but that’s not available from the Qml bindings. The hack we went for was to just assume in the C++ side that the switcher is Plasma styled, created another FrameSvg of same size and extracted the blur mask from that. Sounds fragile, doesn’t it?

And of course there came the day when that broke. About a year ago we noticed that the shadow doesn’t work any more and we needed to bring in some more fixes to make that work. If we would have used a PlasmaCore Dialog this would not have been an issue at all.

Now after the Qt 5 port the blur and shadow was once again broken and once again I was about to start fixing it. But nowadays we use the Plasma Dialog in e.g. declarative KWin scripts (the desktop change OSD being a prominent example). So I took a step back and thought about instead of fixing the fragile code to re-think it and make it possible to use the Plasma Dialog or in general QtQuick Windows from inside the Qml code.

Given that this required to adjust all switchers anyway I decided to go a step further and also improve the API a little bit. So far all the important data was exposed as context properties. I’m not a huge fan of context properties as it’s kind of magic. Instead there is now a dedicated switcher item available, which is just a QObject exposing a few properties:

import org.kde.kwin 2.0 as KWin

KWin.Switcher {
    property QAbstractItemModel *model // The model with either the windows or the virtual desktops
    property QRect screenGeometry // The geometry of the active screen (as KWin thinks of active)
    property bool visible // Whether the switcher is currently visible or not, a Window should follow this
    property bool allDesktops // Whether the switcher includes windows from all desktops or only the current one
    property int currentIndex // The index of the selected window - this is controlled from KWin side
}

The model provides the following roles for Window Switchers:

  • caption – The caption of the window
  • desktopName – The name of the virtual desktop the window is on
  • minimized – Boolean whether the window is minimized or not
  • windowId – Window system specific identifier
  • closeable – Boolean whether the window can be closed
  • icon – QIcon containing the window’s icon

With that we can design a very simple thumbnails switcher:

import QtQuick 2.1
import org.kde.kwin 2.0 as KWin
import QtQuick.Window 2.1
 
KWin.Switcher {
    id: switcher
    currentIndex: listView.currentIndex
 
    Window {
        id: window
        flags: Qt.X11BypassWindowManagerHint // important, without it won't work
        visible: switcher.visible
        color: "white"
        x: switcher.screenGeometry.x + switcher.screenGeometry.width / 2 - window.width / 2
        y: switcher.screenGeometry.y + switcher.screenGeometry.height / 2 - window.height / 2
        width: switcher.screenGeometry.width - 100 // keep 50 px space to each side
        height: 200 * (switcher.screenGeometry.width/switcher.screenGeometry.height) + 20
 
        ListView {
            id: listView
            orientation: ListView.Horizontal
            model: switcher.model
            spacing: 10
            anchors.fill: parent
            anchors.leftMargin: 10
            anchors.rightMargin: 10
            delegate: KWin.ThumbnailItem {
                width: 200
                height: width * (switcher.screenGeometry.width/switcher.screenGeometry.height)
                wId: windowId
                brightness: ListView.isCurrentItem ? 1.0 : 0.5
                saturation: brightness
                MouseArea {
                    anchors.fill: parent
                    onClicked: {
                        listView.currentIndex = index;
                    }
                }
            }
            Connections {
                target: switcher
                onCurrentIndexChanged: {
                    // currentIndex is updated from KWin side - need to keep in sync
                    listView.currentIndex = switcher.currentIndex;
                }
            }
        }
    }
}

I wrote this code directly in the blog post without any testing at all (hey it found one bug in KWin and two in the systemsetting previewer. I should do such stuff more often 😉 ). So let’s see how it looks like:

Aus TabBox

Not a bad result for 50 lines of code. As one can also see from the code it does not do any Plasma styling at all any more. Which is a nice side-effect for the usage of KWin outside the Plasma workspaces. The changes to get Plasma styling is not difficult either. Just use PlasmaCore.Dialog instead of the Window and that’s it.

Posted on

Window Switcher Love

One of my favorite features in KWin is Window Switching (aka Alt+Tab) and it is not surprising that it’s also one of the areas I prefer to work on and love to see others working on.

This week our Window Switchers have received quite some love from various contributors. The compact layout got some usability improvements and the highlight in all layouts does no longer move around when pressing Alt+Tab for the first time. Navigation with cursor keys when the list of windows is open, is properly supported, now. That is in a horizontal layout the left/right keys can be used to navigate through the list, in a vertical the up/down keys are supported and in a grid all keys are supported. Of course the normal navigation is still Alt+Tab/Alt+Shift+Tab – navigation through cursor keys is just an additional useful feature for those who want to use it.

Speaking of grid based layouts: I implemented a new QML layout in a “Present Windows” style. It is a layout showing thumbnails of all windows in the list in a regular grid and taking the complete available screen estate. By that it is similar to the previous Alt+Tab mode inside the Present Windows Effect. The code related to Alt+Tab has been removed from the effect, but otherwise the effect is of course unchanged.

Having this mode as a QML layout has quite some advantages. E.g. it is no longer required to have desktop effects enabled, though it is still recommended 🙂 The layout follows the visual style of our primary Desktop Shell which makes it easier to use. Selection is indicated by the well known selection marker of the Desktop Shell Themeing. Last but not least the layout properly supports multiple screens, which has not been a pleasant experience with the old effect based implementation.

Overall the new grid layout is also incorporating some of the feedback we received for the new layouts introduced in 4.8. Our default layout replaced the BoxSwitch effect, but there has been a slight difference in behavior. The layout makes only use of the horizontal space and by that thumbnails can become so small that they cannot be recognized any more. In the new QML based layout the thumbnails always have the same size at the compromise that some of them are perhaps not visible.

A grid based layout provides here quite some advantages as the screen estate is used in a more efficient way to render high quality and good looking thumbnails even with a large amount of windows.

I like to remember that we are currently selecting ideas on how to improve our default settings and switching to the grid based layout may be one of those. Also I want to remember that all those changes are rather trivial and could easily be implemented by new contributors and are also listed in our ideas page and as milestones for 4.9.

Posted on

KWin meets QML

Two years ago I started to re-implement the Window Switching capabilities of the KDE Plasma workspaces (also known as Alt+Tab or TabBox). The old implementation was based on a QWidget with custom painting which of course had some drawbacks. The new TabBox evolved into a framework around Qt’s Model-View concept to design your own window switcher.

Before I started to work on that area I did evaluate the existing Window Switching capabilities of the various desktop shells (both free software and proprietary solutions) and I noticed that all of them have drawbacks. None is the solution where I would say: that’s the perfect switcher.

The main issue is that depending on the user’s habits different work flows are required. A switcher optimized for users with few open windows fails for users with many open windows and vice versa. Especially thumbnails become useless for many open windows.

So I realized that we need something flexible enough to give users the possibility to use different layouts or to even define their own layout. This ended in a framework for window switching. The layout is described with an XML file and rendered through custom painting code in a Qt delegate.

When I implemented this framework I wanted to add a GUI to easily configure own layout. Unfortunately I never got around to implement this UI and I think there has never been any other layout except the default shipped once. It is a rather unknown feature and I’m quite unhappy that this great functionality never got implemented.

In the mean time the trolls did a great job to make UI development easier: QML. This makes it possible to do what I wanted to have for window switching: easy to design and exchangeable layouts. So this weekend I implemented the default (informative) layout in QML and plan to completely replace the existing View and Delegate code in 4.8. And this gives us the possibility to let users write their own layouts and to share and exchange them.

But this is only the beginning of usage of QML in KWin. I have many more ideas by making window thumbnails available to QML UIs in a limited way. This will allow us to even integrate the BoxSwitch effect into the normal window switching framework and replace the Window Strip in Plasma Active.

So great UI times ahead in KWin.