Distribution management – how Upstream ensures Downstream keeps the Quality

I read Emmanuele Bassi’s very interesting blog post about software distribution this week and thought a lot about it. Emmanuele kind of answers to a presentation by Richard Brown (from OpenSUSE fame). While I haven’t seen that presentation, I saw a similar one last year at the openSUSE conference and also talked with Richard about the topic. So I dare to think that I understand Richard’s arguments and even agree with most of them.

Nevertheless I want to share some of my thoughts on the topic from the perspective of KWin maintainership. KWin is more part of the OS stack, so distributing through means like Flatpack are not a valid option IMHO. As KWin is close to the OS, we have a very well integration with distributions. Plasma (which KWin is part of) has dedicated packager groups in most distributions, we have a direct communication channel to the distros, we know our packagers in large parts in person, etc. etc. So from the open source distribution model we are in the best category. We are better positioned than let’s say a new game which needs to be distributed.

So this is the background of this blog post. What I’m now going to do is to describe some of the issues we hit just over the last year with distributing our software through distros. What I will show are many issues which would not have been possible if we distributed it ourselves without a distro in between (for the record: this includes KDE Neon!). To make it quite clear: I do not think that it would be a good idea for KWin to be distributed outside distributions. While thinking about this topic I came up with a phrase to what we are doing: “Distribution management”.

We do lot of work to ensure that distributions don’t destroy our software. We actually manage our distributions.

Latest example: QML runtime dependency

Aleix created a new function in extra-cmake-modules to specify the QML runtime dependencies. Once it was available I immediately went through KWin sources, extracted all QML modules we use and added it to CMake, so that it’s shown when running CMake. Reference: D7273

This is a huge issue in fact. Plasma 5.10 introduced support for virtual keyboard in the lock screen. We had to add special code for handling that this runtime dependency is missing. I sent a mail to our distributions mailing list to remind distros to package and ship this dependency.

Obviously: if we distributed the software by ourselves this would not have been an issue at all. We would just bundle Qt Virtual Keyboard and be done. But as we are distributed through distributions we had to write special handling code for the case it’s missing and send out mails to remind distros to package it.

Incorrect dependencies – too many

Although we specify all required dependencies though CMake our distributions might add random other dependencies. For example KWin on Debian depended on QtWayland (both client and server), although KWin does not need the one or the other. This got fixed after I reported it.

This is of course a huge problem. A common saying about KDE is that it has too many dependencies and that you cannot install software on e.g. GNOME, because it pulls in everything and the kitchen sink. That’s of course true if distros add incorrect additional package dependencies.

Incorrect dependencies – too few

Of course the other side is also possible. One can have too few dependencies. Again the example is Debian, which did not make KWin depend on Breeze resulting in incorrect window decoration being set. That we depend on it at compile time is also following the “distribution management” as distros asked us to add all dependencies through CMake. So we did and made KWin optionally depend on Breeze. Of course such distribution management does not help if distributions don’t check the CMake output.

Also here if we distributed ourselves we would have compiled with all dependencies.

Outdated dependencies

Another very common problem is that distributions do not have the dependencies which we need and want to use. In case of KWin this especially became a problem during the Wayland port. For a very long time we had to keep Wayland compile time optional so that distributions like Slackware [1] could continue to compile KWin. When we turned it into a hard dependency we had to inform distros about it and check with them whether it’s working. Of course just because you inform distros does not mean that it will work out later on.

That we have distros with outdated dependencies is always a sore pain. Our users are never running the software which we are, never getting the same testing. We have to keep workarounds for outdated dependencies (e.g. KWin has a workaround for an XWayland issue fixed in 1.19) and that of course results in many, many bug reports we have to handle where we have to tell the users that their software is just too old.

Also we face opposition when trying to increase dependencies, resulting in distros consider dropping the software or to revert the changes.

Handling upgrades

Another reoccurring issue is handling updates. Our users are constantly running outdated software and not getting the bug fixes. E.g. Ubuntu 16.04 (LTS) ships with KWin 5.5. We currently support KWin 5.8 and 5.10. We constantly get bug reports for such old software and that’s just because it’s outdated in the distribution. Such bug reports only cause work for us and are a frustrating experience for the users. Because the only feedback they get is: “no longer maintained, please update to a current version”. Which is of course not true, because the distro does not even allow the user to upgrade.

Even when upgrading it’s a problem. We have to inform distros about how to handle the upgrade, of course that does not help, it fails nevertheless. We also never know what are allowed upgrade paths. For us it’s simple: 5.8 upgraded to 5.9, upgraded to 5.10, etc. If we would ship upgrades ourselves we could ensure that. But distros might go from 5.8 to 5.10 without upgrade to 5.9 ever. So we need to handle such situations. This was extremely challenging for me with a lock screen change in Plasma 5.10 to ensure the upgrade works correctly.

Random distro failure

An older example from Debian: during the gcc-5 transition KWin got broken and started to crash on startup without any chance to recover. This issue would of course not happen if we would have distributed KWin ourselves with all dependencies.

The experience was rather bad as users (rightfully) complained about the brokeness of KDE. I had friends asking me in person how it could be that we ship such broken software. Of course we were innocent and this was only in the distro. But still we (KDE) get the full blame for the breakage caused by the distro.

Useless bug reports

Thanks to distributions all crash reports we get are useless. The debug packages are missing and that’s pointless. Even if debug packages are installed mostly the crash point is missing. This is especially a problem with Arch as they don’t provide debug packages. In 2017 KWin got 71 bug reports which are marked as NEEDSINFO/BACKTRACE as the reported bug is pointless.

Misinformed maintainers

I don’t know how often I got asked this: should we change Qt to be compiled against OpenGL ES? Every time I was close to a heart attack as this would break all Qt applications for all NVIDIA users (at least a few years ago there was no GLES support in the NVIDIA driver). Luckily in most cases the maintainers asked (why me and not Qt?), but I remember one case where they did not ask and just rolled it out. With the expected result.

Thoughts

I could go on with examples for quite some time. But I think it should be obvious what I want to show: even for well integrated software such as KWin the distributions are not able to package and deliver correctly. As an upstream developer one has to spend quite some time on managing the distributions, so that the software is delivered in an acceptable state to the users.

Distros always claim that they provide a level of quality. Looking at the examples above I have to wonder where it is (granted there are positive exceptions like openSUSE). Distros claim they do a licensing check. That’s useful and very much needed, but is it required that openSUSE, Debian and Arch do the same check? Furthermore I personally doubt that distros can do more than a superficial check, they would never find a case where one copies code which is license incompatible.

Given the experience I have made with distros over the last decade working on KWin I am not surprised that projects are looking for better ways to distribute software. Ways where they can control the software distribution and ensure it works. Ways where their software is not degraded due to distros doing mistakes.

All that said, I do agree with Richard in most points, just don’t think it works. If everybody would use openSUSE than Richard would be right with almost all points. But the reality is that we have a bazillion of distributions doing the same work and repeating the same mistakes. Due to that I think that for software distribution Flatpack & Co. are a very valid approach to improve the overall user experience.

[1] Slackware for a long time did not have Wayland as that would need Weston and Weston depends on PAM which is only the optional fallback for no logind support, which Slackware of course does not have

19 Replies to “Distribution management – how Upstream ensures Downstream keeps the Quality”

  1. As a user of Arch Linux I miss debugging packages as well a lot to make proper bug reports. But even on opensuse they are not installed by default and if I’m not wrong a lot of bugs are not recorded there at all and send to opensuse. In this regard I think fedora is one of the good examples where even low level bugs can be collected. I think we need a very easy distribution and desktop independent bug report infrastructure… It’s frustrating to see bugs but it is way to complex for normal users to properly report them.

  2. Thanks for your true comment, without blaming other, you emphasis problems we users and we packagers both face.

  3. What do you think of downstream bug reports (report the bug directly in the distribution)? I talked once with an upstream developer, who claims the best way to report a bug is the distribution and if the distribution maintainers recognize an upstream bug, they would report it to upstream.

    1. So instead of one bug tracker to investigate issues one has 20. That doesn’t sound like a good idea to ensure issues get fixed.

    2. (I hope the pro/con list formatting doesn’t get screwed)

      I’m a bit torn over this as well:

      Report downstream (first):

      + Downstream is aware of the bug and can help reproducing it and testing fixes

      + Downstream is aware of the bug and can integrate the fixes into the packages once available

      + Bugs which are somehow caused by downstream (build-time configuration, packaging, broken backports) can be
      identified without needlessly involving upstream

      – It can take longer for the report to reach upstream or it might get lost completely

      – Two reports may have to be managed, more work for the reporter and downstream bug triagers

      Report upstream (first):

      + Bugs can be fixed quicker

      + Upstream has a better overview of the bugs, makes triaging easier

      + Less work for downstream bug triagers and the reporter

      – Fixes can take longer until they reach downstream, usually until the next release

      – Downstream issues might land in upstream bug trackers

      – The reporter might need help for providing necessary info such as debug packages, but upstream might not be familiar with the reporter’s environment to help

  4. While all of that is true, I, as a user, prefer to live with bugs I know until upgrade time comes rather than new bugs every day, which is likely result of riding the edging.

  5. I’m glad you wrote this article, and I agree with most of it, with one exception, named “KDE Neon”.
    I started to use Plasma (after 13 years of KDE 3/4) on Kubuntu 16.04, then read your post about inconsistency with the delivered Qt versions. Therefore I opted for KDE Neon user LTS to wipe away all dependency doubts….and found the same bugs.
    After months of frustration, I moved to KDE Neon User Edition, to once again clear some doubts and stay closer to what KDE developers probably use. And again, the situation didn’t improve.
    Last but not least, a kernel update which broke the mouse cursor, rendering the KDE experience totally awful. Installed Gnome (with related vomit spasms) and everything there was fine with the same packages base.
    So, you can say everything you want about distros in general, but a “loyal” KDE user who choose Neon, shouldn’t run into inconsistencies. Otherwise where is this advertised “KDE experience” gone ?
    Plus, let me tell you one thing: when are you (Neon) guys going to get rid of SDDM ? It’s completely broken. With a 4k display the greeter is awful. Most of the times the avatar picture is broken in the lock screen. And now I have to restart it every time to get rid of the mouse cursor issue.
    What happened to the good old KDM ?
    Also, since I’m here and since I think it’s a KWin related thing, when are you going to fix multiscreen ?
    Every time I plug a monitor on the left side of my laptop, windows go crazy. They open in the wrong monitor, my laptop autonomously go into 4k (when it was set to FHD) and so on. Again, terrible KDE experience.
    Please don’t take this as ranting, but more like a “guys, what’s going on ??”. I know you are all very skilled developers and seeing such awful issues makes me think there’s a serious underneath problem in testing KDE before delivering. Please don’t answer “we’ve got a gazillion of test units”. I am a Qt developer myself and we both know test units don’t always catch bugs.
    So probably the main question is: how do you guys test KDE to say “this is good to go” ?

    1. Please note that I’m not a Neon developer and that Neon is not the KDE reference distribution.

  6. I agree with everything and this explains why with the same version of Plasma in Opensuse you have a great release, while on other distributions there are always problems.

  7. Out of curiosity (and as a thought experiment) could you package plasma as a snap or flatpack?

    what would you need (high level) integration wise from a distro for it to work?

    1. Parts yes, but overall no. E.g. Plasma provides integration for flatpack. If we put that into a flatpack it cannot work as other flatpacks won’t see it. Or KWin/Wayland requires to become the loginctl session master in order to take over the DRM device. If flatpack allows that we have a big security issue 😉

      1. The login screen might be easy enough to solve. Configure system to run flat pack x for loginctl, have it communicate via a dbus api for login check. If system thinks authentication is OK, it launches flat pack y for the desktop.

        As for integration with other flat packs, not sure. Would have to have some sort of anointed process.

        It would require a common library underneath that could be shared across desktops and distros.

        Interesting thought experiment though

  8. I’m somewhat involved in a smaller project and we’ve experienced some of the same issues. Like packagers deciding to not honor our requirements of statically linked library versions because either we had to patch them or upgrading them would break things because the library doesn’t have a stable API. This lead to severe bugs and lots of fighting to get fixed, almost to the point of asking them to remove their package from their distribution because we would get most of the bug reports and complaints. Luckily it’s solved for now, but I expect it to pop up again – it seems to do that every few years.

    Anyway I really hope we can start distributing our software independently but also have proper integration in distros by the way of Software Centre and the like. This may also have the positive effect that distributions won’t need hundreds of packagers for projects not in the core installation, and rather focus these efforts to things like KWin which need full privileges to run.

  9. Why can’t you just add more configuration/build time checks? I’d say this problem is even deeper than distros: any user could go on to try compiling a software (for experimentation, development, other reasons), and sure if there’s no compile time checks for dependencies, they is screwed. This is end-user experience too.

    Aside note: did you change lastname? 🙂

    1. …I do even vaguely remember cases, when I have compiled something (irrelevant to KDE), and then the software is segfaults or something alike, and I am happily start debugging with the thought I can contribute to a project. And after spending a lot of time turns out that the problem is that I didn’t install some dependency which mentioned in README.

      Sure I patched the build files, but can you imagine how I am frustrated — the devs knew about the problem, and all they did is a note in README, so I wasted my time for nothing.

      Everything that could be automated should be automated.

      1. our compile time checks are properly done. We use CMake for it, so that works fine. The problem is more that we must do lots of compromises to support everything the distros out there need. If it were just users compiling it would not be a problem. We could have very strict requirements on the dependencies and it would work. But due to the distros one need to go large hoops to ensure it still works with them and make required dependencies optional and stuff like that. If it were for me, there would not be such a thing as an optional dependency in KWin.

Comments are closed.