Menu Button inside Window Decorations

Peter Penz blogged about removing the Menu from Dolphin. While this is very interesting and nice looking I have a better idea: why not move the menu into the decoration? All what we need is already there. We have the awesome DBus Menu which allows us to send the menu to any other application (in most cases Plasma). So we could use this technology to direct the menu from the window into its decoration. Of course the menu should not be presented in its full completeness but be compacted into one dropdown menu – just like in the Netbook Shell.

So before you all start removing the Menu bar from your apps lets tackle it in a better way. If anyone is interested in working on it, please contact me 🙂

=-=-=-=-=
Powered by Blogilo

KWin at GSoC 2011

At this years Google Summer of Code it is of course also possible to work on KWin. At the KDE ideas page we have three projects listed, but it’s also possible to come up with an own idea. I will shortly outline the three projects:

  1. Refactoring of KWin core: KWin core has grown over the last decade and the development had a shift from being a pure X11 window manager to an OpenGL compositor with X11 window manager and soon to be only a Wayland compositor. In order to get to this new field we need to refactor great parts of the Window Manager to reuse the code in the future. As a matter of fact for this project a very close communication with me and the other developers is required. This is clearly the most important GSoC project we propose and I will be very selective on the student to choose. Developing skills are not the most important factor here, but soft skills will highly influence my decision.
  2. Unit Testing Framework for the X11 Window Manager: Since 4.6 KWin includes a scripting component which now allows us to execute small pieces of JavaScript. This component should be extended into a framework to run automated tests against the window manager. As it is clear this project is especially important for the refactoring task to ensure that any code changes do not break existing code. The student who wants to work on this task needs to be in constant contact with the student working on the refactoring to know what tests are required, soon.
  3. Initial support for Wayland clients: This is a research project and code written in this GSoC would most likely not be merged into master. The idea is to add Wayland clients to the compositor, so that they can be rendered just as a normal X11 window. As Wayland is still a highly moving target including the code directly into master does not make sense yet. The project would help to understand what needs to be done to support Wayland in KWin and would help to outline the porting. To make it clear: a complete port of KWin to Wayland is not possible in the scope of a GSoC.
Additional to the KWin mentored projects, there is also a proposed project by OpenICC to provide colour management in KWin. This project will also require collaboration with the KWin developers as the functionality needs to be either implemented as an effect or in the compositor directly.
Further KWin related topics are in the area of Plasma. There are quite some features implemented in Plasma which should be moved into the Compositor for better performance or parts of KWin which need to be made available to Plasma. Possible areas are joined screen edge handling, a merged context menu for window decoration and tasks manager, better Tooltip transitions by using an effect or synchronising window tabs and task groups. Many more areas exist. If you are interested feel free to contact KWin and Plasma developers.
I hope to see many good proposals and are looking forward to work with you to make KWin rock even more!

=-=-=-=-=
Powered by Blogilo

New KWin Shadows

One of the features which got killed in the process of porting KWin’s Compositor to OpenGL ES was the Shadow effect. For some time already the Shadow effect was not on the level we expect from our components. The code had become too complex to maintain and the port to OpenGL ES would not have been trivial. So the port was a good point to step back and think about what we actually need in the Shadow effect.

For quite some time we have an improved shadow system which allows the window decoration to provide the Shadow for decorated windows. This is used by e.g. Oxygen, QtCurve and Aurorae. So most windows already have a very advanced and good looking shadow. The Shadow effect itself is only used for so-called unmanaged windows such as menus or drop-down lists and for “legacy” window decorations such as Plastik.
The obvious disadvantage is that menus have a different shadow than all other windows. This means the designers do not have control over the shadows. But shadows are an important part of the design. A shadow should follow the widget style’s light model and by that one shadow to rule them all is just not possible.
Since I started to consider dropping the Shadow effect I have been in contact with mostly Hugo from Oxygen fame to gather the requirements for a new Shadow system. After long and fruitful discussions also including members from other Widget Styles and Compiz we designed a new system. Hugo recently implemented the Oxygen part and this weekend I implemented the first OpenGL version of the new system:
The main difference to the old system is that the Shadow is no longer an effect but moved directly into the Compositor. This follows the approach of the Decoration Shadows which are also directly in the Compositor. By moving the Shadow into the Compositor we can better control and optimize the rendering process. The compositor itself knows about the existence of the Shadow and can update the correct screen areas. As the shadow is rendered together with window it follows automatically opacity or transformations like wobbling.
A nice addition of the new Shadow system is that it is completely controlled by the to be rendered window which allows to use it also for Plasma elements such as the Panel shadow or in KRunner (yeah no more clickable Shadow). What could also be very nice is to use it for shaped windows. Up to now windows with custom shape could not get a Shadow as our effect just could not know if it looks good in the end. The window itself knows its shape and can provide a shadow adjusted for the window. So maybe Yakuake will soon have a Shadow 🙂
If someone wants to test it, you can build the kwin/oxygen-shadows branches in kdelibs and kde-workspace. There is still some work to be done like the XRender implementation, some optimizations and currently “normal” windows are not yet supported. Also we would like to have some fallback for windows not useing either Oxygen decoration or widget style.
I hope that we can bring our system as an addition to the EWMH specification. The system has been designed to not be KWin or KDE specific and it’s a usecase actually required in all Desktop Shells.

=-=-=-=-=
Powered by Blogilo

Comparing Commit Histories

This weekend I was looking a little bit at the bug statistics provided by Bugzilla. I was a slightly surprised by the fact that before 4.0 the numbers of open and fixed bugs hardly changed and started to skyrocket around the 4.0 release. During the last three years almost as many bugs have been fixed as in the eight years before. The number of duplicate reports even trippled, nevertheless the number of open bugs is constantly increasing. I will address the implications of this “bug problem” in a separate blog post.

As I joined the KWin development team around the 4.0 release I was interested in seeing how the commit rate around the time was. Did the development speed change, has there been a more severe impact to the codebase than at different times? Thanks to git I have the complete history around, so I did a useful formatting, wrote a small program to get the data I’m interested in and created diagrams using LibreOffice Calc[1].
Commits and Developers in KWin per year
As we can see around before 2008 KWin development clearly did not stagnate. Since 2006 the number of commits per year and developers who did at least one commit per year got a significant boost. This might be related to the porting to KDE 4 and the work on the compositor, but even after 2008 the commit rate was more or less the same. Till 2006 the avarage yearly commit rate was 360 commits, from 2007 till 2010 around 890 commits per year. Please don’t misinterpret the last data paint as a reduced activity. In fact the opposite is the case. At the end of 2010 I worked outside our SVN repository and the commits hit git in 2011 which explains that we already have half of the commits of 2010 in the (not yet) three month of 2011.
This brings me to the next point. How much does KWin depend on the development of a small group of developers? We can see that in average about 45 inidividual people committed to the KWin source code per year. Altogether 275 individual developers contributed to KWin and 10 developers have committed more than 100 changes, and a further eight have committed more than 50 changes. The top ten contributed 68% of all commits, Lubos alone contributed 28 % of all commits. Matthias Ettrich is with 4 % of all commits still in the top five. What really surprised me is that I already contributed 11 % of all commits and Hugo only working on the Oxygen Window decoration is third most active KWin developer – over the complete history.
Now the data is not perfectly accurate. KDE switched the version control system twice and the different VCS bring different patterns of work with them. Due to the bad branching feature of SVN large parts used to be developed outside of SVN and committed as a large bump. This is not the case any more with git: you have all commits out in the open. As at least I have been using git-svn bridge for quite some time, it’s not that surprising that I already have a large number of commits.
Now I had my tool to parse a git history into a statistic diagram and I thought what would be more fun than to compare the history to the one of another big window manager and compositor? So I dared to run the script on the Compiz history. The history in general is not comparable or more to say: it’s just numbers, it’s not possible to draw any conclusions from the history for comparing Compiz and KWin. Here are a few of the reasons:
  • The Compiz history only dates back to February 2006. According to Wikipedia Compiz was first released in January 2006. So some history is definitely missing.
  • Compiz had a vivid history. It was forked to Beryl, re-merged to Compiz Fusion, re-merged to Compiz, rewritten as Compiz++. I have no idea if the complete history of the forks is present in the repository.
  • Compiz has many, many plugins not living in the main repository. It is a question whether those have to be honored. KWin in comparison has all (with a minor exception of two or three effects) effects in the main repository.
  • KWin includes complex window decorations (c.f. the number of commits by Hugo) while Compiz includes wrappers around KWin’s and Metacity’s decoration. The theme engine for Compiz (Emerald) seems not to be part of the main repository.
  • Compiz code repository is split into several repositories. Here again the question which should be included into a comparison.
  • KWin is part of the KDE Plasma Workspaces, which gives us a slightly different development focus and the helping hands of many developers correcting common mistakes (e.g. i18n).
Given the constraints I decided to combine the history of Compiz core and of all the plugins in plugins-main. First again a look at the years:
We see that Compiz had a really strong start. In 2006 and 2007 Compiz had each time around twice as many commits as KWin, the difference is fading out lately. In 2009 KWin had more commits, in 2010 it’s very close and during the first three months of 2011 KWin has clearly more commits. What looks really bad in the Compiz graph is the loss of developers. In 2007 54 developers contributed to Compiz, in 2009 just 14. This illustrates the challenges Compiz faced in the beginning of 2009. At the end of 2008 the number of commiters per month dropped below 5!
Altogether Compiz had 92 developers. Three (David Reveman, Danny Baumann and Dennis Kasprzyk) have more than 1000 commits and make up more than 63 % of all commits. Compiz has only 7 developers with more than 100 commits and those 7 developers contributed more than 88 % of the complete code base. Only four more developers contributed more than 50 commits. Adding those we come to more than 92 %. Now to illustrate the problem I see here: David Reveman who is the initial author and holds 25 % of all commits did his last commit to the core repository in December 2007, Danny Baumann (number 2) in November 2010 (but  seems to be still a regular contributor) and Dennis Kasprzyk in April 2010. Of the last 500 commits 392 have been done by Sam Spilsbury. This looks like a bus factor of 1 🙁 For KWin I have 265 of the last 500 commits, though this includes the GLES port with more than 100 commits for itself.
Now I have one more graph I want to show: it includes the commits and authers per month and the accumulated number of commits.
Commits and Developers in KWin and Compiz per Month
The start Compiz had is really impressive. In a very short time they had almost as many commits as KWin had collected in the many years before. The graph is unfortunately difficult to read as it has a great variance between the months. Nevertheless it highlights the strong momentum Compiz had after the launch. In a few months Compiz had more active contributers than KWin and significantly more commits. In August 2007 Compiz reached a maximum of 415 commits! Unfortunately for the development Compiz failed to keep the momentum and both number of commits and developers seems to fade out.
[1]: All KWin commits are without scripty commits. For formatting the git log I used:

 git log –format=format:”%ad %h %an” –date=short –no-merges

My application parses the output and counts the commits per months, the developers per month and the commits per developer. All shown diagrams have a logarithmic scale in y direction.

=-=-=-=-=
Powered by Blogilo

KWin über den Verlauf der Zeit

Eine der kontroversesten Entwicklungen in KDE in den letzten Jahren war sicherlich das KDE 4.0 Release. Obwohl es klar als Entwickler Release bezeichnet wurde: "KDE 4.0.0 is our "will eat your children" release of KDE4, not the next release of KDE 3.5." (Aaron Seigo, Plasma Maintainer, am 04.01.2008 in talking bluntly), wurde es auch von Anwendern verwendet und nach dem Feedback was man damals gehört hat, waren sie damit nicht wirklich zufrieden. Vor allem Plasma in KDE 4.0 wurde (zu Recht) als unausgereift und nicht stabil bezeichnet. Selbst heute kann man auf heise oder pro-linux kaum einen Artikel zu KDE sehen ohne Hinweis auf das nicht ausgereifte Release von vor drei Jahren.

Immer wieder hieß es, dass KDE sich durch das Release selbst Schaden zugefügt hat und massiv Nutzer verloren hat. Solche Zahlen sind schwer zu erfassen, da es generell keine Statistiken zu den Anzahl Nutzern gibt. Von Entwickler Seite selbst wird das 4.0.0 Release als äußerst wichtig angesehen, alleine schon weil man in der Open Source Welt "Release Early, Release Often" folgt. Persönlich gehöre ich zu den Entwicklern (Hinweis: mein erster Patch für KWin war kurz vor 4.0.0 und wurde am 08.01.2008 eingespielt), die 4.0 unter dem Aspekt was KDE selbst erreichen wollte, als vollen Erfolg ansehen. Als sehr einfache Metrik um das zu bestätigen: in 2008 wurde fast jeden Tag ein neuer SVN Account erstellt.

Nun bleibt die Frage offen, ob KDE wirklich Nutzer verloren hat durch das Release und sich selbst geschadet hat. Wie gesagt ist es schwierig Daten dafür zu bekommen. Als Versuch möchte ich mal die Bug Statistik von KWin über die Geschichte von 1999 bis heute heranziehen. KWin ist hierbei interessant, da es in 3.5 bereits eine sehr ausgereifte und extrem stabile Anwendung war. in KDE 4 kamen die Desktop Effekte dazu, jedoch bis 4.2 standardmäßig deaktiviert und als experimentelles Feature markiert.

Die Statistik beinhaltet leider auch die "Wishlist" (aka Feature Requests), welche die Statistik leicht verfälschen, da sie meistens nicht bestätigt werden oder implementiert werden. Aktuell haben wir ungefähr 320 Feature Requests wovon 270 zu der roten Linie der Unconfirmed Bugs gehört. Eine Entwicklung über die Zeit steht mir hier leider nicht zur Verfügung.

Erklärung der Begriffe:

  • UNCONFIRMED: ein Bug, der noch nicht reproduziert werden konnte oder ein Feature Request
  • NEW: reproduzierbarer Bug
  • ASSIGNED: ein Entwickler hat sich den Bug zugewiesen – wird in KWin eigentlich nicht verwendet
  • REOPENED: ein Bug der als behoben markiert wurde und noch immer vorhanden ist.

Der Graph zeigt sehr deutlich einen massiven Einschnitt in der Entwicklung. Dieser Einschnitt liegt um die Jahreswende 2007/2008. Zwischen 2005 und 2008 haben sich die Zahlen der bestätigten Bugs kaum verändert. Der Abstand zwischen den UNCONFIRMED und NEW bleibt in diesem Zeitraum in etwa gleich. Was hat sich also in 2008 ereignet, dass sich die Zahlen so verändern? In diesen Zeitraum fällt das 4.0 Release. Seit diesem Zeitpunkt hat sich die Gesamtzahl der offenen Bugs mehr als verdoppelt und nehmen etwa einen linearen Verlauf an.

Zuerst einmal stellt sich hier die Frage: wie kann es sein, dass Bugs gemeldet werden, wenn das Release doch in einem Zustand ist, dass es nicht benutzbar ist und der neue Code in der Anwendung standardmäßig nicht aktiviert ist (und damals) auch auf der meisten Hardware nicht funktionierte?

Eine naheliegende Erklärung zu den wachsenden Kurven wäre, dass vor 4.0 die Entwickler neu aufgemachte Bugs behoben haben und ab 4.0 der Bugtracker vermüllt. Diese These kann man mit einer weiteren Statistik widerlegen:

Zuerst die Erklärung der neuen Linien:

  • FIXED: ein behobener Bug
  • DUPLICATE: ein Bug der bereits gemeldet wurde und durch Triaging manuell als Duplikat markiert wurde
  • NEEDSINFO: ein Bug bei dem Rückfragen an den Nutzer gestellt wurden und keine Antwort kam
  • UPSTREAM: Bug in einer Komponente auf die KWin aufbaut – meistens Treiber

Bei den neuen Kurven ist wichtig zu bedenken, dass es zu erwarten ist, dass die Werte immer weiter anwachsen – im Gegensatz zu den offenen Bugs wo man erwartet, dass sie konstant sind oder zurückgehen. Am interessantesten ist die Kurve der FIXED Bugs. In dem Bereich zwischen 2005 und 2008 ist die Kurve fast konstant – genauso wie die Kurven der UNCONFIRMED und NEW Bugs. In 2008 verändert sich die Kurve ebenfalls und nimmt auch ein lineares Wachstum an. In den drei Jahren seit 4.0 haben die KWin Entwickler fast so viele Bugs behoben wie in den acht Jahren zuvor.

Die Kurve zeigt deutlich, dass zum Auslauf von KDE 3.5 keine neuen Bugs mehr gemeldet wurden und seit 4.0 die Anzahl sowohl gemeldeter als auch behobener Bugs deutlich ansteigt. Dies kann man auch an der DUPLICATE Kurve sehen. Diese wächst auch während KDE 3 linear an. D.h. es wurden immer wieder die gleichen Bugs gemeldet. Jedoch verändert auch diese Kurve Ende 2008 ihre Form (Kubuntu mit 4.1 und Effekten standardmäßig aktiviert). Die Kurve hat klar die stärkste Steigung von allen hier gezeigten und scheint gänzlich unbeeinflusst von den anderen Kurven zu sein. Dass wir viele Duplikate bekommen habe ich ja schon vorher gewusst.

Nun will ich noch eine weitere Statistik zeigen: die Entwicklung von KWin. Mit Hilfe von git log und einem kleinen Programm hab ich mir die Anzahl Commits und Anzahl Committer pro Jahr generieren lassen (Achtung: Y-Achse ist logarithmisch):

Der Graph bricht zum Schluss so stark ab, da wir in 2011 noch nicht viele Monate hatten 😉 Was dieser Graph nun zeigt, ist dass in der Zeit in der keine neuen Bugs gemeldet wurden die Enticklungsgeschwindigkeit stark angezogen hatte (KDE 4 Portierung/Entwicklung der Effekte) und auf dem hohen Level sich hat halten können. Das 4.0 Release hat die KWin Entwicklung weder in die eine noch in die andere Richtung beeinflusst. Insbesondere konnte KWin das reduzierte Engagement des Maintainers (immernoch etwa 27 % aller Commits) durch neue Entwickler kompensieren.

Fazit:

  1. Die Aussage, dass es das 4.0 Release brauchte um den Code getestet zu bekommen, ist im Falle von KWin klar bestätigt. Seit 4.0 steigt die Anzahl der gemeldeten und behobenen Bugs stark an.
  2. Zumindest KWin hat durch das 4.0 Release keine Nutzer verloren. Andernfalls hätte die Kurve nicht anziehen können
  3. Die Anzahl der Bugs ist gestiegen obwohl die neue Funktionalität nicht standardmäßig aktiviert war. Dies deutet auf ein Wachstum der Benutzerzahlen hin.
  4. Das spätere standardmäßige Aktivieren der neuen Funktionen hat keine Auswirkungen auf offene und behobene Bugs sondern nur auf Duplikate und Treiber Bugs
  5. Die massive Stabilisierung in KWin hat keine Auswirkung auf die Anzahl der gemeldeten Bugs. Die Rate ist nahezu konstant. Auch dies lässt sich nur mit Anwachsen der Benutzerzahlen erklären.
  6. KWin ist als Projekt sehr gesund.

Ich bitte davon Abstand zu nehmen mich in den Kommentaren daruaf hinzuweisen, dass 4.0 doch ganz fürchterlich war. Das 4.0 Release war vor meiner Zeit als Entwickler und ich bin daher in keinster Weise für das Release verantwortlich.

=-=-=-=-=
Powered by Blogilo

Proactive Spam Fighting with Akonadi

One year ago I finished my Master Thesis and I just realized that I never really blogged about it. I had the chance to build my implementation during the Thesis upon the Akonadi framework, which is the reason why I want to share what I worked about. For all who might be more interested in my Thesis, a publisher was so kind to publish it.

The title of my Thesis was "Implementation of Proactive Spam Fighting Techniques" and the Thesis covered the implementation of two orthogonal techniques. Both techniques shared the idea to eliminate Spam before it hits the user’s inbox. Current Spam fighting techniques like for example SpamAssassin are reactive. SpamAssassin uses a rule based approach and a Bayesian filter. Rule based approaches cannot identify Spam messages reliable and have the danger to incorrectly mark ham messages if like last year the date grossly in the future becomes present. Bayesian filtering requires to collect a rather large amount of mail messages before messages can be filtered. This illustrates that reactive Spam fighting is no real solution.

Both implementations required to interact with the user’s inbox and in one case it was required to automatically send out mail on mail reception. The original idea of my tutors at the Laboratory for Dependable Distributed Systems was to develop either a plugin for Mozilla Thunderbird or Microsoft Outlook. With my background of KDE development I had to think of one framework which handles this much better: Akonadi. Thanks to Akonadi I was able to implement the solution in a client and platform independent way. Instead of just supporting Mozilla Thunderbird the solution works on all systems supporting Akonadi and no specific mail client is required. It can even be used by users who just use a web mail client.

Mail-Shake

Mail-Shake is the name of one of the two projects. The concept is difficult to grasp, therefore I just copy the description from my thesis:

The basic idea behind Mail-Shake is to block all mails from unauthenticated senders and to provide senders an easy way to authenticate themselves. The process of authentication is done in a way so that humans are able to participate, while computers – and by that spam bots – are not. After authentication the sender’s address is put on a whitelist. The whitelist is used by Mail-Shake to decide if a mail is authorized or not. By that the concept is proactive as it blocks spam before it is read by the user.

The Mail-Shake process

These initial steps of authentication are illustrated in the Figure above. A sender (User A) has to send a mail to User B’s public email address. All mails sent to the public address are discarded, but answered with a challenge mail containing a unique identifier. User A has to solve the challenge, which reveals User B’s private address. Now User A can resend the original mail with the identifier in the subject. Mail-Shake compares the identifier and puts User A’s address on a whitelist. In future User A can send mails directly to User B’s private address. The authentication step is required only once. As well there is no need to include the identifier in each single mail. Other mails sent to the private address are discarded if the sender address is not on the whitelist.

For the implementation I wrote an Akonadi Agent which monitors a set of public and private collections. Whenever a mail is received at the public address, the agent generates the unique Id, stores it in a database and sends the challenge mail to the sender’s address. The handling of the private collection is more complex. The agent has to decide whether the mail is authorized to be send to the user or not. It first checks the whitelist, if not on the whitelist it has to decide whether it’s a reply to a challenge mail and if it is whether the id is correct and update the whitelist.

The original paper considered that a mail gets just dropped if it is not on the whitelist. That is where I disagreed and implemented an automated mail notification to the sender in the case of not being on the whitelist or incorrect copying of the unique id. Furthermore I did not just delete the mails but used the functionality of Akonadi to allow the mails to be be moved to a different collection.

The concept so far works wonderful for human communication. Humans are able to solve the challenge (in case of my implementation reCAPTCHA Mailhide) and to resend the mail. All spam messages are automatically deleted without disturbing the user. Wait, there is more: communication with machines. I receive far more mails send from machines than from humans: forum, mailing lists, web shops, review board and many, many more. Nobody in the web shop will answer the challenge mail; Mail-Shake will break.

The solution to this problem is to allow users to modify the whitelist by themselves. Of course this is tricky and could make the usage too difficult. Because of that I played again with KDE technology. My Mail-Shake agent uses a StatusNotifier which is hidden most of the time. Whenever a mail is received, which would be discarded, the StatusNotifier changes to the active state and becomes visible. It allows to just add the addresses of the mails in the current to be discarded queue to the whitelist.

Notification on receipt of a not-whitelisted mail

In order to make it even more usable to the user, Mail-Shake can show normal Plasma notifications with an Action to add the address to the whitelist. The idea behind it is, that whenever you expect a mail from an automated system, you turn on the notification through the StatusNotifier and wait for the mail to arrive. (Btw a nice example for a notification which is completely useless when not having actions on it).

After the implementation I had tested Mail-Shake on my own mail system for two months. It illustrated that the implementation and the concept works. Spam mails were successfully removed and humans authenticated themselves. Nevertheless some problems occurred. The first is that the assumption for interaction with web-shops does not hold. Just because you buy at shop foo and receive a mail notification from @foo.com does not mean that further mails (e.g. the bill) will not be sent from @bar.com. This makes interacting with automated systems hardly possible without manually checking the filtered mails, in which case nothing is won in comparison to the existing reactive systems.

The second problem occurred when sending out the response challenge mails. As most mails are sent in reply to Spam mails we can expect that there is no mail system to accept the mails. Our mail transfer agent (MTA) will therefore send us a delivery status notification (DSN). Mail-Shake can handle those and will not send another challenge mail in response to the DSN and remove it, so that the user does not see it.

Mail-Shake supports delivery status notifications as described in RFC 3464, which does not specify a reference to the original mail, which makes it difficult to map the DSN to the sent mail and to decide if the DSN needs to be delivered to the user or not. Luckily during my evaluation each DSN included either the complete original message or at least the header section. So this seems to not be a problem in praxis. But the real problem is that there are MTAs not sending DSNs as described in RFC 3464. They use a custom format. Some of the non-standard conform MTAs are Exim, QMail (at least in older versions) and the MTA used by Google. The mail format is custom which makes it impossible to parse the mail automatically. In case of Exim it seems to be possible to customize the notifications. Mail-Shake is not able to recognize the DSN and starts to go into an infinite mail sending loop.

My personal resume is that Mail-Shake is not in a state to be used in a productive environment. The problem is not in the concept or in the implementation, but in MTAs not supporting RFC 3464. It is unlikely that MTAs not supporting the RFC for years will start to support it and even if, there are enough legacy installations to destroy any hope in having this concept in a workable state. It was rather frustrating to notice that the concept breaks because of the external state of the system.

Mail-Shake Client Integration

In my implementation I relied on reCAPTCHA mailhide API to protect the private mail address. The recipient of a challenge mail has to visit the reCAPTCHA website to reveal the address. This means we encourage users to click on links in mails. That is of course the opposite of what we teach users: never click a link in a mail you did not expect! To solve this issue we need an integration into the mail clients. The user should not notice that he received a challenge mail, the client has to guide the user in solving the challenge.

As at the time of writing my Master Thesis KMail was in heavy porting to Akonadi, I decided to implement the proof-of-concept integration in Mailody (which already supported Akonadi and WebKit at that time). The integration was straight forward and easy to implement. Mailody was a very nice application from that point of view. I extracted the Mail-Shake headers and used them to recognize that the mail is a challenge and integrated a link to solve the challenge. If you clicked the link a dialog opened, connected to reCAPTCHA, extracted the CAPTCHA and offered an input field to enter the text. The solved CAPTCHA was submitted to the web server and the result parsed, providing a localized info whether the challenge was solved or the CAPTCHA incorrect.

Solving a Mail-Shake Challenge in Mailody

The next step would have been to automatically resend the mail with the solved challenge, but I did not implement that. The idea was to just show that a client implementation is possible to solve the mail clicking problem.

Spam Templates

The second part of my Thesis is much smaller, but in my opinion the more interesting one. The laboratory had setup a system to execute Spam Bots in a controlled environment and to intercept the Spam mails it was sending out. The communication to their Master&Control servers was kept alive. The bot and the Master&Control server is by that thinking that they are working correctly. The Spam Bot is regularly fetching new templates from the control server and generates mails based on that template. These mails are intercepted and the template is reverse engineered. The exact process is described in the paper Towards Proactive Spam Filtering.

From the previous existing setup I had a set of regular expression based templates and a set of spam messages which were used to construct the templates. As the system was not up and running I did not have recent templates. Because of that I could only evaluate against the existing messages.

My implementation was based on another Akonadi Agent which reacted on the reception of new mails. Any new mail is checked against the templates and a score is generated from the matched lines of the template. The Agent would receive new templates through an RSS system (thanks to libsyndication). In the ideal situation new templates would be created at the same time as a new Spam campaign is started and distributed to all clients. So at the time the client receives the first Spam message of the new campaign the template would already exist. As the template is only generated from Spam messages it would be impossible to match Ham messages. This has an incredible advantage compared to the existing reactive filtering solutions: you don’t have to collect Spam messages before you are able to filter them.

Unfortunately this is just a research project, so there is no server broadcasting the Spam templates. Which makes my implemented system rather useless and did not allow me to verify if the implementation works correctly – I only know that it does not filter Ham messages.

Where to get the Code?

Of course the complete code has been written as free software, so where can you get it? I used git as version control system, so it has not been available in the KDE repository and after the switch to git I have not yet considered to upload it. The code and packages are available through the openSUSE Build Service and a live CD is available through SUSE Studio (awesome tool). It is possible that some parts are not working anymore as it seems that reCAPTCHA changed their web site (which fails my parser).

What else?

Of course I could only present a small part of my Thesis in one blog post. There are lots more interesting things to find in my Thesis. For example I was able to break the scr.im email address protection system using OpenGL Shading Language and OCR with a reliability of more than 99 %. I also did an investigation on existing CAPTCHA systems which showed me that most systems are in fact completely broken and just an annoyance to the user who has to use them.

Last but not least I want to thanks the KDE PIM team for providing such an awesome framework. Without Akonadi this Thesis would not have been possible. It is just incredible what can be done with Akonadi and I’m looking forward to KMail 2 ever since I started to work with Akonadi. I always feel sorry if I have to read the bad user responses to Akonadi. Don’t get frustrated, keep up the good work!

=-=-=-=-=
Powered by Blogilo

Upstream und Downstream

Das Verhältnis zwischen den Open Source Projekten und ihren Distributionen ist ein sehr besonderes. Die Distributionen brauchen die Open Source Projekte, denn ohne sie haben sie nichts zu paketieren und die Open Source Projekte brauchen die Distributionen um den Quellcode zu verteilen – wie der Name schon sagt. Dieses Verfahren ist so eingespielt, dass richtig große Open Source Gemeinschaften wie z.B. KDE überhaupt keine Pakete bereitstellen, sondern nur den Quellcode zum Download anbieten. Im Falle KDE sind die Distributionen so eingebunden, dass sie Zugriff zu den Quellcode Paketen eines neuen Releases bekommen bedeutend vor dem Rest der Welt (mich eingeschlossen) um Pakete rechtzeitig zum Release bereitstellen zu können.

Die Beziehung zwischen den Open Source Projekten und ihren Distributionen ist so speziell, dass sie sogar eigene Namen haben. Die Open Source Projekte bezeichnen sich als "Upstream" und die Distributionen als "Downstream". Das Projekt ist also flußaufwärts und der Quellcode strömt zur Distribution den Fluß hinunter. Patches, die von den Distributionen zurück an die Projekte fließen, wir mit "to upstream" bezeichnet.

Mir persönlich ist das Verhältnis zu den "Downstreams" sehr wichtig. Ich kenne die Paketbetreuer unserer wichtigsten Distributionen und helfe regelmäßig falls es darum geht KWin besser in die Distribution zu integrieren und Bugreports von "Downstream" werden von mir priorisiert behandelt. Genauso wie einem als Upstream die Downstreams wichtig sind, erwartet Upstream auch, dass Downstream sich um Upstream kümmert. Einer der Punkte, die einem da sehr wichtig sind, ist die Anzahl der Patches, die Upstream integriert. Von einer guten Downstream erwartet man, dass Patches geupstreamed werden. Wenn nicht stellt sich immer die Frage: warum? Hier erfolgt dann sehr oft der Vorwurf, dass Distributionen ihre Upstreams kaputt patchen.

Natürlich ist es einer Downstream erlaubt den Quellcode zu verändern und zu verbreiten. Dies ist durch die dritte Freiheit der Free Software Definition klargestellt:

The freedom to distribute copies of your modified versions to others (freedom 3). By doing this you can give the whole community a chance to benefit from your changes. Access to the source code is a precondition for this.

Wer die Open Source Welt in den letzten Tagen verfolgt hat, dem dürfte klar sein auf was ich mit meiner Einleitung hinaus wollte: dem Fall Banshee und Canonical oder verallgemeinert Upstream und Downstream. Zu dem Fall an sich will ich gar nichts mehr sagen. Das haben andere schon viel besser erledigt.

Für mich als Maintainer einer Upstream Software ist das Verhalten der Downstream aber ein sehr interessanter Fall zum Studieren und ich bin mir sicher, dass sehr viele Upstream Entwickler den Fall sehr genau betrachten und auch andere Downstreams schauen sehr genau darauf. Faktisch ist alles klar: Canonical hat durch die dritte Freiheit das Recht den Code zu ändern in welcher Art sie wollen. Ob sie es sollen ist eine andere Sache. Ich möchte hierzu den relevanten Quellcode in Banshee zitieren:

// We ask that no one change these affiliate codes. ALL (100%) revenue
// generated by these affiliate IDs is sent directly to the GNOME
// Foundation. The GNOME Foundation controls/owns these affiliate IDs.
// Please help support Free Software through the GNOME Foundation!

Ob es nun moralisch vertretbar ist, dass ein Unternehmen diesen Code ändert um selbst Profit daraus zu ziehen, muss im Endeffekt jeder selbst entscheiden. Für mich persönlich ist es befremdlich zu sehen, dass eine Downstream gegen den Wunsch des Upstream Projekts den Code ändert – auch wenn es ihr gutes Recht ist. Für mich als einer der vielen altruistisch arbeitenden Open Source Entwickler ist es noch befremdlicher zu sehen, dass das Geld eigentlich an eine gemeinnützige Organisation gehen soll. Der finanzielle Beitrag ist dabei übrigens eher marginal. Banshee bringt aktuell etwa 10.000 $ pro Jahr ein. Wir können nicht wissen wie viel Ubuntu beisteuern würde, aber meine Vermutung ist, dass es nicht viel mehr wäre. Der aktuelle Beitrag reicht vllt. um einen Entwickler einen Monat zu finanzieren oder aber um einen Sprint auszurichten. Ich überlasse die Bewertung dessen dem Leser.

Der "Skandal" ist für mich aber nicht, dass Canonical den Code geändert hat, sondern das Vorgehen. Zuerst eine Diskussion mit den Entwicklern zu starten und zwei Optionen zur Wahl zu stellen und danach einstimmig zur abgelehnten Option umzuschwanken, ist meiner Meinung nach mehr als nur schlechter Stil. Persönlich denke ich auch, dass hier das Hauptproblem ein Kommunikationsproblem von Seiten Canonicals war. Und ich meine nicht, dass überhaupt Optionen angeboten werden, wie Mark Shuttleworth es nun verteidigt hat, sondern dass man das Upstream Projekt falsch angesprochen hat.

Als Distribution muss man seine Upstreams kennen. Bei den zwei Optionen hätte es Canonical bereits im Vorfeld klar sein müssen, wie sich die Entwickler entscheiden. Canonical hätte also wissen müssen dass die gewünschte Option nicht gewählt wird und das anschließende Erzwingen stößt jeden Entwickler nur vor den Kopf. Natürlich wäre nicht zu kommunizieren genauso falsch gewesen, genauso wie zu kommunizieren, dass man das jetzt einfach so macht. Ich halte es aber für durchaus denkbar, dass man die Community so führen kann, dass sie selbst den Vorschlag zum Aufteilen der Einnahmen erbringt – wenn es unbedingt sein muss. Eine klare Aufgabe für einen Communitymanager – den Ubuntu ja hat.

Canonical hat sich mit der gesamten Geschichte einen riesigen Bärendienst erwiesen. Canonical und auch Ubuntu stehen schon länger unter genauer Beobachtung von Upstream Projekten. Immer wieder mehren sich die Vorwürfe, dass Canonical nur nimmt und nicht gibt (vgl. Diskussion zu Quellcode Beiträge zu Kernel und GNOME). Immer wieder wird auch die Art und Weise wie Canonical mit der Community umgeht kritisiert (vgl. Copyright Assignment, Ayatana, Unity). Persönlich habe ich auch schon sehr lange und intensiv mir Gedanken zu Canonical gemacht und auch immer wieder gefragt, ob Canonical überhaupt Open Source versteht. Für mich passt (leider) das aktuelle Geschehen in mein Bild von Canonical – denken viele Entwickler so, dann ist dies eine sehr gefährliche Entwicklung für Canonical. Verscherzt es sich Canonical zu stark mit allen Upstream Projekten, dann stehen sie sehr alleine dar und ob sie dafür Manpower und Kompetenz auf allen Feldern haben, ist mehr als fraglich.

Hinweis: in diesem Text fehlen viele Fußnoten. Ich habe mich aus aktuellem Anlass dazu entschlossen meine Quellen nicht anzugeben und überlasse dem Leser das Suchen der Quellen selbst. Bei der großen Anzahl von Quellen zu diesem Blogpost habe ich leider den Überblick verloren, dennoch ist es kein Plagiat, auch wenn Gedanken aus anderen Quellen übernommen worden. Dies sieht man schon daran dass alle nicht angegebenen Quellen auf Englisch sind und dieser Post auf Deutsch.

=-=-=-=-=
Powered by Blogilo