In this blog post I want to outline some thoughts on how the gaming experience can be improved on Linux.
Situation on X11
On X11 the big problem for gaming is the Compositor. Games need access to the GPU and need to pretty much exclusively use the GPU. Compare to a true console like a PlayStation: while the game is running you can be sure it’s the exclusive user of the GPU. The way how compositing works on X11 this cannot be provided. There is the compositor which needs to render the scene. The setup looks simplified like:
- Game renders through OpenGL/GLX
- X-Server notifies Compositor through XDamage extension
- Compositor schedules a repaint for changed area
- Compositor uses the XComposite extension to get a pixmap for the Game window
- Compositor binds pixmap to an OpenGL texture
- Compositor renders the texture using OpenGL/GLX to the composite overlay window
- X server presents the rendered image from the Compositor through kernel mode settings
In this setup we have a few things which are not optimal for a fullscreen game: the roundtrips through Xserver-compositor and the possible delay added by the compositor. In such a setup the compositor will always
slow down the game as it performs vsync, etc.
Workarounds on X11
There is an existing solution to improve this which is known as “unredirection of full-screen windows”. The idea is that if there is a full screen window the compositor won’t run for it and the “normal” non-composited functionality of the X server is used.
In my opinion this is the wrong solution to the problem, because the Compositor is still running, it still gets damage events (from possibly other windows) and might start to composite the scene at any time again (e.g. a notification pops up as a override-redirect window).
In KWin/Plasma we have a better solution: blocking compositing. We can do that as we do not require compositing, other environments which require compositing can only use unredirection as best measurement. In fact we even have high-level API for games to tell us that they want to block compositing. It’s just one X atom, please use it.
This also explains why in KWin/Plasma the option to fullscreen unredirection is not enabled by default. It has drawbacks for a non-gaming usage, while we offer a better solution for games. It also explains why I don’t care at all about gaming benchmarks through the PTS as that in my opinion tests an invalid setup for us. If we cared about it, it would be easy to just ensure that the games used by PTS disable compositing.
Situation on Wayland
On Wayland the setup is better as we don’t have X11 in between. So a setup looks like the following:
- Game renders through OpenGL/EGL
- Compositor gets notified through wl_surface damage
- Compositor directly presents the wl_buffer through KMS as it knows there is nothing else to see
So the situation is much improved in an ideal case. I need to point out that KWin doesn’t support this ideal case yet and still renders through OpenGL, but that’s where we are going to.
But I think there are still problems. Our Compositor still gets damage events from other windows, it gets woken up, etc. Running a game in a desktop environment means that there are other session processes which the game needs to share resources with. We want to have a PlayStation like setup: game everything, everyone else nothing. I don’t want KWin to take away important CPU/GPU time from the game.
Kernel Mode Settings in games
So what can we do? I thought about this and propose that we change gaming completely on Linux: remove the windowing system! Games should talk to kernel mode settings directly, games should interact with libinput directly. Let’s remove everything in between, we don’t need it, it only can worsen the gaming experience.
When a fullscreen game starts, it can create a “sub-session” on a new virtual terminal and become the logind- session controller for that session. This would allow the game to open the device files for rendering and for input just like a Wayland compositor does. Rendering could be done through EGL on top of DRM/GBM just like a Wayland compositor. The game would have full control over rendering, there is no desktop environment to slow it down any more. And it would have control over mode setting. Need a different resolution? No problem, just set it. On a desktop environment that’s always problematic (terrible on X11, better on Wayland). For games in windowed mode nothing should be changed, those should stay on the desktop environment.
Of course this would remove all interaction with the desktop environment. This is something which needs to be considered, like how to get Mumble to work in such a setup? Maybe the game could launch its own Wayland server?
That breaks Alt+Tab! Well not really. For one on X11 at least games often grab the keyboard, so Alt+Tab won’t work anyway. And of course one can still switch with ctrl+alt+f1 to the running session. Games should also have a common way to achieve this in my opinion.
There are certainly a few things which still need to be solved to get there, like how to start such a sub-session, how to return to the running session without needing to unlock the screen. The experience should be as good as on dedicated gaming console. So it’s still some work, but I think this is work well invested and better than trying to somehow make games work in Desktop Environments as that just cannot work as good as a dedicated gaming console.
“That breaks Alt+Tab! Well not really. For one on X11 at least games often grab the keyboard, so Alt+Tab won’t work anyway.”
Completely lame… You’re breaking something that works just fine in Windows (and even in Playstation, as you’re using it in your example).
“The experience should be as good as on dedicated gaming console.”
and
“And of course one can still switch with ctrl+alt+f1 to the running session.”
can’t really go in the same text.
Just my 2 cents
Except it doesn’t work in every second game, it crashes the game, slows down whole system or cripples it. People are actually afraid of alt-tabing.
Yeah, alt+tab is a mess on Windows. If we told Windows gamers that on Linux alt+tabbing works flawlessly every time (with Wayland it will, if I understand correctly), they would be impressed and more likely to consider the switch.
Alt-Tab isn’t always a mess. On DX10 or DX11 games, I can seamlessly switch between apps with no glitching. The problem is that the GPU grabbing model has evolved over time as hardware capabilities have improved. Remember the 3DFX Voodoo that could only accelerate fullscreen? Modern 3D accelerators are much more flexible. The DX3D API has evolved along with it, and the modern API handles switching very well.
Older APIs, however, said it was the app’s job to handle context upon restore, hence the ugly resolution switches and everything as it had to basically reboot the GPU to return to its state. When the game even supported it! With game development schedules being what they are (and always were), handling that usually wasn’t a top priority.
And we’re still dealing with that today thanks to Microsoft’s efforts in supporting backwards compatibility (and yes, it’s not always perfect. You only notice it when it doesn’t work), which means you can still run DX9 or older games on modern systems, even when that game is trying to reset your GPU!
Games like that need to disappear… i hate games which do weird crap when using alt-tab.
I play all my games on borderless windowed mode if the game is making problems it is asking for the uninstall.
> Completely lame… You’re breaking something that works just fine in Windows AND linux (and even in Playstation, as you’re using it in your example).
^^ fixed
I can’t remember the last game I had problems with it… and even if it’s still no excuse for breaking it…
I wouldn’t like that. I still want to e.g. get unobtrusive notifications or to be able to change the system volume using the key bindings i defined in the compositor.
This is an interesting idea. It might work for the use case of having a dedicated gaming pc (ie, a steam machine). I do not think it would work well for non-dedicated machines, though. I 82 games on steam right now (I know, I can’t help myself). Of those 82 games, I have found 1 — just 1 — game that will not allow me to alt-tab out of it. Most of the time, I am able to quickly pause and check my email notification or whatever alerted me. In this example, I do not think having a separate ctrl+alt+f1 setup would work.
Alexander said that the idea is “completely lame”. I do not agree with this (or his choice of being an internet flamer), but he does raise a valid point at the end. I have a ps4. I can switch out of the game to the main console dash screen with the press of 1 button. So — same as on the desktop — if someone instant messages me, etc, I can quickly check it out.
Well that just means it needs to be integrated better. I don’t see a reason why the “dash console setup” shouldn’t work.
Or the damage reports idle so long until you switch from the game to desktop!
When I play games (Windows), I very often switch to different applications, or, since I have a multi monitor setup, I keep them running on the second screen. A browser to look a game related stuff or just to spend idle time while waiting for the next multiplayer game to start. Teamspeak/mumble is a must. Some people record and stream their gaming. Sometimes I even open a second casual game, to spend the idle time waiting.
^
The desktop session cannot be eliminated, at least by default.
It’s interesting to be able to remove most intermediaries between the game and the hardware, but it should be considered an advanced option, easy available, but for those who understand the implications (no quick window switching, recording, teamspeak, idle-browsing, etc).
If I understand it correctly Vulkan will have a Windowing System Extensions for all the different Windowing Systems (X11, Wayland, Surface Flinger etc.). If it is like Mantle which was used as basis for Vulkan it should look like on page 13 here:
https://www.amd.com/Documents/Mantle-Programming-Guide-and-API-Reference.pdf
Maybe it would be possible to add a DRM/KMS backend extension for Vulkan so it can run directly on a TTY like other applications that have a DRM backend (MPV Player).
When I’m on my desktop nvidia-settings tells me that 430 MBs of VRAM are already in use and my card only has 1.5 GB. On my 2560×1440 monitor the VRAM is alway close to being completely full. I wonder if this VRAM would be freed up when the gaming session starts. Booting into systemds multi-user.target which only takes up 50 MBs of RAM and using a ncurses interface for Steam to launch the games on DRM/KMS would be pretty cool.
good idea with Vulkan and DRM/KMS. And yes the “waste” produced by DEs is part of my thinking for this blog post. A wallpaper is huge and it’s normally not optimized for memory efficient storage in GPUs.
“A wallpaper is huge and it’s normally not optimized for memory efficient storage in GPUs.”
Then wouldn’t it make sense to add a button or checkbox to Desktop Settings which applies texture compression? It’s not as if there’s no precedent for KDE features appearing to take forever.
(eg. I don’t know what caches it’s regenerating, but changing file associations is instantaneous via UIs like PCManFM’s file properties dialog while KDE still has that slow progress dialog reminiscent of the Windows 9x “Building a driver information database” dialogue.)
This is a stupid idea. I absolutely hate it when games block alt+tabbing, and I really want to be able to interact with my desktop, and the desktop to interact with my game to possibly show notifications, for example. On X11 so far, 2 out of 3 games prevent me from doing so and it always gets me close to raging. Especially when the game is locked up and you can’t even ctrl+alt+esc kill it either. Now finally, wayland would force the games to behave and you want to take that away?
Btw, too many Windows games prevent you from alt+tabbing too, often to outrageous results, such as ctrl+alt+entf getting you your task manager, but that one not rendering at all or you’re only able to control it with your keyboard. As I understand it, wayland ought to prevent that too, no? While still retaining the benefits of the game still running as a client to your desktop.
I understand from your and others comments, that you are concerned about no longer interact with the session. That is not what I want to remove. An interaction can still be possible, even in very convenient ways like the “home” button on the PS-controller. It might be that you need to press a certain key to go back to the desktop, but that will also be the case in a Wayland setup. For running it on Wayland (and even windowed mode) it will result in a host key needs to be pressed to be able to use Alt+tab again.
First press a host key? How to do “push to talk” then, when using TeamSpeak?
integrate TeamSpeak into games!
We need to face that games need to grab input, there is no way around it. And on Wayland that will probably mean that a host key needs to be pressed to go back to ungrabbed mode. If we want to have decent integration it needs to be done in the games.
Integrating TeamSpeak is not a solution because then it would not work with Mumble or whatever background software you may want to use (a music player for example).
And that would not even be possible with all those closed source games.
I understand the point of grabbing the GPU, but not why input should be exclusive too.
That’s a problem of proper interface, though
To be perfectly honest, that response lost you a lot of credibility in my eyes.
It’s bad enough that my older flight sticks are useless in many modern games because they assume the OS will have calibration support at least on par with Windows 95 but the move from joydev to evdev killed that.
I have yet to see a convincing argument for not providing some kind of analogue to XGrabKey and I don’t want more analogues to “every DOS game bundles its own set of sound drivers and clone hardware pretends to be a soundblaster” creeping into my desktop.
How many voice-chat applications do you want to integrate in every game?
Sorry, but that doesn’t sound like a solution at all.
We should face the fact, that the requirements for PC-Games are different than for Console-Games.
All, define a standard
Let’s face it: games are a mess and we can’t ask their devs to adhere to our rules and standards because most of them won’t listen. Either we enforce the rules or assume the games are not going to abide.
Experience shows games can’t even be trusted to properly use OpenGL/DirectX, hence all kinds of crazy workarounds in proprietary drivers (“new driver version optimized for game X”).
Applications can’t be thrusted to do key grabbing right.
Get a key grabbing mediation API where certain keys can be defined to be ungrabbable by other applications.
Have keys that can only be used by certain applications.
Only allow random applications to grab a welldefined subset of available keys.
Keygrabbing management API is needed if applications can’t share keys and resolve their grabbings by themselves.
Or mumble, which is just as good and is open source.
You’re completely losing it. Go take a nap, then get back and re-read what you’re saying.
Inegrating TeamSpeak into games is a bad idea, bordering on very bad idea.
A1. TeamSpeak on Linux is broken by design – it doesn’t support more than 3 mouse buttons, removing the most obvious push-to-speak button while gaming using a multibutton mouse, with the developers response being “It’s not broken, it’s designed that way and we’re not changing it.”. There were many angry threads on their support forums over the years.
A2. The whole choice thing – some people prefer Ventrilo, some prefer Mumble, some prefer TeamSpeak, etc. The important part is that people often use those out of game (talk to friends, see who’s online, ask who wants to join them in a game, etc.) – I for example always disable the ingame voice chat whenever the game provides one as it’s useless to me.
Now back to the new session for a fullscreen game idea.
B1. I often alt-tab and visit links posted by my friends on TeamSpeak while I wait for respawn. I use Gnome 3, and unlike on Windows alt-tabbing from games always works as expected. (for example on Windows alt-tabbing from Payday 2 and then back to it would sometimes leave the cursor visible)
B2. How would this cooperate with Steam? How would this cooperate with Steam Controller? (the whole configuration thing)
I’m all for an optional “performance mode” that has its own restrictions and limitations as long as the default is a proper windowed mode.
Now my question is: is a proper windowed mode possible with Wayland? And by “proper windowed mode” I mean:
– I am able to use alt+tab, change workspaces, enter Gnome Shell’s Overview, etc. (Now I can play TF2 on X.org, press the Super key and the game seamlessly and smoothly transitions to GS’ Overview. From other comments I read some games don’t support it.)
– I am able to change sound volume, screen brightness, etc. via special keys
– I am able to game on one monitor and do something else on another
– I am able to get notifications, play music or use Mumble in the background, etc.
– Normally game’s window should take an entire screen and have no decoration visible
All of this without assuming that the game plays nice and does something special to support it. (Other than supporting Wayland, etc.) Because if game devs can do something wrong, they will.
I must say, I like your thought process in this. For people who care to read, you’ve pretty much answered any real concerns about the proposed setup. I like it!
There is already a key defined to interact with your desktop its called Alt-TAB, no need to invent something that already exists.
If a PS4 with a FreeBSD based OS can do it, then I’m sure that Linux could do it.
Interesting idea, but I doubt I’d use it. I run a triple-head desktop and prefer to run my games either windowed or fullscreened at the WM level so I can have supplementary information on my other two monitors.
I also have plans to write LD_PRELOAD hooks to push games more in the direction of windowed mode. (And not just because resolution-changing is broken under X11 and games like Dungeons of Dredmor react badly to having the resolution locked at 3840×1024 by nVidia MetaModes option.)
Oh, I forgot to mention. I also sometimes choose to run the Windows version of a game in Wine rather than the native Linux version simply because Wine helps to provide features I’ve been hoping to get from Wayland like forcing Alt+Tab to work and denying the application the ability to go fullscreen (Wine’s virtual desktop… hopefully KWin’s Window Rules).
yeah, I think your usecase doesn’t fit, just like any other windowed game usage.
I see the ideal situation of the “Gaming Mode”/Kernel Mode on Linux, but to me I feel like Linux falls back to Windows DOS days where to play a game it had to switch to DOS (not Desktop Mode) which involves with restarting the computer to load into DOS days. Not saying it’s a bad idea, hell it can even give a better performance to let the game decide what it wants (essentially). The problem I see here is the use case to switch from Full Screen to window mode, this use case tells me that it’s the developers of this certain application is the one at fault if the switch does not occur as it should.
I suggest to have a routine to be able to switch from “Gaming Mode” to Desktop Mode safely FIRST (if it’s possible), then this would solve a lot of the issues ahead and make it easier to for the less knowledgeable Linux developers. And think about those with multiple monitors, that might even give them an edge for this exact use case scenario.
Like this Blog Post suggest, “that we change gaming completely on Linux”.
I don’t see why that should be a problem. We can get that as a smooth experience, if we want.
To clarify what I mean with multiple monitors:
If it’s possible to have one monitor with Desktop Mode and another with Kernel Mode/”Gaming Mode” (which gives those who have multiple monitors an edge).
No, that is not possible, at least currently. Only one process can be the so-called ‘drm master’ and control the screens at a time.
well it could be delegated (in some way).
YES, BUT OPTIONAL.
For a console use to let the default as this solution offers, would be great for benchmarks, and even better to be able to switch resolution if the game is slow (almost always will be) or make games able to look for stable FPSs (30 45 or 60) switching to the resolution (720p or even lower) that allows smooth gaming.
But let configure and put as default the actual mode for the desktop gamer that wants to keep their desktop services on.
And perhaps for “console mode” add “console” versions for Mumble and others
Sounds like a great idea, Martin. Surely all the concerns raised are addressed by having the option of a windowed mode, as you mentioned. People can play that way if they want to multitask, or they can use your stripped down mode for maximum performance.
Hello, in my opinion there are better solution. Take a look at how applications are handled on mobile devices, where resources are limited. My knowledge in that area is not deep, but as far as know, application that are not shown are freezed on disk. While the reality is more complicated, this way is probably a better one, and is in line with the KWin custom protocol for blocking composition.
1. Game ask for fullscreen,
2. composited acknowledge and give the fullscreen and an exclusive access (some-how),
3. compositor freeze all non-deamon applications
On special keys combination compositor cancel the exclusive mode.
There was project introducing Window Manager into Framebuffer.
In my opinion, we should also introduce special escape sequence to run a new virtual terminal in current session for program like screen and user-space console implementation.
Interesting post but I have the opposite problem when gaming on Linux instead of Windows. On Windows, I can minimize every game easily and go do something else while the game is minimized. On Linux, I half expect all kinds of bad things to happen when I minimize a game.
To be honest, I’m not sure I trust even half the games I play not to screw up and make it impossible to switch back to my normal session. I will accept a small performance penalty with Wayland if it means that games don’t ruin my session.
> For one on X11 at least games often grab the keyboard, so Alt+Tab won’t work anyway.
This has improved a lot lately (SDL2 fixed this for example). And personally I wouldn’t like to lose Alt+Tab switching. However I agree, that having games bypassing the DE completely would be good. Is there a way to do that within Wayland itself without requiring to run a separate session?
I like this idea but agree that there should probably be a middle ground.
This might sound crazy – but what about a small freedesktop.org library (not to be X of course!) which each different compositor such as kwin can use as its base / root window / gaming context? That component would be the one which listens to libinput on behalf of the compositor and captures some events (such as Alt+Tab) early and passes them as calls to the compositor. The compositor (such as kwin) could then pass buffers with the content of the alt-tab window (or whatever else) for the root window library to overlay onto the game.
Once the game has been tabbed-out-of, the compositor gains full control of that lower component.
I really thought you made this post to make fun of phoronix, with a lot of irony. Then I read to the end and now I believe that you’re serious about this.
Well, why not. What would be the benefits in terms of performance ? (%FPS) I don’t expect something huge.
What about craches ? (blue screen and reboot ? 😉 )
DE fallback? When the game crashes, it falls back to the DE.
Why don’t you use the 2d compositor for fullscreen windows?
Has performance advantages over compositing but still you can interact with the desktop.
I think direct KMS and input could be implemented without changing semantics.
Multilayer with zero overhead – but maybe it’s ridiculous because you win only a dozen microseconds per frame.
A privileged wayland client could request from the compositor to handover all device-handles and then instantiate a wayland-server library in the same address-space as the client-library so that context-switch-free API-calls are possible. If the request is not granted it just continues without. Cooperatively the client gives the device-handles back and leaves them in a sane state when requested by the compositor. For instance when another wayland-client wants to display. Of course the device-handles could then wander recursivrly through a larger wayland compositor tree this way. All relying on cooperativity, which sounds insane, but otherwise it is transparent.
This is definitely an interesting look at some of the problems of gaming on a PC. I expect that as a developer with such deep experience with window management, you have a very special insight into these matters. That said, from my perspective this is a step in completely the wrong direction. I find these days I play most of my games in window mode – not because I want the space, but because asking a game to manage the monitors is just too much for them. They can’t handle multi-monitor. And things like pop ups are always painful. And the flickering when they switch resolutions. And when X fails to switch the resolution back when the game dies unexpectedly. And when X dies because the game left the graphics in an inconsistent state when the game died. It all goes away when you confine the game to a simpler abstraction – ‘you get a window – don’t touch anything else’. Every time I see the flicker of a changing resolution I cringe – just scale the stupid output, don’t touch my resolution X can’t handle it. I’m desperately longing for Wayland in the hopes it solves some of these issues. I don’t trust X. I don’t trust games. Games need a structured environment where they can’t mess things up, and they need services provided by the environment because they can’t be trusted to get them right themselves. I don’t care about the FPS, I don’t care about the battery – if the integration isn’t right, if things are broken, if I can’t control the thing, then what good is performance?
I may have forgotten the detail, but isn’t it the compositor job with wayland to tell the application window when to send a new frame ? If that so, then the overhead of the compositor is pretty limited and you still keep the benefit of having a normal session (like you could go back to composite for some frame to display some notification, allow all king of shortcut, rewrite key send to the game and so on).
I proposed something very similar a few years back (https://mail.gnome.org/archives/wm-spec-list/2012-October/msg00054.html) It’s great to see a guru like you have a similar opinion on it and I really hope something like that gets adopted.
I’m sure the stuff like Alt-tab could be ironed out – eg. by letting the game session communicate with the desktop session. Something like if Alt-Tab is pressed, the game session temporarily passes the control to the desktop session and let the desktop session do the hard work. That would mean the game was automatically hidden, but I don’t see that as a really big issue, as currently half of the games does exactly that and in the other half Alt-Tab doesn’t work anyway.
And Martin, I do hope that you are not put off by some comments. They are just expressing their ideas and everybody’s is respected like yours. I hope you will stay motivated for whatever reason. I am getting old and the problems on X11 are not fixed yet regarding games. I have been a gamer since 2004 on linux and I would really like to see working solutions before I die!
Well, technically speaking, I think your right, Martin. Opening a new session might yield best performance. The actual issue is one of coordination, though: due to industry constraints, you cannot trust game devs to produce code that “works”, i.e. leaves control to the user. I think everyone has experienced these situation where games just break and leave everything broken. The same issue affects all those supporting tools (TeamSpeak etc) — they will not face a stable, reliable interface of games to talk to.
Maybe we should design a minimal compositor for game sessions providing such reliable interfaces for users and software to, “sandboxing” (in a way) these games.
Something similar might work for example with Steam. The Steam client as a Wayland compositor runs a game in full screen without compositing using planes and if possible no frame callbacks for the other surfaces.
But games should not deal with low level stuff, that would be kinda like the good old DOS games and sounds like a lot of work to be replaced on each and every game.
Martin, this is a excellent idea. Some thoughts to address the naysayers concerns:
Teamspeak: processes can be backgrounded in Linux and I see no reason why TS couldn’t run as such and respond to a keypress. Or run TS as a system daemon. Or run a daemon that uses a common TS-like API so that all games can use it.
Notifications: Steam and consoles have addressed this issue; present notifications as a popup from the bottom or side of the screen via an overlay. It hasn’t hindered performance for them, so there’s no reason why it should in this case.
Multitasking: One can switch to and from their DE by pressing Alt + Fx. When this happens, the game process can be suspended until the user switches back. Is Alt+Tab so important to people that replacing one key in that combination becomes an obstruction?
I don’t know. Some people still complain even when a good idea is offered to them on a plate.
How do you intend to make that work when the TeamSpeak devs are already stubbornly insisting that refusing to support push-to-talk on mouse buttons other than left/middle/right is as intended and it won’t be fixed?
As I understand it, for security reasons, Wayland APIs don’t allow apps like TeamSpeak to just grab keys/buttons and getting buy-in on a common TS-like API will be a hard sell. (Given what I know about effort-return trade-offs for niche markets and human personalities, they’ll be more likely to just slap a big notice saying “KWin ‘Gaming Mode’ not a supported configuration” on their website and call it a day. Martin doesn’t have the pull to implement an incentive like the Microsoft “Designed for Windows” certification program… especially when KDE is used by neither “Ubuntu” nor “SteamOS” and that’s what most things “support” these days.)
The whole point of what Martin is proposing is to give the game the most direct, unadulterated access to the GPU feasible… bypassing as many of the intervening layers where overlays are implemented as possible. Also, apparently, the SteamOS compositor’s fullscreening is significantly inferior to what KWin already does.
To be blunt, “why don’t you give games their own equivalents to all the standard keybindings, then?” They’re standard for a lot of very good reasons.
I’d rather put up with slightly reduced performance than have to deal with complicating my muscle memory. It’s bad enough that I’ve got muscle memory tied up in a Vim config that no “Vi mode” can replicate and I occasionally type things like :wq in non-Vim windows or vice-versa.
Your suggestion about notifications amounts to building a compositor into every single game rather than just getting it right in kwin.
This is a very interesting idea. If it works, it seems a quite nice trade-off between performance and usability, as I don’t care using ctrl+alt+f1/7 for switching between sessions…
Interesting, thoughts…
First question which comes to my mind is do you have any numbers on this?
E.g. for the interrupts caused by kwin graping cpu/gpu time, etc.
Secondly, to add from what others already said they need interaction with their DE: I sometimes need to change my keyboard input from DE to US, to properly interact with games. I see this as very problematic, when the game handles the input itself…
Third, to others who complain about their (missing) team speak:
In digital distribution platforms (e.g. steam, desura, etc.) – which are obviously the way to go/future – you have already overlays (ie. interaction) IN the game from that platform.
So this means that actually Steam/Desura would provide your third party team speak service (and others etc.), not the individual games…
And lastly a funny related video, which I had to thought of 😉
While I appreciate that you want to think about the performance in games, I don’t think your solution would work.
First of all you would offload the burden to implement this to all game developers and everyone would have to duplicate this effort.
Secondly you would lose a lot!
You can’t alt+tab anymore. Strg+Alt+F7 is not a solution to this, as it is different to how it works, when the game is windowed.
You would lose notifications and overlays. Your battery is running low? Well, don’t save, we will just shut down. Want to change the volume? You can’t. You won’t know if your sound is muted or if it is already on the maximum volume. Also your shortcuts won’t even work.
Want to switch your game to windowed mode to open a walkthrough next to it? Won’t work without restarting the game. Bad luck, if you play a roguelike without saving or and instantiated online game.
I don’t even want to think about multi screen setups.
I would propose an alternative solution. You already shouldn’t receive damage events on minimized applications. Implement the same, if one application is already fullscreen, either by telling the apps they’re minimized, blocking damage events altogether or make it possible to suspend damage events and rendering of all non-visible windows, as no window, that can’t be seen should use resources on drawing and notify damage. If they are visible again, redraw the full window.
Then do the next step and hand over the page flipping to the application, if it is running fullscreen and uses buffering, like it can be done on windows 10. ( https://www.youtube.com/watch?v=E3wTajGZOsA ) (That is probably what the unredirection is about)
That way all of the desktop goodies should still work but kwin should be able to completely go out of the way and only enter the action, if it is asked to show an overlay. Hotkeys should also still work, but I don’t know if kwin would need to wake up for propagating the input. All rendering of unneeded applications should be suspended and no damage events should be received. (That would also benefit all applications running in fullscreen or maximised, not just games.)
Also running a game on a separate tty is possible on X by just launching a separate Xserver, that only executes the game (e.g. xinit /usr/bin/game — :1). Same should be done with the game if the user decides it is worth it on wayland, by implementing a minimal wayland compositor that only provides input to the game and the game draws directly to the screen buffer (unredirected). That would do what you suggested, but take away the burden from the game developers and only needs a simple wayland server coded up. That way all the funky input stuff is handled, you can run the game under the DE of your choice, if you don’t want to switch ttys etc.
That however doesn’t free wayland compositors from their responsibility to minimize their overhead depending on the situation. I would really like to hear, why that would not be possible in on wayland.
Although I disagreeing with you here, still always admiring your hard work and efforts spent,
Nicolas Werner
FYI, more discussion on it here: https://www.reddit.com/r/linux_gaming/comments/3w8845/gaming_on_linux_move_to_next_generation_from_kwin/
I don’t know Martin, I’m not sure losing desktop integration is a good idea.
It may be good for a gaming focused system (steamos maybe? I’m not sure) but on a desktop people expect some basic features like alt+tab, workspace switching, being able to drag the game’s window over different monitors, media player on the background, external voip software, etc.
Sure, from a pure performance (and code simplicity) point of view it does make sense, but it’s important to keep in mind that while the desktop environment may look like a useless obstacle, it is still providing features that users want and expect and that we otherwise would have to replicate elsewhere.
Do yo know if it’s possible to setup an activity that automatically disables desktop compositing and so when leaving that activity it’s enabled again?
You don’t need activities for that, you can do per window.
But then you need to manually configure kwin for every game that benefits from it. Annoying.
then use: http://kde-look.org/content/show.php/GameMode?content=156659 🙂
bad idea because all it does is increase complexity and solve nothing. if other apps are sending damages – they STILL SEND THEM regardless but now it just goes to another process. kernel still has to wake up and handle these other processes and their scheduling. it still has to transport ipc to the other wayland compositor (the desktop one) and it will still wake up and do things.
so it fixes nothing. if you really want totally ZERO interruption, then SIGSTOP other process id’s of other clients. work on protocol to ask clients to go into “deep freeze” where they voluntarily do as little as possible? don’t use drm/kms. don’t use another wl compositor.
by doing the above solution you propose martin, you break things like multi-screen. if i game on 1 screen “fullscreen” and have my regular desktop on the other. i do that myself today in x11 and it works great. in a good wayland universe we can have the fullscreen game on 1 head running without compositing (buffer swap that one output only) and the other keeps working as normal. my mouse moves from one screen to the other when i move it across (unless game grams/uses relative positioning, but games normally turn their mouse grab off when you pause and bring up a menu, some don’t grab at all).
so summary – don’t do this. adds complexity, breaks functionality and doesn’t actually improve/fix anything. 🙂
Indeed, that’s how the problem should be approached. And pretty much how things have been working in Android for years.
Use cgroups to starve stupid background processes out of CPU time. Optionally use other mechanisms (IO scheduler? D-BUS?) to further prevent unnecessary processing. Group processes by assigning to VTs. Problem solved.
Too bad, this does not require much work on part of Wayland guys (or anyone else from Freedesktop).
Martin, what do you think of looking at the problem in a “thing explainer”-style?
I mean a compositor is something that provides input and visual output devices to programs in a similar way that it accesses that hardware itself. It is some sort of multiplexer. And the way it does this multiplexing is the personal character of a compositor like kwin.
Screen space is a limited resource. Programs either need to wait until the compositor provides screen space or they need to be happy with a small space.
Similarly for input devices: Programs need to wait until the compositor provides the input devices or they need to be happy with a subset of inputs (e.g. only 1 mouse when 2 are attached).
One mind-boggling thing is now that the compositor uses part of the input signal itself to decide who gets the input device (Alt+Tab).
Okay – that sounds really toyish, but using that “abstraction” it’s just much clearer that the virtual terminals are also just a “compositor” that runs on the kernel level – its advantage is that introduces fewer context switches, but their implementation is not as flexible – but still a compositor like kwin could use its API for some very special cases (fullscreen) in addition to using wayland in order to provide more direct hardware access to clients and to suspend itself as much as possible and I think that’s where your ideas come in.
But on the logical side the problem did not change. Logically one could still speak of one compositor that is provided by “a soup of compositors” and that soup of compositors may contain kwin as the master and it commands virtual terminals and maybe LD_PRELOAD-hooks inside clients to do slave jobs like grabbing Alt+Tab and forward to kwin (I’m not sure if this is necessary aren’t input devices multicastable on the kernel side anyway?).
I don’t know what it will be, but window management is a philosophy too and given the big picture it will be easier to find a good implementation, I guess, so I think we should draw some beautiful graphs.
I’m also unconvinced, as:
• You should be able to get notifications from your system. Network up/down, sound up/down, low battery, new messages, etc.
• Security/power management: is it possible to lock the screen and go to sleep when you close your laptop? Who manages that?
• Control keys: who handles volume, brightness, etc?
• Also in the examples you stated, most (all) current consoles do draw notifications on top of the game, and also you can press to access an overlay to do stuff. Of course in those cases I’m assuming it’s done with the cooperation of the game, so maybe that could be a solution for the above problems: have a sdl plugin that gives you most of what your desktop gives you, in this simpler session, and games should not have to care about it, just load it, but I’m not sure if it would be that easy.
• How to deal with unresponsive games?
A further problem is that many game developers have limited resources, especially when developing for linux. This proposal seems to leave it to game developers to play nice in many cases, and history has shown us that they don’t have the resources to do so, and linux is not a static platform as consoles are, so we need to have some abstraction and isolation so that current games will work correctly 5 years from now, instead of being awkward because they were developed or never patched to suport linux-games-standard-somenumber.
You don’t get them currently. Notifications are stacked below fullscreen windows.
Sure, AFAIK logind handles these cases by default and Plasma (powerdevil) has to actively disable them.
Same as above, logind could handle them or a dedicated mediator process.
Well of course it needs a mediator process which can monitor them. I simplified the outline of the setup in my blog post. Of course there needs to be a way to monitor the game, notify whether it crashes, notice if it hangs, etc.
>> • You should be able to get notifications from your system. Network up/down, sound up/down, low battery, new messages, etc.
> You don’t get them currently. Notifications are stacked below fullscreen windows.
And that sucks, right? I was hoping that wayland would provide a solution to this issue.
But thinks for your answers 🙂
I’m not not entirely convinced, but you do know a lot more about this than me.
indeed on wayland fullscreen surfaces don’t have to be on top of the stack, so you can have notifications on top. if that was not possible on X that’s yet another reason to switch.
no, it has nothing to do with Wayland or X11. It’s how we in Plasma/KWin designed it.
I hope it will be an option to change the way notification are shown in both, even if it would be disabled in default setup.
It’s worth noting that there was experimental work done on switching desktop environments without having to log out of your existing session a while ago.
The work hasnt landed because user bus hasnt landed yet(was going to be an alternative to session bus with kdbus).
It might be worth asking the kdbus and logind guys how things like unresponsive sessions where going to be handled.
I really like your idea Martin, I was thinking about something like you have described for a long time now.
I don’t think it would be as difficult for games developers to work with as some people think. SDL could abstract the session away.
Valve distribute SDL independently from the games themselves as well so that’s something to ponder in terms of having legacy games “just work TM”.
Would be great to see a cross desktop effort here.
Freedesktop.org specification maybe.
Another thing that would be interesting to look at would be multiheaded gaming through kvm.
E.g https://www.pugetsystems.com/labs/articles/Multi-headed-VMWare-Gaming-Setup-564/
That’s also how I see it. The idea I outlined was meant for “game toolkits”, not for “games”.
No, it’s an xatom. Xlib or xcb will do. if one doesn’t want to use those directly then we have high level api for it.
that’s intended behavior in Plasma/KWin. We don’t want to go notifications above the fullscreen application. What we designed for is the embarrassing chat message during a presentation.
Then again, *SOME* notifications are important enough to be shown over a fullscreen app. Battery drained, for instance. Maybe loss of network connection. I believe it should be configurable which notifications are allowed to appear over fullscreen apps.
Why not just make user (or the presentation app) mute notifications for the duration of a presentation?
This idea is to radical for real world. It breaks lots of use cases like TeamSpeak, screen capture, important notifications, weird gamepads, hanging games and so on. And, what is worse, it requires code changes from game developers.
But having dedicated wayland server for games could overcome most of this problems. So this way is probably is worth a try.
I think this would make more sense as a custom wayland compositor that you spawn on a new vt, which only maps a single surface, always fullscreen, always scanning out from the buffer passed in by the app.
This would get similar results, except it would reuse all the code for modesetting, input, etc. And this mode is optional, you can also run it on your regular wayland compositor.
In fact, you could probably switch the running instance of a game from the custom compositor to the regular one, reusing your existing DRM buffers when you press the “ungrab” key.
The situation on Windows has for a long time been about moving from fullscreen to borderless window, and this is what Wayland should strive for. Minor performance optimization don’t matter for gaming, whereas losing the ability to alt+tab, display certain notifications/overlays above fullscreen windows, is a big problem.
The worst part about the X11 gaming situation is that it allows games to grab any and all input, meaning a crashed game, or one set to an invalid resolution kills the whole machine for any user not familiar with fixing it either remotely or via CLI mode.
Anyway, that’s just my five cents. Focus on use cases and usability, not implementation details and eliminating nowadays-insignificant indirection.
+1 for mentioning the gaming trends on Windows.
That’s nice but it would mean the game has to link against the KDE libs, right? I’m not aware of any game doing that and can’t imagine any developer (who often only officially support one series of GPUs already!) doing that. If this is available over D-BUS or something similarly generic, then maybe you could convince developers to set this hint.
Apart from that, I’m not sure your idea is very good for the user (besides the performance increase, obviously). Losing the ability to switch between multiple applications is bad. I think the real solution will be something like what Vulkan is set to offer. And if you want to reduce resource usage, how about switching the desktop background out for a plain black area while some application is in full screen mode? There are probably a lot of other small things, that can be done, to improve the situation incrementally.
Anyway, thanks for the post to get the discussion started. Did you reach out to game developers (Feral, open-source game developers like the team behind Battle for Wesnoth, Aspyr, Valve, CD Projekt)? Or even the Mesa/Wayland/X developers? I think it would be good to get the ideas of all involved and find a solution that can really work for most if not all games (and other full screen applications, that could benefit). Maybe a BoF could be set up at FOSDEM or the next X Developer Conference?
Once you design and implement a usable and useful protocol in just one DE and a few example FOSS applications, you can then lobby the Freedesktop people to adopt it as an official standard that others will follow. Most current Freedesktop standards started this way.
Could you use proprietary drivers for this game mode and open source for desktop?
If we get Wayland to work with proprietary drivers, it will work
Cool, even with separate graphic cards, IGPU for desktop and dedicated one for Games?
How would steam fit into this all with its ingame overlay etc?
My guess is that you will still need both drivers to share a single, open source kernel driver. This is an architectural decision that driver vendors will make, though the benefits are numerous and I think all of them will eventually go down this path. AMD already works on that for GCN 1.2+ cards, Nvidia engineers made some noises about internally working on something similar too.
As you know, in PTS no game ask the compositor to disable the compositing and it’s probably the same for the vast majority of games. As a result, and it’s a shame, KDE is seen by the vast majority of users as poor desktop for gaming whereas it’s not. It just need to be correctly configured. It may be not the perfect solution but fullscreen unredirection is a good trade off. In a perfect world every fullscreen app will ask to the compositor to block the compositing if they need it but unfortunetly that is not happening. So it’s maybe not a so bad idea to enable it by default ?
If I remember correctly the fullscreen unredirection is not available with intel drivers due to an old problem. Maybe it’s time to allow intel users to use it again if the problem is fixed ?
Whereas advertising what seems to be a kwin only feature (setBlockingCompositing) don’t you think it should be better to advertize the use of the more standard _NET_WM_BYPASS_COMPOSITOR (http://cgit.freedesktop.org/xdg/xdg-specs/tree/wm-spec/wm-spec.xml#n1579). By the way there is a bug report about this feature that doesn’t work in kwin : https://bugs.kde.org/show_bug.cgi?id=349910
Thank you !
No the Intel but is not the reason why it is not enabled by default. We think it provides a bad experience for office usage.
I understand that fullscreen unredirection can provides a bad experience for office usage but other compositor activate it by default and users don’t seem to complain. On the other hand KDE is seen by the vast majority of users as poor desktop for gaming.
I didn’t say that was because of the Intel problem if fullscreen unredirection is not enabled by default 🙁
Is it possible now to allow Intel users to use it again if the problem is fixed ?
What about this https://bugs.kde.org/show_bug.cgi?id=349910 ?
Thanks !
There is an important difference: we have the possibility to disable compositing. Given that unredirection of fullscreen windows is not well integrated in our setup resulting in a bad experience. We don’t invest time into it because games are better served with disabling compositing.
We thought so once and enabled it again and it broke again. Fool me once shame on you, fool me twice shame on me. No, sorry the risk isn’t worth it.
It needs work and someone caring about it. Currently all KWin devs seem to think that it’s not needed. So someone interested in it, needs to implement it. Want to do it?
If unredirection of fullscreen windows is not well integrated maybe it worth it to improve this because even if games are better served with disabling compositing kwin and kde are perceived as bad for gaming.
I understand the frustration with the intel driver but someday it could be good to reactivate the fullscreen unredirection feature for the users of intel hardware. After all it will undoubtedly work one day.
I’m not a dev so I can’t help to implement _NET_WM_BYPASS_COMPOSITOR but it’s weird that you have your own system but don’t use and not interested in something standard.
I have zero motivation to invest time in something which I consider a bad solution compared to the real thing. I absolutely don’t care if people think we are bad at gaming because we don’t perform tricks to make meaningless numbers bigger. Sorry.
Our system predates the standard by several years. Implementing the standard means quite some work and hits the motivation problem, because I think it’s stupid.
Martin you are a very intelligent person and your work is absolutely amazing. That said I understand that you can be frustrated by some things like the Intel drivers or your system that predates the standard by several years but if you think as a user for a minute you will see that them too can be frustrated. Whether by the lack of performance by default in games, whether by the lack of the now standard _NET_WM_BYPASS_COMPOSITOR, whether by the lack of motivation of the developer of their loved software, etc…
It’s not really fair to indirectly “punish” the user for problems that are not here anymore (_NET_WM_BYPASS_COMPOSITOR) or lack of motivation. It’s often like that in open source, some wonderful people spend a lot of time developing really great software that are 99% perfect but almost always lack the one more percent of polish.
Remember that is us that use what you make about 10h per day everyday and an insignificant thing for you can on the long run drive someone else crazy 😉
That’s the kind of little things that may be insignifiant to KDE devs but can drive crazy users 😉
https://youtu.be/WSuCGYsYocE?t=33m39s
What about a root layer compositor (shared between kwin, gnome, etc) which handles a single, accelerated and direct context, while using libinput to capture all input. This layer would trap certain agreed key combinations for window switching, etc – and receives buffers from the compositor via an agreed interface which contains compositor-specific alt-tab window contents. Everything else gets passed straight to the compositor.
Framing the problem is key.
User control > performance.
Most gamers on Windows use “Windowed Fullscreen” when they reach even a basic level of sophistication. You should make sure you are aware of the history of why.
Alt tab isn’t trapped in a game I’ve played in many years (I’ve used Linux as my sole development/work platform since the late 90s but I keep a dedicated Windows gaming box).
PC gamers use a broad variety of overlays, webbrowsing game walkthroughs, chat/voice clients, alternate control schemes via macro programs or exotic hardware (frequently needing overlay), another game, home/officework and really a broad variety of things that you can’t completely foresee.
It’s been the work of two decades to get this all working reasonably well for gamers on Windows.
Your proposal seems to be a substantially worse deal for people coming from Windows.
Your proposal seems to be handing a proprietary application+some random version of SDL, a full term and leaving the user to fend for themselves.
If kwin/wayland isn’t focused on solving these problems with user control as the priority, it’s not going to be a window manager I’ll use.
At some point I’ll expect sandboxing to be viable against these games and proprietary apps as Wayland becomes the primary display (even if this means another performance hit).
Hi again
I put some thoughts for such zero-overhead window management together,
which solves everyones usecases, is glitch-free, minimizes context-switches to the absolute minimum, keeps the window manager securely in control, and is a non-problem for game developers, but requires changes in KMS and how wayland internally works.
I call it demediation and it takes unredirect one step further.
http://repository.violetsky.ch/other/demediation/window-manager-demediation.html
it’s based on my vague knowledge unfortunately, but I think I’m not technically that inaccurate that the concept becomes wrong. I think it works.
About alt+tab. In my opinion, games should use more generic API to assign action to key combination. Currently is a mess.. I my opinion games should assign actions to path, such like /player[0]/move_forward and special library should send /player[0]/move_forward instead of key code. That will solve some problems with different keyboard or other controller configuration and allow to not break game or system, because key is assigned to system action and game action.
Of course – there should exist backward compatibility mode for actions, which isn’t defined by special parameter to action assigned API call.
Action assigned API should take flags, like allow user to calibrate controller in fallback mode.
Such mechanisms like Alt+Tab are a fallback for the user, most needed, when the situation gets ugly…
As long as everything is fine, we won’t need it – but if unnormal situations occure, the user only will remind an “uncomfotable” experience more – and rejects playing on Linux (for example).
I’m depending on the omnipotent Alt+Tab too, and I can’t remind a game which doesn’t support Alt+Tab flawlessly (or i did stop playing crapware)… 🙂
About the high-performace-modus:
You shouldn’t try to solve it your own way, because the implementation of the Wayland system is a “historic moment” in a very important time: Linux is about to get widely accepted as an alternative to Win/Mac, finally getting all consumer important applications on Linux… We all know, that the fundament you are preparing has to persist for for quite a long time.
I’m sure that the game developers already have some ideas, whishes or kriteria for previliged rendering.
So go ahead and ask them about their ideas and whishes, and they will probably submit you some help – at least they are depending on a good solution too. Some big-shots are about to provide Linux support by standart, so how about contacting Crytec (Frankfurt), Valve or Blizzard? Even if the Steam Machine runs Ubuntu… 🙂 At least ask them on how the API’s should work; industrial developers often have to do stange work arounds, because the system developers didn’t expect those unsecases …
Thank you for this discussion, because this is a really important issue and will be critical for KDE’s future – so keep up your great work!
I think the solution you propose is great on a single monitor system. Not so much on a multi-monitor system. I like having a web browser open on the second monitor if I’m following a walkthrough. Assuming nothing is showing if you’re gaming is only for single-monitor people.
I’ve learned newer and more effective things through your blog. One other thing I’d prefer to say is the fact newer computer operating systems tend to allow additional memory to be utilized, but they furthermore demand more memory simply to operate. If a person’s computer can’t handle additional memory as well as the newest computer software requires that storage increase, it might be the time to shop for a new PC. Thanks
Old post, and I’m not a gamer nor know anything about software engeneering, but I have never understood how Linux wasn’t way better for gaming than Windows since the OS is independent of desktop environments and window managers, and, in theory, a game could be launched from a command line session, no? So, if I’m right, all the “potency” of the machine would be for the game, no QT/GTK/whatever libraries “parasiting” resources, no other graphical apps or events interrupting, etc; that’s impossible in Windows, where the desktop is tied to the OS, am I wrong? Then, if one really wants to spend an aternoon just playing games at home, why not the possibility of running them from a “nox” session?
Anyway, whatever happens with games (as I told, it isnt something that really interests me much) we are hearing of Wayland near to a decade; could it happen that when it is finished it might be born a little aged? I mean, since its foundations back to ¿2006, 2007?, haven’t modern cards and DEs evolved a lot so Wayland might not be on pair? As a user who wants the best performance not only for me but also to help me to convince my friends to try (and specally stay. Many have tried and abandoned) Linux I hope not, of course, but 10 years eem dso much in computers stuff…
A lot of interesting improvements regarding gaming on Linux.
However: “We want to have a PlayStation like setup: game everything, everyone else nothing. I don’t want KWin to take away important CPU/GPU time from the game.”
Probably great for some, but I don’t like to get isolated from my system. When using a multiple monitor configuration setup with 3-4 monitors I have experienced with X11 that all but one monitor is turned off when in fullscreen. Why not let the system still use the other monitors while the game is playing. Your suggestion that the game runs in its own Wayland server sounded like a great approach.