I wish that unoccluded or maybe just the frontmost window could be unredirected too, even if it's not fullscreen. The decorations and shadow could still be drawn in the Compositor Overlay Window.
I work on Linux on a program which is entirely drawn by OpenGL (I target ARM devices, but I test on the desktop before doing an ARM build). Application GL performance suffers immensely under a compositor, so on most machines I can't use GNOME3 (though I would like to). On machines where I get acceptable GL performance in my app, vsync is broken so there is constant tearing.
I'm sure there are some caveats with unredirecting the frontmost window -- like when it becomes occluded it would have to be redirected and that requires a round trip to the client (which might hang, etc), but there are probably other situations like this which the window manager has to deal with already (resize timers?) so maybe there is a way to implement it...
I wish I could see this kind of discussion in the wayland/name-your-compositor mailing lists. Currently it would be easy to implement a new attribute in the ICCCM so that applications that don't need foreground unredirection (ie: they actively want to be transparent, which is a minority) can signal so, but it looks like that with the unification of the window manager and compositor we are losing this kind of extendability.
I do realtime graphics with GL for a living, and I'm really sad at the state of the linux desktop, but especially worried at the future. GL performance with any modern distribution is worrysome due to the compositors, and removing it is the first thing that I need to tell any customer lamenting performance issues. Being unable to set vsync on a per-window/context basis is really a major problem, and the performance hit that you get is unacceptable. It seems that removing vsync is all hip these days for games and toys, yet tearing is unacceptable in any other context when you can actually get triple buffering with modern hardware.
Why are we optimizing for toys, when what matters are actual applications and everyday's performance? People don't seem to realize that while the GPU can be used as a general purpose computing unit, the usage pattern is vastly different. The process handling the screen often must have either higher or total control over it many times. You immediately notice video stutter and frames lagging.
But what do I know... I do the same with audio trying to squeeze decent latency with PortAudio and/or alsa directly, while people go through the pulseaudio pipeline...
Part of the problem is also that the compositor often can't know if the frontmost window (or any window, for that matter) is 100% opaque. If your window has partial transparency, or (for example) implements non-rectangularity by using an ARGB window and setting the alpha for some pixels to 0, then unredirecting it will break that.
I suppose you could just restrict the optimization to windows with an RGB visual, though. But then you need to make sure there isn't something else modifying the opacity of the window. I've written a standalone compositor before (that is, a compositor that runs independently of the window manager, as a separate process), so you can't always assume that the compositor "knows best"; for example, the WM may want to make the window decorations semi-transparent. Or maybe the user has specified slight transparency for all their windows. Or something.
Hmm... in the end, though, you could probably come up with a heuristic that works in most cases.
I think that Mac OS X unredirects unoccluded windows; I've noticed windowed computer games suddenly getting much slower when I move another window in front of them.
It appears to, or has some other trick... My coworker has a huge monitor, and his MBA doesn't hit 60 FPS on any of the fullscreen animations, but individual window contents can go at 60 making me think they don't repaint the world every time a window changes.
I work on Linux on a program which is entirely drawn by OpenGL (I target ARM devices, but I test on the desktop before doing an ARM build). Application GL performance suffers immensely under a compositor, so on most machines I can't use GNOME3 (though I would like to). On machines where I get acceptable GL performance in my app, vsync is broken so there is constant tearing.
I'm sure there are some caveats with unredirecting the frontmost window -- like when it becomes occluded it would have to be redirected and that requires a round trip to the client (which might hang, etc), but there are probably other situations like this which the window manager has to deal with already (resize timers?) so maybe there is a way to implement it...