I wouldn't do that. Only move to Wayland when there is a real path for what you want and need. And it does it without hacks.
About porting I would look at what MATE is doing or at the code of Wayland shells, protocols or things like Waybar at GitHub. You don't have to write an entire compositor if you don't want to. Even better get in touch with the people who know what they are talking about, I think they have a wayland IRC channel.
There are a lot of unmaintained WMs that work well and are amusing for historical value or retro computing, and probably some of them have a small number of real world users too. I like the idea of a compatibilty layer for those.
Maybe not for a big project with ongoing development. But as a last resort to run some cool stuff.
It seems it would be difficult to do that without combining both protocols into one server. (This is not quite what xwayland, xweston, xwlnest do. They run the server X out of process and can't manage wayland windows because they don't know anything about the internal wayland state)
To that point, you would have to decide if it's more worth it to fork xwayland and merge with an existing wayland compositor, or to fork xfree86 and add wayland.
I bet you could achieve it with three layers. First, run a "top level" Wayland compositor. Then, use XWayland to run an X session (with your favorite window manager). Then, use Weston's X client mode to run Wayland applications inside the X session.
As silly as that sounds, it might actually be a good solution. Red Hat gets to push their shiny new display server protocol, and I still get to make screen recordings and use my favorite window manager. I wonder how well it works in practice...
That would sort of work too, but it comes with its own set of issues. Keep in mind that there is very little reason you would actually want to do it that way when either way you're dealing with the same impedance mismatch. In other words, you will have to solve the same issues in the display server you're running into now with screen recordings and strange window manager behavior. It's not something that you'll just get for free. If it was, the other Wayland implementations would have it fixed by now. There's no magic solutions to this without any brokenness or workarounds—the crux of the problem is that you have two protocols you're trying to match up that are compatible in some ways and incompatible in others.
Also I think it's a stretch to say the push behind Wayland is coming from Red Hat. Maybe within GNOME, but there are other implementations with no connection to Red Hat.
I don't think the impedance mismatch is the same either way. The Wayland protocol is much more restrictive than X's -- things like screen capture and window managers don't work because the APIs they need aren't present. Conversely, there are X APIs for everything a Wayland application is able to do, so a Wayland-app-in-X shim would probably work a lot better.
The outermost layer, where you run the X session inside a Wayland compositor, would probably work alright too because it just needs to send pixels to the screen and accept mouse/keyboard input. Wayland is able to do both of those things!
>Wayland protocol is much more restrictive than X's -- things like screen capture and window managers don't work because the APIs they need aren't present
This is not quite true. There are protocols for these things in some implementations. You'd want to keep compatibility with these because otherwise you'd have no reason to be supporting Wayland. Keep in mind either way you're talking about using a specific compositor—it doesn't matter if the APIs aren't present in the protocol. In this situation you'd re-use one from another implementation or add another private protocol extension. That's always been what individual compositors are encouraged to do. It's no different here.
> This is not quite true. There are protocols for these things but they live in multiple places.
Every Wayland window-manager-compositor-hybrid invents their own protocols for these things. This basically means that you can't write a program that does these things -- you need three different versions of every API call you make, and that's just to support the popular desktop environments that people happen to be using this week. So no one has, so the only Wayland protocol features you need to support are the basic ones to open a window, draw into it, and get mouse and keyboard input.
> Keep in mind either way you're talking about using a custom compositor
No, I suspect the "top-level compositor" could be any off-the-shelf Wayland compositor that supports XWayland and can make a window full-screen.
Here is the thing: it's trivial to map those basic features back to X. That's already mostly been done. If you continue with that solution you'll have the same problem in reverse—assuming X support ever gets removed from the toolkits, then in that situation you'll get Wayland apps that depend on some Wayland functionality that doesn't work. See what I mean?
(IMO we're pretty far off from X support being removed if it ever even happens, so I wouldn't worry too much about that right now)
And I actually don't think it's a big deal that there are separate implementations. You can absolutely still write programs that do these things, and people do. Usually you start with targeting just one implementation and then you port it to the others. All you need is an API to abstract the custom protocols. A lot of those differences are already abstracted in the portals API which has those three separate implementations so you may not even need to do any porting if that fits. Worst case, if something isn't in the portals API then you have some three clause switch statements when you make protocol requests. Then after you stabilize it, move those switch statements into a library and then other people can use it as an abstraction layer, and nobody needs to write the switch statements again. Yeah it's work, but for an app developer it's actually easier to do that than getting all the upstreams to standardize everything.
Here's the thing about Xwayland: Xwayland is Xorg. If you look in the xorg code base, you will find that it's separated into components called DDX and DIX. DIX (Device-Independent X) handles protocol and server state; DDX (Device-Dependent X) does the drawing. One of the back ends in the DDX code is for Wayland; when you compile Xorg with just this back end, you get Xwayland.
So if Xorg gets abandoned -- oops! There goes Xwayland as well. I do believe that this was the plan all along: keep Xwayland around as a solution for "legacy applications" for a while, then stop supporting X altogether to make it less attractive as a back end for new applications and toolkits.
Yes, I am familiar with the Xorg code base :) The point I was getting at is that if you want to keep using an old protocol far into the future, someone from the community has to step up and maintain something. That's true of any old technology faced with waning influence over time.
About porting I would look at what MATE is doing or at the code of Wayland shells, protocols or things like Waybar at GitHub. You don't have to write an entire compositor if you don't want to. Even better get in touch with the people who know what they are talking about, I think they have a wayland IRC channel.