I'm not playing Minecraft, but this has fascinated me from afar. AIUI, Microsoft has two teams on two different continents essentially developing two very similar but subtly incompatible versions of the same game. Funky.
Do they share any code or formats? Are the client/server interoperable? Does the C++ one even have a server implementation?
Servers are not interoperable by default. There is 3rd party software that acts as a proxy letting Bedrock players connect to Java servers but it'll just remove anything that the more limited client doesn't support.
They both use completely different network stacks, Bedrock using RakNet over UDP, Java using Netty (?) over TCP, so they have to be translated accordingly.
Yes. So clearly it's not a question of being able to run, or of weird graphics quirks or something. It's a strange omission. Given that Mojang is now owned by Microsoft, it's slightly suspicious…
You can use Geyser [0] to "bridge the gap" to Bedrock clients. There are bugs here and there, but it works pretty great to get everyone on the same server.
I recently got into Minecraft due to kid reaching Minecraft age. I keep meaning to write an intro for technical parents, because it's very confusing at first.
The original was written in Java, and people decompiled and reverse engineered it almost completely, so there are tons of mods that reach in and change things deeply.
They then created MCPE (pocket edition) which was a "cut-down mobile edition". That evolved into "Bedrock Minecraft", which is not "cut-down", and is what runs on mobile devices and gaming consoles (XBox, Switch, PS*). (It also runs on Windows, but not on Macs.)
At this point, the Bedrock and Java editions are more or less equivalently complex, although there are still differences in how many mobs spawn (generally fewer in bedrock, presumably due to (legacy?) performance constraints), as well as tons and tons of differences, subtle and unsubtle, in where and how things spawn, how the mobs behave, etc. Notably, there are significant differences in how redstone works that render most Java edition contraptions non-working in Bedrock.
There are actually many, many more Bedrock players, because lots of kids play on iPads and consoles, but most of the notable YouTube personalities and servers people stream from (Hermitcraft, etc.) are Java, and Java has an incredibly diverse universe of mods and modded servers.
Mojang (now a division of Microsoft) seems to value parity between Java and Bedrock: new features seem equivalent in both, and many point releases contain minor parity fixes. But it's unlikely they would make complex dynamics like redstone equivalent.
IIUC, there seems to be a vague consensus that Bedrock is probably the future, for several reasons:
a) As you point out, having two completely different codebases and teams is crazy
b) Bedrock runs (or can run - MacOS?) on everything
c) Bedrock is C++, and hence can perform better
d) There are more recurring revenue opportunities for Bedrock: Mojang-provided servers ("Realms") are Bedrock, and there are the "microtransactions" mentioned in a parent comment: you can pay for skins and add-ons (essentially "mods" that use a provided and sanctioned extension API rather than modifying the binary/bytecode). Minecraft is unusual in that you pay for it once but they keep updating and adding content forever (so far), so recurring revenue is important for Mojang.
If you're getting into Minecraft, I highly recommend Pixlriffs' Minecraft Guide for Java edition players, or Prowl8413's Bedrock Guide for Bedrock players. They will walk you through the early game, and then explain how the various "farms" work ("farming" is when you figure out how to build structures and systems that exploit the game mechanics to produce large quantities of particular resources).
To answer a couple of your questions, the client/servers are not interoperable, but there are proxies like GeyserMC that translate.
The C++ one has a server implementation (Bedrock Dedicated Server - BDS) available in binary form, that you can run on Linux or Windows. As far as I know, there are no third-party bedrock servers that implement all (or even most) of the dynamics of the vanilla game, since that would essentially require re-implementing everything, but if contributing to that process is your jam, I highly recommend the Dragonfly folks: their Go server implementation is quite clean, their discord is active and helpful, and their various proxies and tools and libraries are very useful.
Has there ever been any community-led effort to create and maintain a "Minecraft spec" (similar to the Tetris spec) and make an open source reference implementation, or is the game to complex for that?
Kind of surprising Mojang didn't go into that direction (e.g. to match Redstone behavior).
IIUC, aside from non-deterministic timing of simultaneous updates, the things that are different about redstone in bedrock are mostly things that _could_ be considered bugs. eg. block spitting (extending a sticky piston for only one tick leaves a pushed block behind, rather than pulling it back in, which makes flying machines more compact in Java), and BUD power (interaction between block updates and redstone state changes)
Modders have done so; I believe the Minecraft map data structure, at least for the Java version (because it can be decompiled), is pretty well understood. The client/server protocol as used nowadays (?) was actually developed as an open source project at first, because the open source server software ended up being better than their own in-house one.
> IIUC, there seems to be a vague consensus that Bedrock is probably the future, for several reasons:
I doubt it.
First of all, usually the workflow is that the Mojang team implements new features in Minecraft Java, and then the remaining Microsoft teams follow along on the Bedrock versions.
Secondly, and most relevant one, throwing away the Java edition means throwing away a large subset of the userbase that will never bother playing the Bedrock edition, and in games from business point of view, the number of eyes per screen counts very much.
Yes, as the project says, Minecraft. Not the cut-down mobile game with microtransactions that claims to be Minecraft.