The Minecraft Server protocol has been documented by the community since 2009, there are a wide variety of custom servers in existence today, for example Myne which is open source: https://bitbucket.org/andrewgodwin/myne2/src

When Java is compiled, it is converted into java bytecode language that is almost one for one with the Java language. This means it is possible to decompile it, although you'll often lose the variable names. This is why there has been so much modification, and custom clients made.