In short: it would be a legal mess. Most inhouse game engines have a lot of dependencies on closed-source code, mostly 3rd party specialized engines for AI, path-finding, physics, loaders for asset management code, or code that talks to the game console SDKs. Even releasing code which only calls into those closed source libs might be illegal. And even if that would be possible you would need (commercial) licenses for each of those 3rd party libs if you want to compile the engine code yourself. Also, by releasing the code you're inviting patent trolls to harvest your code for patent violations. Hardening a big inhouse engine for an open source release is easily just as much work as creating the thing in the first place.

[edit: typos]