I don't know about homebrew, but macports has clang and other toolchain stuff, so I think it might be possible to uninstall Xcode after installing all necessary tools through macports?