LaTeX will get you excellent results if you master it. I don't think you can get comparable results using anything except for professional typesetting software and a lot of manual labor. But LaTeX will cost you a lot of effort to get right, and you will not get it right without a LaTeX expert coming in to help you.
If, however, you aren't great with LaTeX, I'd use Pages on a Mac, with LaTeXit for equations and EndNote to manage your bibliography. Styles in Pages really work well and make editing large documents easier, not more complex. EndNote integration is clunky, but works. The only major drawback is lack of support for numbered equations and figures in Pages.
Mind you, I'm not saying this is the perfect solution, but it is much better than Word or OpenOffice.
If, however, you aren't great with LaTeX, I'd use Pages on a Mac, with LaTeXit for equations and EndNote to manage your bibliography. Styles in Pages really work well and make editing large documents easier, not more complex. EndNote integration is clunky, but works. The only major drawback is lack of support for numbered equations and figures in Pages.
Mind you, I'm not saying this is the perfect solution, but it is much better than Word or OpenOffice.