I've not tried org-mode, but I've started using Markdown with pandoc (http://johnmacfarlane.net/pandoc/) to do much the same thing. Markdown makes actually writing much quicker and more readable, but you can still insert raw LaTeX where you want to do something extra (e.g. \cite).

Additionally, Markdown is far less threatening to technophobe collaborators. I've successfully written grant applications with clinicians using Markdown in an etherpad document (free hosted version at http://piratepad.net/), which I later converted to LaTeX/pdf for submission. They would never have put up with writing lists or headings in straight LaTeX.

EDIT:I should have stated that pandoc is a nice tool for conversion between a bunch of formats including Markdown, Latex, Html, s5, reST, RTF, epub and a few others. In response to the comments below about conversion to word always sucking, pandoc provides the least sucky attempt I've found so far.