Definitely; pandoc is great and most people are better off using it than LaTeX directly. The comments above mine meant "replace" as in "displace," though and some of the important functionality of LaTeX, stuff that matters a lot when you're writing a mathematically technical book (cross-references, indexing, etc) is unlikely to be implemented in markdown. Especially since, in this context, pandoc's approach of converting the document to LaTeX doesn't help: we're still stuck with LaTeX's limitations!

It would be interesting if some of these lightweight alternatives --- pandoc, org-mode, etc --- would target TeX and handle the LaTeX functionality themselves. It would obviously be more work, but it might lead to something self-contained and distinct.