Even for building a mobile backend, having a framework can help. Tools like Cornice (Pyramid), Django REST framework, and Flask-REST can make building API's much easier.
Absolutely. If you already know one and like it, go forth and hack. If you're not sure, or don't like the one you have, there are tons of great options. Not just Flask/Django/Pyramid either. Tornado, Bottle, CherryPy, Falcon, and more all have dedicated users (some more than others).
For some things ORM's get in the way, but avoiding them by your own data layer sounds like a pretty big time-suck.
Being able to write a single line of SQLAlchemy to build relationships into your models seems way simpler than implementing relationships in a custom data layer.
I don't deny that it's more work. The question is whether it's worth the investment. I think for large, complex codebases (not little one-off apps) the gains in performance and long-term maintainability are worth it. Having the ability to know the exact performance implications of a query (and not having to waste brain cycles on questions about lazy vs eager loading, etc) is more desirable.