No experience with py2app, but I've used py2exe and pyinstaller in production.

They can be made to work, but you regularly stumble on some more or less obscure bugs. The short of it, is that you have no guarantee that the compilation step won't introduce bugs in your program, so you need a solid test suite for the compiled binary. It's feasible, but clunky, and you really get the feeling that the compilers aren't first-class citizens.