It can go rather hideously wrong, with the open source version being neglected, downgraded, or simply made non-functional to push people onto the paid version. This was happening for a while with SugarCRM - I don't know if anything's changed there since I last looked, but it feels just like a bait-and-switch.

Where I know of it working well, it's where there are separate legal entities to manage the open source code and the premium version.