This. It's the otheredness of the conversations which happen at a cafe that makes it work. If you found yourself at a cafe sitting next to, say, the co-worker you can't stand, the director you've heard is trying to kill your project, an ex-partner and their new partner, I suspect the scene could get quite distracting rather quickly.
That's why you never date your co-worker. And if you couldn't help it and fell in love then you should quit your job first and ask him/her on date later.
My point was that an ex-partner, regardless of co-worker status, is a distraction. Having to encounter one at work all the more so, yes, but that wasn't what I was limiting my example to.