To be honest, I find it ridiculous (just like with social security numbers) how much you can apparently do just by virtue of knowing a passport number.

It shouldn’t work like that.