> The only difference is that third party apps can't install apps "silently" (you have to manually confirm each install/update through a system dialog).

That's a huge difference. Try downloading and maintaining apps installed with F-Droid on a device where it doesn't have system privileges. It's a massive pain in the butt.