With the current setup, people are trusting mozilla/google to: Give you the correct software, Update silently, Determine which CA certificates to trust by default, and Determine which certificates are valid by pinning.
The CA is trusted to do: Determine which certificates are valid.
Not really. For firefox anyone can build from source (see also iceweasel), disable automatic updates. For chrome it's mostly the same, but then for Chromium instead.
For people who build from source, all control is at the user. They are responsible for the security, and they do not need to trust anyone. The question about who should have trust invested in them do not involve them, as they operate outside the system.
Just building from source doesn't guarantee anything.
Firefox is giant. It shouldn't be hard for a malicious party — should one appear someday — to hide some tiny backdoor somewhere in a more-than-a-hundred-megabyte source code tarball.
Verifying GPG signatures of the tarball could prevent some (but not all) issues, but from my observations it's rarely done. And when I've seen it done public key's origin wasn't thoroughly verified, just blindly `gpg --recv-keys`'d from keyserver.
This comes up again and again. Sure, the individual users is unlikely to wade through the complete delta for every published version, but it's not uncommon for packagers to be involved upstream as well (with a few unfortunate outliers of course). Backdoors have been catched this way before.
There's a difference between "building from source" and "obtaining software from a trusted party". Your suggestion (trusting the packagers) implies the latter and is almost irrelevant to the former. If one trusts a team to catch possible issues, one may trust the binary this team builds as well.
The CA is trusted to do: Determine which certificates are valid.