Your source would also have to have a reference to which exact version of which compiler to use, which versions of which external headers to use, etc. and now you're inventing Nix.

Conceivably there could be a standard for a sidecar file to specify how something was built (e.g. nixpkgs commit hash, or all of the parameters that went into the build). Or content address the inputs, i.e. invent Nix again.

So we could solve this problem by having everyone standardize on using Nix.