If hexl-mode on the binary works on my home PC but compiling the source code costs me millions of dollars in compute then I want the binary. Someone with millions of dollars to spend on compute may have a differing opinion.
This argument only barely holds water for those big SOTA models like llama derivatives, and that's only because of practical costs involved.
Or should I say, it held water until few days ago.
Personally though, I never bought it. Saying that weights are the "preferred form of the work for making modifications to it" because a) approximately no one can afford to start with the training data, and b) fine-tuning and training LoRAs are cheap enough, is basically like saying binary blobs are "open source" as long as they provide an API (or ABI) for other programs to use. By this line of reasoning, NVIDIA GPU stack and Broadcom chipset firmware would qualify as open source, too.
So, as you state yourself basically, the result also depends on training data, which makes it part of the "compiled source" in a way, just like the architecture of the model. If you have the training data, you can modify that.
But probably it is impossible for them to release the training data, as they have probably not made it all reproducible, but live ingested the data, and the data has since then chanced in many places. So the code to live ingest the data becomes the actual source, I guess.
Cost of building is a true concern, but it doesn't stop people from forking large open projects like Chrome or Firefox and try to build a project to pursue their own ideas and be able to contribute back to the upstream projects when it makes sense.
I don't build my browser, it's too expensive, but the cost of building has nothing to say with how open the access to things. It'd be cool if the community could fork the project, propose changes and maybe crowdfund a training/build run to experiment.
This is like saying, hey, a regular binary executable is fine because I can edit it with hexl-mode.