The sand and rocks in the soil is where phosphorus and trace minerals come from. Ideal soil has a combination of living carbon, dead carbon, and minerals. The fungi and bacteria breakdown the minerals along with weathering and make it available to plants. All the living carbon builds itself from the dead carbon. The dead carbon came from dead plants that pulled the carbon from the air with photosynthesis.
If it were a good idea from the perspective of farmers, then farmers would already be paying to do it themselves. This isn't the case though, so you will have to pay farmers to do it.
Why does it even have to be on farmland in the first place? If it's such a great idea with no environmental downsides, then dump the crushed rock onto other land instead. Then you can dump more of it without being limited by whatever is optimal for farm yields.