The presence of stone gardens can be explained simply because everybody in that place was cutting rocks to build giant sculptures, and this should be done somewhere or dumped somewhere. The really strange part would be not finding rubble and stone gravel accumulations in many parts of the island.
IMAO, the stone gardens could have at least three reasonable uses.
It seems that they are distributed in coastal areas. It seems also that many moai were built or moved to those areas. Therefore there must be a place, somewhere in the island with a lot of accumulated rubble, but this place will not be just a lot of tuff gravel.
To cut a rock, you need something harder. Egyptians used metals, quarz sand and granite but you can't always find rhyolite in a volcanic island. I assume that there is not a source of granite on Eastern Island and that mining metals or diamonds is out of the table.
We could speculate that antique sculptors learned soon that basalt was the most convenient source of polishing moai. Can be found everywhere in a volcanic island, and is harder than tuff.
If we don't find art carved in basalt in that culture, that would suggest that the culture was restricted to carve soft rocks only, by lack of a source of tools harder than basalt in a significant amount.
Is also logical to assume that after a while the polishing material gets worn and must be discarded and replaced after shrinking to a size too small to be manageable.
If this size is the same of the gravel found in the stone gardens, we have a theory here about the origin of this rubble accumulations, even if they were recycled later for culturing sweet potato wines.
