So was non-permanent building materials such as wood, to be fair.

Not really, that's the resource that disappeared faster than anything, being the simplest to get. Mediterranean forests have never been particularly dense, already the Greeks were moving lots of wood on the sea from the best locations. Stone was easier to get from the areas around Rome.