This is pretty common in Mexico. Go to the Mayan village of Coba on the Yucatan Pensinsula and you will see hundreds of small mountains. What used to be pyramids, are now hills as trees and bushes have grown through them. When I was there some years ago they told me only 5-10% have been restored and made back into pyramids. It's an extremely time-consuming process involving large teams to ensure each brick is put back into the right place. Most times you just see a pile of bricks with no sign of a pyramid structure.
I was in Lima recently, and there was a ruin right in the nice Miraflores area (with plenty of high-rises) that had been a temple complex of some sort. Where they hadn't excavated, the outer layer of brick had just crumbled to dirt, and looked like nothing special. Apparently part of the temple complex had been leveled and used as a soccer field as recently as the 1980s.
Even the structures built from stone were dismantled for the materials over the years. It's why Machu Picchu is such an amazing find--the locals didn't know about it and thus didn't tear it apart over the centuries. Compare this to grand sites like Saksaywaman that were picked apart down to the foundation by people looking for building material.
