Asphalt is itself very energy inefficient, and finding a replacement is rather important. Quoting kurzgesagt, the impact if manufacturing an electric car is equal to just two meters of asphalt road.
On top of that, asphalt is very easy to recycle and is likely the most recycled material on the planet by weight. My original point was that we will still need to extract some oil for a long time into the future.
