The whole 'virtual photons' thing is a physical interpretation of the mathematics, but it is a mistake to conclude that this is how it 'actually' is. There is a mathematical model that works. I don't think we know how it 'actually' is. There is every reason to doubt that this sea of 'virtual photons' has any physical reality in an independently verifiable way. Occam's razor doesn't favor solutions that involve multitudes of invisible, unmeasurable, particles popping in and out of existence.
In the end, all we have are our mental models and experiments to back them up.... we'll never know what anything 'actually' is - all we can do is model things to whatever degree of accuracy we can.
A construction worker doesn't need to go around thinking about his hammer as being made up of mostly empty space, and of the electromagnetic forces that give it it's properties, it's a hammer! We all know what it "IS" right?
But it's not that. It's not even what we think it is. It's not even what the best physicists think it is.... and in the end, all we're left with are incomplete mathematical models, or models that are only accurate on a certain scale.
BTW - how do we then explain hawking radiation, if virtual particles are "just math" and not real?
Well, first and foremost: the existence of Hawking radiation hasn't been experimentally verified. There's no direct evidence (measuring radiation that could have no other source than the evaporation of the black hole) nor any indirect evidence (black holes losing mass at a rate that cannot be explained unless Hawking radiation is included). So although it seems to fit nicely into the picture, it may not exist at all.
A second type of answer is: I don't know and I admit that the physical picture of virtual particles hopping in and out of existence indeed nicely includes an explanation of Hawking radiation. However, I'm reminded of the texts that illustrate how many physical phenomena were considered to be satisfactorily 'explained' within, for instance, the aether model, even though it later turned out the model was fundamentally flawed and all of those physical interpretations were hogwash.
A third type of answer is: sometimes things simply aren't what they seem. For instance, we can describe phonons as if they're particles: they are described by the same statistics as 'real' particles are. However, they decidedly aren't particles. They aren't properties of individual particles either: they only exist in macroscopic, well-ordered, amounts of material. You could look at all individual atoms one by one and never arrive at phonons as partial explanations for their movements and properties. For that, you'd have to look at things at a different level.
