So if objects are just perturbation in a field, and field is a mathematical construct, just like the concept of a number, or a line or a circle, then isn't it meaning less to look for a start for the existence of a field, and hence the whole universe?
No, for a couple of reasons. It's a mathematical construct used to describe observed phenomena, the phenomena itself isn't necessarily that similar to the construct. Further, it may be a terrible description.
A sinus is a field, deductively reduced to lower dimensions, e.g. a slice cut of a sound wave. Waves on the sea form a wave field, figuratively, and it extends to the bottom of the sea. Whereas a quantum field is a consequence of the Schroedinger Wave equation positing that the potential to observe a particle is distributed in space. As far as I know that's purely virtual, because observation of the particle collapses the potential. The crux is, the observation is not exact; The location of the particle is a probability distribution over a field as well. So ... it's fields all the way down. I'm not an expert, just stipulating.