This kind of sounds like someone is abusing perf cores and high priority threading in your stack. iirc, on MacOS audio workgroup threads are supposed to be scheduled with the highest (real time) priority on p cores, which shouldn't have issues under load, unless someone else is trying to compete at the same priority.
There is some discussion online on whether this happens when you have a Rosetta app running in the background somewhere (say a util you got via Homebrew, for example).
Even when I remove all "Intel" type apps in activity monitor, I still experience the issue though.
