Can't find the article where I first read it (something like "Queuing theory for software engineers") but average latency increases as, IIRC, serving time ÷ (1 - utilization). Get half as close to 100% utilization, and you double your average latency. A system with 87.5% utilization has double the latency as at 75%. At 100% it's infinity (averaged over infinite time - on shorter timescales it's an unpredictable scale-free random walk).
This is fundamental - the closer utilization is to 100%, the higher the chance a newly arriving work item has to wait for one that's already running, and several already in the queue. What's astonishing is how steep that curve is. At 95% utilization the average queue length is 20 tasks. At 99% it's 100 tasks. At 99.9% it's 1000 asks. If you find yourself at 98% utilization, you should not think "nice - in fully utilizing the server I paid for" - you should buy another server and lower it to 49%. (Or optimize the code more)
One way to deal with this is to have separate low-latency and high-latency queues. You can then run low latency tasks at say 50% utilization and fill up idle time with high latency tasks. Presuming and you actually want the HL tasks to ever get done, you can't guarantee 100% utilization, but you can get arbitrarily close as long as there's high-latency work to do. I have no idea whether this is something Kubernetes can do. You can of course have more than two priority levels.
This applies everywhere there's a queue, which is basically everywhere there's s contended resource. Hyperscalers know this. It's even been theorized that S3 Glacier is just the super low priority disk access queue on regular AWS servers (but Amazon won't tell us).
This is fundamental - the closer utilization is to 100%, the higher the chance a newly arriving work item has to wait for one that's already running, and several already in the queue. What's astonishing is how steep that curve is. At 95% utilization the average queue length is 20 tasks. At 99% it's 100 tasks. At 99.9% it's 1000 asks. If you find yourself at 98% utilization, you should not think "nice - in fully utilizing the server I paid for" - you should buy another server and lower it to 49%. (Or optimize the code more)
One way to deal with this is to have separate low-latency and high-latency queues. You can then run low latency tasks at say 50% utilization and fill up idle time with high latency tasks. Presuming and you actually want the HL tasks to ever get done, you can't guarantee 100% utilization, but you can get arbitrarily close as long as there's high-latency work to do. I have no idea whether this is something Kubernetes can do. You can of course have more than two priority levels.
This applies everywhere there's a queue, which is basically everywhere there's s contended resource. Hyperscalers know this. It's even been theorized that S3 Glacier is just the super low priority disk access queue on regular AWS servers (but Amazon won't tell us).