FWIW we have a live product for all users against gpt-3.5-turbo and it's largely fine: https://www.honeycomb.io/blog/improving-llms-production-obse...

In our own tracking, the P99 isn't exactly great, but this is groundbreaking tech we're dealing with here, and our dissatisfaction with the high end of latency is well worth the value we get in our product: https://twitter.com/_cartermp/status/1674092825053655040/