That's fair, but different LLM behave differently, so you would have to redo your testing from scratch if you were to swap the model. I think that would be the primary problem.

Testing for LLMs is an evolving practice but you need to have tests even if you stick with one provider, otherwise you won't be able to swap out models safely within that provider either.