I've done this, but it can be hard. Some designers can be very protective of their design, and you have to fight entire battles for something like "this white header text on a light-grey background image is almost unreadable, so I tweaked the colour a bit and added a text shadow" and other really basic no-brainers like that.
I don't even have very strong opinions about design; anything is fine with me. The only part I really do have a strong opinion on is "it should be readable", and enforcing only that has been very tiresome at times.
I totally agree it's not always easy. But again I think with something like this it's about savvy communication rather than enforcement. How do you articulate a challenge to a designer that respects their role in the process? Contrast is easy, there's defined standards you can defer to. But for sure, it gets tricky where it's just a difference of opinion or priorities.
Regardless, just going ahead and changing the design is probably the nuclear option. It's only going to start a war. Put the shoe on the other foot: consider same designer dropping a 2008 era jQuery snippet into your carefully constructed page to resolve an issue in the build. Not fun for anyone.
Edit: I should add, I'm basing all of this on personal experience. Appreciate some situations just suck and you have to get on with it.
I don't even have very strong opinions about design; anything is fine with me. The only part I really do have a strong opinion on is "it should be readable", and enforcing only that has been very tiresome at times.