If they're injecting targeted ads in the stream, then the stream producer must be 'smart'. It's not much of a stretch for it to enforce playing out the segments at approximately realtime (or whatever speedup they want to allow), and to force the advert segments to play out before anything past them. Some sidechannel could be used to inform the client about what's going on and produce a sensible playhead position.

It seems inevitable that this is the end game, and I don't really see viable ways around it for realtime playback. For offline playback, yeah, presumably that sidechannel includes enough information to cut out the ads.