I agree -- it's silly to worry about this when there's so much more low-hanging fruit, like the endless JavaScript APIs that can access more and more hardware. For example, did you know that you can turn someone's webcam into their favicon? [1] I'll bet there are some exploits lurking around there. And that's even before we get into WASM/WASI. We're piling up complexity fast enough that I doubt Spectre will ever be worth exploiting at any scale in the real world. Maybe by a state-level actor against an individual, but that's not my threat model[2].
I wasn't saying it was an exploit, but just an example of how, as we give JavaScript more access to our hardware and browsers, there will be unanticipated interactions between features. Those surprises have often led to exploits in the past. All JavaScript should be considered untrusted (or malicious) code, and giving it new capabilities seems like an endless source of bugs and security problems. Those will probably be easier to exploit than timing attacks.
[1] https://twitter.com/davywtf/status/1119783380734836737
[2] https://www.usenix.org/system/files/1401_08-12_mickens.pdf