The JavaScript of shady adverts that sometimes pop through can also occur on no. Shady websites. So you are not entirely safe by only browsing safe sites.
That doesn't answer the question, of course; it's just a sales pitch for the proverbial tiger-proof rock. To reiterate: how long does s/he have to spend on a shady site before a successful SPECTRE exploit takes place?
If this were actually happening in the real world, maybe we'd know.
I guess now the question is, how much time to I have to spend on that site before it can get my private ssh keys?