That actually really surprises me. Is this just a limitation of the major implementations, or is there something fundamental about the design that slows it down? (I'm not familiar with the details of the protocol)

Protocol itself is ok, the problem is that it is tunneled inside TCP, so on top of TCP SYN/ACKs you have HTTPs certificate exchange and then cypher negotiation just to establish connection. Once established the cypher becomes another slowdown.

Would it make things unbearable? I've accessed https websites and I didn't even know they were slower up until now. Am I missing something?

That makes sense. Thanks.