I think you would need to be careful to not put sensitive information into those terminals but that's not too hard to avoid in general. Generally I'm not doing anything that sensitive on my computer and with how my computer is configured I almost never have to actually type passwords.
Except for most people (the kind who don't read Hacker News) their phone is their primary computing device and likely the only one they own. There already is loads of sensitive data on these peoples' devices. The only difference is they aren't yet plugging it into random hardware.
I doubt you would give it access to arbitrary memory so to actually get the data they would need an exploit of the USB connections and while it increases the attack surface I don't think it increases it unreasonably. The larger concern to me would be someone could be capturing the keyboard and monitor you connect to and could get most of your information through that.
USB3 alternate modes happily allow DMA with host devices. If the device supports Thunderbolt as an alternate mode, that gets...tricky. And given that it's pretty likely that this posited setup would encourage the use of external graphics processing hardware...red flags, Batman.
