I doubt you would give it access to arbitrary memory so to actually get the data they would need an exploit of the USB connections and while it increases the attack surface I don't think it increases it unreasonably. The larger concern to me would be someone could be capturing the keyboard and monitor you connect to and could get most of your information through that.
USB3 alternate modes happily allow DMA with host devices. If the device supports Thunderbolt as an alternate mode, that gets...tricky. And given that it's pretty likely that this posited setup would encourage the use of external graphics processing hardware...red flags, Batman.
