Presumably in the case of a public terminal, you would have to trust that your keyboard inputs and screen outputs weren't being recorded. But then you also have to trust that the VM container wasn't being malicious too, which the VM itself wouldn't be able to protect against...

Even if there's a keylogger on the machine, it should not be enough to access your accounts without you present. The trick is the USB key should use a crypto method so that when it's unplugged, the public computer can't access your data anymore.