Why does it matter? They can certainly be compelled to use the HSM to decrypt data, even if they can't extract keys.
If you aren't doing client-side encryption and keeping the keys private, the server has access one way or another.
Why does it matter? They can certainly be compelled to use the HSM to decrypt data, even if they can't extract keys.