This is probably the minimum you should do. I remember a time when IPMI on (some?) Supermicro boards was really really insecure. From the IPMI client, you could set the encryption mode to "0" ("null encryption" or something, I dunno it's been years)- setting it allowed you to bypass the password completely. Assume if you can touch the IPMI, the system is yours.
IPMI is still a festering cesspool no matter which vendor. Assume that layer 3 access to the IPMI grants you unrestricted persistent code execution on the managed system and design for with it in mind.
Restrict access as tightly as possible. If you only need to power up/down/reset the system and access the serial console most IPMI implementations expose that via SSH. A small SSH proxy that exposes only those features would be a good investment. e.g. `ssh bastion [status|up|down|cycle|reset|console] <server>`. You could probably write it in <100 lines as SSH forced command. Deploy the SSH client to a different VRF (aka network namespace/vnet jail/rdomain) than the SSH server to make it harder to leak traffic from the bastion.
