If they give you a valid subpoena, then yeah, you have to give them the video in a format they can use, you can't give them a bunch of encrypted video files and say "Good luck trying to watch it".

If you refuse to hand it over and they get a warrant and seize your NVR, then I'm not sure if they can compel you to decrypt it, but you may already be in jail for contempt and probably are spending hundreds of thousands of dollars in legal fees at this point, so most people would cave long before this and just hand it over.