You can go to the control centre or whatever it's called when you pull down from the battery side of the screen and tap and hold the volume widget, you can then drag your finger up/down to change the volume in small increments. I have the same issue as you when I'm listening to stuff at night.