I password protect my key for the sole threat model of me physically losing my device. I am aware that all other threat models that involve someone taking remote control of my device are not fully protected against, but it at least requires significantly more effort on their part versus just doing a scan for private keys on the file system.