MacOS just seems to have a lot of stability problems related to peripherals. I don’t think it’s specific to M1. Even on my late-era Intel MBP, Bluetooth and USB often triggers a kernel panic.
When I got my first MacBook Pro, there was actually bug in iPhone USB Tethering that caused a guaranteed kernel panic if you tried to do it. Managed to crash my laptop a few times with my iPhone 3GS.