On the Pi Zero and Pi CM (maybe also others) you don't even need an SD card to boot it. You can boot it via rpi-boot https://github.com/raspberrypi/usbboot
so no need for qemu for testing. You can just test it on real hardware in no time.
Without qemu you have to implement serial interfaces early in order to debug your own OS. Qemu is a huge benefit early on, and provides a sane environment for deterministic development as opposed to potentially quirky hardware.
Usually hobbyist OS projects don't start directly on hardware. Many of them never reach the point of running on hardware.
I don't know about qemu for a PI/arm in general, but you do have to be careful on x86, because segmentation limits aren't checked by default, and you can be lulled into doing things that don't work on hardware (I messed up secondary processor starting), and if you don't frequently test on hardware, it's easy to forget things. (See also homebrew games that only work on emulators)
But, you can certainly wait to start on hardware until you've got it started on qemu. Debuggability is a lot better unless you've got specialized equipment for your hardware setup
On Arm a couple of only-on-hardware pitfalls are (a) cache maintenance -- QEMU doesn't model caches so won't notice if you forget to clean the dcache and flush the icache before executing code you just wrote or modified and (b) synchronization/barriers -- QEMU doesn't reorder memory accesses and generally makes writes to system registers take effect immediately, so won't notice if you forget necessary barrier insns.
The Pi 3 and 4 can also boot from TFTP with an NFS root. This also allows you to switch the OS your Pi is booting just by renaming a symlink on your server. All my home Pis boot that way. As a bonus, you never need to worry about an SD card going bad.