You should really try out Yocto. There is a lot to learn but you can configure it from super-minimal tiny distros to mega kitchen-sink-included distros.
Everything is customizable, every file, every compiler flag and option of every package, with easy overrides in your own layer which only 'appends' to the original instructions, or adding and removing packages as you see fit in the base install.
What I dont underatand (yet) is in the example you provided you gave "qemuarm64" as the BSP. But the NetBSD example johnklos gave is not just for QEMU. I dont see an option for the RPi in poky's local.conf. NetBSD of already has configs for bootable RPi in its tree. This illustrates my point about automated downloads. Yocto requires, among other things, wget. For building images. This implies there is more to be downloaded before we can begin. With NetBSD, network access is not required to build images.
Everything is customizable, every file, every compiler flag and option of every package, with easy overrides in your own layer which only 'appends' to the original instructions, or adding and removing packages as you see fit in the base install.