Uh, with Yocto at least, it's also pretty simple for a basic bootable disk image/packages... downloading all the source and compiling the cross toolchain along the way...

    git clone git://git.yoctoproject.org/poky
    cd poky
    git checkout -b hardknott -t origin/hardknott
    source oe-init-build-env
    MACHINE=qemuarm64 bitbake core-image-sato

But after the first build you may want more - bringing in other functional layers for different features (maybe you want docker?), support for different boards (same OS distro for x86_64 and ARM, or your own custom board... plus it can build a whole SDK for you for local development/debugging?), customization of target files, pulling from updated upstreams, support in deployment and throughout the device lifecycle, custom package feeds, etc.

There's a lot going for Yocto, it has a deep learning curve but does a pretty good job of taming the complexity of managing an entire OS distro.

Cannot offer any advice to the parent^1 however the build.sh instructions he provides bring to mind a general observation: By comparison, running buildroot's script will initiate downloads of files. This is a key distinction I see with Linux/GNU distributions (cf. the Linux kernel) in general (there are exceptions). Linux distribution maintainers try to automate everything. The idea of the user downloading the source tree for the entire distribution manually is not contemplated. Another example: When NetBSD is first installed, generally no networking programs are enabled by default. Its expected the user will turn things on, if she wants them on. By contrast, when a popular Linux distribution is installed, there are usually networking programs enabled by default to listen on the network. I use both Linux and NetBSD. I am not invoking a Linux v BSD debate nor arguing for one's approach over the other's, I am just pointing out how they are philosophically different.

1 The "NetBSD way" would be to study the NetBSD source tree and then modify certain files to suit one's needs. For example, I like to create custom crunched binaries; this requires editing a list. By comparsion, with Linux, I have never read any advice to study the Debian source tree and then modify files to suit one's needs. Rather, the advice one finds directs readers to Yocto, buildroot, etc.

I understand and appreciate that. But the distinction between manually downloading or automatically downloading isn't important to me.

Perhaps buildroot does the closest thing. I'll have to give it a go.

Maybe Gentoo is the closest thing.

You should really try out Yocto. There is a lot to learn but you can configure it from super-minimal tiny distros to mega kitchen-sink-included distros.

Everything is customizable, every file, every compiler flag and option of every package, with easy overrides in your own layer which only 'appends' to the original instructions, or adding and removing packages as you see fit in the base install.

What I dont underatand (yet) is in the example you provided you gave "qemuarm64" as the BSP. But the NetBSD example johnklos gave is not just for QEMU. I dont see an option for the RPi in poky's local.conf. NetBSD of already has configs for bootable RPi in its tree. This illustrates my point about automated downloads. Yocto requires, among other things, wget. For building images. This implies there is more to be downloaded before we can begin. With NetBSD, network access is not required to build images.

Maybe nix/guix, but most distros are going to have difficulty because they aren't a single thing to build, they're a collection of separate packages that you'd have to build individually.

I would use buildroot for that. It's very convenient even to just build random kernels for random targets