> and since you need to repartition to create standard Linux partitions anyway
If it's possible to do this in APFS (not sure), the installer could do something similar for APFS to what WUBI does for NTFS: fallocate(2) a contiguous-disk-block single-extent file inside the APFS volume, write the rootfs into it, and then configure the bootloader entry's kernel params with the APFS container+volume and the path to the file within the container.
In this setup, linux-apfs would then only need to work well enough to expose that file's single extent's raw-block-device offset during early boot; then the initramfs script could unmount the linux-apfs volume and mount the ext4 rootfs directly from the block device at that offset.
Yes, it's almost the same as resizing the disk smaller — but it's at least a less-"permanent" change, in the sense that deleting the rootfs file from macOS would give the space back, without needing to know to re-grow your macOS APFS container.
I wrote a tool to do this many years ago on the Xbox 1 (the original) to avoid having to use loopmounted files, but it's not the kind of thing you want to rely on for a modern filesystem like APFS. There's no way to guarantee that the OS won't mess with your data, move it elsewhere, etc (and if it doesn't today, it might in tomorrow's macOS update). I would never trust this approach enough to recommend it for our users, to be honest.
> There's no way to guarantee that the OS won't mess with your data, move it elsewhere, etc
I mean, "moving it elsewhere" is why you'd call into linux-apfs on every boot — to find out where the rootfs backing file's extent is living today. It might move around between boots, but it won't move while macOS isn't even running.
And this approach presumes that there exists an explicit file attribute for keeping a file single-extent / contiguous-on-disk (as far as the host-NVMe interface is concerned, at least.) Such an attribute does exist on NTFS, which is why WUBI continued to work there. I've never heard about such an attribute existing on APFS, which is why I said I'm not sure whether APFS "supports" this.
Usually there is such an attribute in most-any filesystem, though — it exists to be used with swap files, so that kernels can directly mmap(2) the disk-block range underlying the swap file rather than having every read/write go through the filesystem layer.
I know that macOS uses an APFS volume for swap (the "VM" volume), which is kind of interesting. Maybe in APFS, there isn't a per-file attribute for contiguous allocation, but rather a per-volume one?
In that case, it might be possible for the Linux rootfs to be an APFS volume, in the same way that the macOS VM volume is an APFS volume.
If it's possible to do this in APFS (not sure), the installer could do something similar for APFS to what WUBI does for NTFS: fallocate(2) a contiguous-disk-block single-extent file inside the APFS volume, write the rootfs into it, and then configure the bootloader entry's kernel params with the APFS container+volume and the path to the file within the container.
In this setup, linux-apfs would then only need to work well enough to expose that file's single extent's raw-block-device offset during early boot; then the initramfs script could unmount the linux-apfs volume and mount the ext4 rootfs directly from the block device at that offset.
Yes, it's almost the same as resizing the disk smaller — but it's at least a less-"permanent" change, in the sense that deleting the rootfs file from macOS would give the space back, without needing to know to re-grow your macOS APFS container.