safetensors can mmap too because the tensor data are just offsets and you are free to align to whatever you want.
It is hard to keep metadata minimal, and before long, you will start to have many different "atom"s and end-up with things that mov supports but mp4 doesn't etc etc. (mov format is generally well-defined and easy-to-parse, but being a binary format, you have to write your parser etc is not a pleasant experience).
If you just want minimal dependency, flatbuffers, capnproto, json are all well-supported on many platforms.
mmap() requires that you map at page aligned intervals which must be congruent with the file offset. You can't just round down because some gpus like metal require that the data pointers themselves be page aligned too.
Yeah, safetensors separates metadata and tensor data. The metadata is an offset reference to the tensor data that you are free to define yourselves. In that way, you can create files in safetensors format but the tensor data itself is paged aligned offsets.
It is hard to keep metadata minimal, and before long, you will start to have many different "atom"s and end-up with things that mov supports but mp4 doesn't etc etc. (mov format is generally well-defined and easy-to-parse, but being a binary format, you have to write your parser etc is not a pleasant experience).
If you just want minimal dependency, flatbuffers, capnproto, json are all well-supported on many platforms.