Link at the correct address so there's 1MB between the end of the application image and the bottom of the stack. Allow the linker to load the shared libraries (or just link statically) and then use mmap to fill the rest of the virtual address space. You can implement this in multiple ways.
It's a bit of a weird exercise, you'd generally just calculate the maximum space used by the algorithm, maybe account for stack frame size if recursion is used.
There are problems on many accounts: you can forget all the dance around the heap, — malloc can (it sometimes will) just use mmap; catching malloc is not enough, you can directly mmap, or, and there's perfectly a fine stack you can utilize for 1M; you cannot restrict memory space because necessary libraries take considerate to 1M amount.
1M is too small for what? For sorting? Dunno, will see next time when try to implement external merge sort :-). Too small for the whole program? Of course, it won't load runtime - I've shown it in container and qemu parts.
Where did the addresses used in the linker (.text and data) script come from? I work mostly in bare-metal embedded world and I use linker scripts but in my world the micro architecture defines where FLASH and RAM are located. How does it work when the application runs inside an OS?
It's a bit of a weird exercise, you'd generally just calculate the maximum space used by the algorithm, maybe account for stack frame size if recursion is used.