A lot of Linux-based DSP solutions do this. You either purposely tell the kernel that you have less memory than you do at startup, reserving the rest for the DSP, or you load a kernel driver which can allocate contiguous memory. This driver never actually uses the memory, it just scoops it up before anyone else can.