I was looking at GPU graphs and neglected my physical RAM. This machine only has 32G and I didn't notice I was hitting a ceiling on memory allocations too - I took the error message about GPU memory allocations at face value.

I bumped my system commit cap (increased paging file size) by 24G and now I can use all my VRAM.