Preallocate them and don't increase their size one at the time. And in them use indexes to positions instead of pointers. Then GC won't make you problems anymore. It would be used but it won't be a bottleneck as it would simply get much less but bigger chunks of memory and much less pointers to worry about.
http://dlang.org/struct.html
http://dlang.org/arrays.html
Preallocate them and don't increase their size one at the time. And in them use indexes to positions instead of pointers. Then GC won't make you problems anymore. It would be used but it won't be a bottleneck as it would simply get much less but bigger chunks of memory and much less pointers to worry about.