Experience with C/C++ is exactly why people tend to value compaction. I've absolutely encountered servers and other long-lived apps written in C++ that suffer from heap fragmentation, and required serious attention from skilled developers to try and fix things (sometimes by adding or removing fields from structures). It can be a huge time sink because the code isn't actually buggy and the problem is often not easily localised to one section of code. It's not common that you encounter big firefighting efforts though, because often for a server it's easier to just restart it in this sort of situation.
As an example, Windows has a special malloc called the "low fragmentation heap" specifically to help fight this kind of problem - if fragmentation was never an issue in practice, such a feature would not exist.
CMS was never designed for 100GB+ heaps so I am not surprised your experience was poor. G1 can handle such heaps although the Intel/HBase presentation suggested aiming for more like 100msec pause times is reasonable there.
The main thing I'd guess you have to watch out for with huge Go heaps is how long it takes to complete a collection. If it's really scanning the entire heap in each collection then I'd guess you can outrun the GC quite easily if your allocation rate is high.
It's true heaps can be a pain with C/C++. 64-bit is pretty ok, it's rare to have any issues.
32-bit is painful and messy. If possible, one thing that may help is to allocate large (virtual memory wise) objects once in the beginning of a new process and have separate heaps for different threads / purposes. Not only heap fragmentation can be issue, but also virtual memory fragmentation. Latter is usually what turns out to be fatal. One way to mitigate issues with multiple large allocations is to change memory mapping as needed... Yeah, it can get messy.
64-bit systems are way easier. Large allocations can be handled by allocating page size blocks of memory from OS (VirtualAlloc / mmap). OS can move and compact physical memory just fine. At most you'll end up with holes in the virtual memory mappings, but it's not a real issue with 64 bit systems.
Small allocations with some allocator that is smart enough to group allocations by 2^n size (or do some other smarter tricks to practically eliminate fragmentation).
Other ways are to use arenas or multiple heaps. For example per thread or per object.
There are also compactible heaps. You just need to lock the memory object before use to get a pointer to it and unlock when you're done. The heap manager is free to move the memory block as it pleases, because no one is allowed to have a pointer to the block. Harder to use, yes, but hey, no fragmentation!
Yeah, Java is better in some ways for being able to compact memory always. That said, I've also cursed it to hell for ending up in practically infinite gc loop when used memory is nearing maximum heap size.
Well, I can only say that your experience is different than mine. I worked with C++ for 10 years, on mostly server side software, and never encountered a problem that we traced back to heap fragmentation. I'm not sure exactly why this was the case... perhaps the use of object pools prevented it, or perhaps it just isn't that big of a problem on modern 64 bit servers.
At Cloudera, we still mostly use CMS because the version of G1 shipped in JDK6 wasn't considered mature, and we only recently upgraded to JDK7. We are currently looking into defaulting to G1, but it will take time to feel confident about that. G1 is not a silver bullet anyway. You can still get multi-minute pauses with heaps bigger than 100GB. A stop-the-world GC is still lurking in wait if certain conditions are met, and some workloads always trigger it... like starting the HDFS NameNode.
Ouch, not even upgrading to JDK8? That's not really a new JVM anymore now.
G1 has improved a lot over time. What I've been writing was based on the assumption of using the latest version of it.
Yes, full stop-the-world GCs are painful, but they'll be painful in any GC. If Go runs out of memory entirely then I assume they have to do the same thing.
As an example, Windows has a special malloc called the "low fragmentation heap" specifically to help fight this kind of problem - if fragmentation was never an issue in practice, such a feature would not exist.
CMS was never designed for 100GB+ heaps so I am not surprised your experience was poor. G1 can handle such heaps although the Intel/HBase presentation suggested aiming for more like 100msec pause times is reasonable there.
The main thing I'd guess you have to watch out for with huge Go heaps is how long it takes to complete a collection. If it's really scanning the entire heap in each collection then I'd guess you can outrun the GC quite easily if your allocation rate is high.