Pardon the nature of my question, but I'm really interested in what your experience has been so far building a database with Go? Has its runtime (the GC for example) posed any issues for you so far? Looking at other RDBMS's, languages with manual memory management like C or C++ seems to be the go-to choice, so what were the reasons you chose Go?
I'm quite frankly amazed that Go's runtime is able to support a database with such demanding capabilities as CockroachDB!
More technically, here's a somewhat random set of thoughts on the subject:
The Go GC is performant and predictable, unlike the JVM GC. We do have some very memory-allocation-conscious code patterns to minimize the performance impact of working in a garbage-collected language runtime, but in the end it's not as bad as you might expect if your expectations are coming from the JVM world.
Library support is good. To quote our CEO, "Most of us on the team have done extensive work with C++ and Java in the past. At Google, C++ was the standard for building infrastructure and there are a lot of good reasons for that. It's fast and predictable. It would be a good choice for Cockroach, except that in the world outside of Google, in open source land, the supporting libraries for C++ are either terrible, incredibly heavyweight, or non-existent. We didn't want to rebuild everything which you take for granted at Google from scratch. It turns out that Go has many of the necessary libraries, and they're straightforward and very well written."
Basically, if Google's internal C++ libraries, tooling, style guides (and the tooling to enforce them) were available externally, we might have gone with C++.
Some of us are fans of Rust, but Rust sadly did not exist in a stable state when CockroachDB started. I'm not sure we would pick Rust were we to start today (tooling is still a concern there), but it would certainly be part of the discussion.
The native support for concurrency in Go is a huge plus. We use thousands of goroutines in CockroachDB, and that's been a huge blessing.
I can answer any more specific questions if you have them.
Why do you think the Go GC is better than any the JVM options? From what I've seen, while the Go GC is well tuned for low latency, by picking the right JVM GC parameters you can on balance get a better throughput latency tradeoff. I'm just wondering if you have any reliable benchmarks or evidence to support what your saying? I don't use either language for work, so I think you might have better information than I.
I talk about this in the presentation I linked in another subthread (https://www.cockroachlabs.com/community/tech-talks/challenge...). The key to getting good performance out of any GC is to generate as little garbage as possible, and in our experience Go makes better use of stack allocation and value types keep many objects out of the garbage-collected heap. We've found that idiomatic go programs tend to produce less garbage than similar java programs, and in the presentation I discuss some tricks we use to get that even lower in critical paths. Admittedly, we're not JVM tuning wizards so maybe there's more that could have been done on the JVM side.
As I understand it, Java needs a complicated GC implementation because it produces, by design, a makes a huge amount of heap allocations -- lots of very short-lived little objects.
Much of Java's GC focus has been on correctly partitioning the heap so that long-lived objects can be less aggressively collected than short-lived ones. (An example of a challenging long-lived object is the entire set of classes used by a program, all of which need to available to the runtime for reflection. For many bigger apps, the class hierarchy alone takes up many megabytes of RAM!)
Go can make use of the stack to a much larger degree (structs and arrays can be passed by value), and so it can get by with a much less advanced GC. As a result, Go team's main focus has been on reducing pause times more than anything else.
We write our own GPU algorithms, Java native interface transpiler (eg: we generate JNI bindings) as well as our own memory management.
We've found the JVM to be more than suitable. Granted - we wrote our own tooling and had reasons we can't move (those customers are a neat thing most people don't think about :D)
I understand why you guys did go though. Congrats on pushing the limits of the runtime.
Thank you for the reply! You and the presentation video Ben posted covered pretty much all of my questions, and I'm going to keep an eye on the issue tracker regarding performance to see what interesting things you might run into and how you deal with them in Go!
This is more my personal opinion, and perhaps more revealing my ignorance on the existing equivalent tools in the Rust ecosystem, but here is a list of some of the Go tools we use when developing CockroachDB:
1. gofmt and goimports really helps enforce a single uniform style. We don't really care what the style is, as long as it's consistent across our 30 engineers and 200k lines of code. We have hand-rolled more Cockroach-specific linters on top of this as well, but we could do that for Rust too.
2. go tool pprof is a great profiler. Being able to quickly dig into allocations, cpu usage, etc. is great, and we do so regularly. As a result, the overhead of the GC is minimized, since we can rapidly identify and mitigate the allocation overhead with the application of a few known patterns.
Now I don't know what the state of the art of rust profiling is, but if we were to litigate Rust vs Go starting CockroachDB from scratch today, we'd probably pay close attention to what the answer is here. The Xooglers on this team have a tonne of C++ experience, and were very happy with C++ profiling tools, and thought the Go profiler matched up to the best tools they had used previously. If there is a Rust equivalent, this isn't a problem.
3. Consistency of code (in both style, but also patterns used) across third party libraries is a concern. The existence of a single toolchain that enforces a single style in Go really helps keep the whole ecosystem healthy here. Even if tools exist for Rust, if they aren't universally used, that is not as powerful.
I honestly think that Rust would probably be a close contender if we litigated this question today. The TiDB folks use Rust for their KV side, but Go for their query engine, which is an interesting mix. If faced with this decision today, I personally would push for Rust; I'm not a fan of the Go type system's various limitations, which we are running into particularly as we write a more sophisticated query optimizer that has to do more classical programming languages reasoning. But I am one of the most junior engineers on the CockroachDB team, so I'm not sure I would prevail in this fight! :)
Thank you so much for the thorough answer! This is stuff we're always working on, so it's helpful to know about this stuff. Since you're not actively looking, I won't go into all the details, but if you ever are in the future, happy to give you a rundown of the state of the art whenever that is :)
Overall we've been happy with the choice. The GC is sometimes a performance issue, but it's manageable (and Go gives you better tools to limit the cost of GC than many other garbage-collected languages)
We have started parallelizing our tests with the new subtest feature: leaktest in the top-level test, t.Parallel in the subtests. This means we only check for leaks in between batches of parallel subtests. This works OK for us for now since our slowest "test" is really a huge data-driven test suite, and that's the only place we're currently parallelizing, although it would be better if we could parallelize more of our tests.
I'm quite frankly amazed that Go's runtime is able to support a database with such demanding capabilities as CockroachDB!