At my previous company, we had a lot of tests. They were a mix of C and Python. Running all of them on a single machine took on the order of an hour or more. Even just limiting the tests run to those that could theoretically be affected by your change could take minutes or even tens of minutes.
We ended up building a shared cluster of ~1000 cores that was available to all developers, and that was used by CI. This changed our developers' workflows quite a bit. It was now possible to run large amounts of tests regularly: like every few minutes instead of a once or twice a day. This in turn encouraged developers to write more tests and do more test-driven development.
On top of that, having the cluster available provided other benefits. If a test was flakey, it was easy to run it tens or even hundreds of thousands of times, making it easy to reproduce and identify the bug. We also occasionally did Monte Carlo simulations, and it was really handy to have a lot of cores available for general developer use.
I got used to working that way and I've missed it since I left that company. So this project is an attempt to make a more general-purpose implementation of that system. I hope others will find similar workflows that make them more productive using this system or something like it.
Regarding the container-per-test idea. It really comes about because it's the obvious way to package up jobs to submit them to a cluster. Plus, it makes tests reproducible for all developers in a project, and between developer machines and CI. Using Linux namespaces, the overhead of running tests in individual containers isn't much more than running tests in individual processes.
We ended up building a shared cluster of ~1000 cores that was available to all developers, and that was used by CI. This changed our developers' workflows quite a bit. It was now possible to run large amounts of tests regularly: like every few minutes instead of a once or twice a day. This in turn encouraged developers to write more tests and do more test-driven development.
On top of that, having the cluster available provided other benefits. If a test was flakey, it was easy to run it tens or even hundreds of thousands of times, making it easy to reproduce and identify the bug. We also occasionally did Monte Carlo simulations, and it was really handy to have a lot of cores available for general developer use.
I got used to working that way and I've missed it since I left that company. So this project is an attempt to make a more general-purpose implementation of that system. I hope others will find similar workflows that make them more productive using this system or something like it.
Regarding the container-per-test idea. It really comes about because it's the obvious way to package up jobs to submit them to a cluster. Plus, it makes tests reproducible for all developers in a project, and between developer machines and CI. Using Linux namespaces, the overhead of running tests in individual containers isn't much more than running tests in individual processes.