They aren't, because those discussions are all related to link-time stuff (if I update foo.h and bar.c that depends on foo.h, I can do so atomically, because those are built into the same artifact).
As soon as you discuss network traffic (or really anything that crosses an RPC boundary), things get more complicated, but none of that has anything to do with a monorepo, and monorepos still sometimes simplify things.
So there's a few tools that are common: feature flags, 3 stage-rollouts, and probably more that are relevant, but let's dive into those first two.
Feature "flags" are often dynamically scoped and runtime-modifiable. You can change a feature flag via an RPC, without restarting the binary running. This is done by having something along the lines of
if (condition_that_enables_feature()) {
do_feature_thing()
} else {
do_old_thing()
}
A/B testing tools like optimizely and co provide this, and there are generic frameworks too. `condition_that_enables_feature()`, here is a dynamic function that may change value based on the time of day, the user, etc. (think something like `hash(user.username).startswith(b'00') and user.locale == 'EN'`). The tools allow you to modify these conditions and push and change the conditions all without restarts. That's how you get per-user opt-in to certain behaviors. Fundamentally, you might have an app that is capable of serving two completely different UIs for the same user journey.
Then you have "3-phase" updates. In this process, you have a client and server. You want to update them to use "v2" of some api, that's totally incompatible with v1. You start by updating the server to accept requests in either v1 or v2 format. That's stage one. Then you update the clients to sent requests in v2 format. That's stage two. Then you remove all support for v1. That's stage three.
When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine.
Note again that all of this happens whether or not you use a monorepo.
> When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine.
But the supposed benefit of the monorepo was that you could update everything from v1 to v2 in one atomic commit, and that hasn't actually happened. You have to make the changes in exactly the same way as if your client and server were in separate repositories; indeed it may be harder to get your integration tests to test every combination that will actually be deployed. So when there's a network boundary in between - or more to the point a deployment boundary in between - you end up with the costs of the monorepo but not the benefit.
> But the supposed benefit of the monorepo was that you could update everything from v1 to v2 in one atomic commit
No, a benefit of a monorepo is that you can update your build-time dependencies atomically. No one has claimed that a monorepo lets you magically do atomic RPC API migrations.
> You have to make the changes in exactly the same way as if your client and server were in separate repositories; indeed it may be harder to get your integration tests to test every combination that will actually be deployed.
It won't though: if you cover both code paths at each stage of the three stages, you'll cover all possibilities (its also not clear that you need integration tests to cover this, you can fake the client or server and test against both versions and be fine). This is the same if you've got multi or mono-repo, although with a monorepo there's a decent argument that maintaining integration tests is easier, as you can build both the client and server from within the repo.
> So when there's a network boundary in between - or more to the point a deployment boundary in between - you end up with the costs of the monorepo but not the benefit.
This presumes that there are some costs. I'd counter that there's no real difference in this situation, and so when choosing mono- vs. multi-repo, this kind of situation shouldn't influence your decision, because it doesn't matter. You should base your decision on the cases where the choice has an impact.
> No one has claimed that a monorepo lets you magically do atomic RPC API migrations.
Plenty of people have claimed and continue to claim that a monorepo lets you magically do atomic API migrations, and skate over how the fact that many APIs go over RPC these days.
> It won't though: if you cover both code paths at each stage of the three stages, you'll cover all possibilities (its also not clear that you need integration tests to cover this, you can fake the client or server and test against both versions and be fine). This is the same if you've got multi or mono-repo, although with a monorepo there's a decent argument that maintaining integration tests is easier, as you can build both the client and server from within the repo.
This relies on assuming that you perfectly maintain the previous codepath as-is, otherwise your tests won't cover what's actually going to run. In a multirepo setup the idea that you run your integration tests against version X of server code and version Y of client code is already something you're conscious of (and you absolutely do need to integration test your real client with your real server, otherwise you won't uncover your false assumptions about the behaviour of one or the other), so testing the new release of the server with the unmodified old release of the client is something you can do very naturally, whereas in a monorepo you have to make a very deliberate effort to test anything other than current client against current server.
> Plenty of people have claimed and continue to claim that a monorepo lets you magically do atomic API migrations, and skate over how the fact that many APIs go over RPC these days.
Nobody that knows what they are talking about makes this claim. People making this claim are confused and misunderstand. Just because there are a group of people who don't understand the thing they are advocating for, doesn't make the thing any better or worse for whatever purpose they are espousing it for. It just means that they don't know what they are talking about (see also "even a broken clock is right twice a day").
To wit: if I'm using a monorepo to produce my back-end service and the client libraries I ship to my customers, who must update their deployed version themselves, how do these proponents of "magic monorepo atomic RPC rollouts" propose that works exactly? Clearly, it doesn't. Updating your own environments is really no different. All deployments are eventually consistent. C'est la vie.
> whereas in a monorepo you have to make a very deliberate effort
When dealing with API or data model changes, you have to deal with the same, deliberate effort to schedule/stage the release of your changes regardless of approach that is used.
> Nobody that knows what they are talking about makes this claim. People making this claim are confused and misunderstand. Just because there are a group of people who don't understand the thing they are advocating for, doesn't make the thing any better or worse for whatever purpose they are espousing it for. It just means that they don't know what they are talking about (see also "even a broken clock is right twice a day").
Right, but, frankly, that's the statement that a lot of hype and even adoption of monorepos is being driven by people who don't know what they're talking about, for reasons that are nonsense.
> When dealing with API or data model changes, you have to deal with the same, deliberate effort to schedule/stage the release of your changes regardless of approach that is used.
If you align the repository - particularly, the level of granularity at which you tag and at which you check out - with the level of granularity at which you deploy, that can simplify your working practices and make some important details more visible. If your day-to-day development workflow is that you check out version 506 of "everything", than naturally nudges you to thinking of this global monotonic version which is misleading when it comes to actually deploy things. If it's natural to checkout version 523 of foo and version 416 of bar in your day-to-day work, that can help you align more closely with a world where version 523 of foo and version 416 of bar is what you actually have deployed on your servers.
If you're using anything approaching continuous integration/deployment, the value of checking out version 523 of foo and 416 of bar is low, because by the time you test and submit your change, one or the other will have changed.
When you have relatively small services, continuously updated, there's not a need to do the sort of precise cross-version testing you're advocating for, as long as you maintain some level of per-feature forward and backward compatibility across your api surfaces, which is far easier than doing cross-version integration tests for every change.
Put another way, you should decouple yourself from caring about particular versions and think more about which features are supported by what's currently deployed. The question you ask shouldn't be "is version X in production", it's "is feature Foo safe to depend on". Unless something is wrong, feature Foo should work just as well in version X and X+1 and X+2, so caring about precisely which you're using is too low level.
Well, that's a matter of opinion, and by no means the only way to run a development workflow.
I find testing the new version of the server against the actual version of the client that is deployed in production, or vice versa, extremely valuable for avoiding issues.
So let's say you make a change on the client today, how do you ensure that the version of the server you test against today is the same version of the server that you deploy against tomorrow (keep in mind one of your concerns was that at any given time you'd have multiple versions in prod at once)?
You have as part of your deployment workflow that you run the integration tests with exactly the version you're about to deploy. And you have a process step to ensure that you don't deploy both the server and client at the same time. That's quite doable and worthwhile once you step back from the idea that everything needs to be continuously deployed, IME.
> This relies on assuming that you perfectly maintain the previous codepath as-is, otherwise your tests won't cover what's actually going to run.
At each commit, your integration test covers the code path that runs at that CL. So for the client, when you switch from the V1 API to the V2 API, you also swap the integration test. Sure you're no longer testing V1, but that's fine, because the artifact being deployed no longer uses the V1 api.
Since the server needs to support both versions, then you'd probably want, for some period of time, tests exercising both paths, but its not challenging to do that. You don't touch the existing tests, and solely add a new test case exercising the new path.
There's no need for explicit cross-version testing with particular versions, because the way you stage the upgrades ensures all the cross-compatibility you need.
> so testing the new release of the server with the unmodified old release of the client is something you can do very naturally
No, this happens perfectly naturally in the monorepo case. When you upgrade the server to support both APIs, you haven't yet touched the client, so you're naturally testing it on the unmodified old client behavior. I'll reiterate, by doing the upgrade in the three-phase way I mentioned, this isn't a problem, and I'm not sure why you think it is.
> Since the server needs to support both versions, then you'd probably want, for some period of time, tests exercising both paths, but its not challenging to do that. You don't touch the existing tests, and solely add a new test case exercising the new path.
But that's a manual, deliberate process. You're effectively recreating what your VCS does for you by having explicit separate codepaths for each API version.
Can you explain to me how you'd avoid having said separate codepaths and tests in a multi-repo situation?
You'd still need to update the server to support both API versions during the transition period. We agree the change can't be made atomically in a monorepo, but it also can't be done atomically in a multi-repo environment.
So, since you have two code paths in prod, you also need them in your tests. It's a manual process either way, and completely independent of your vcs.
Say you currently have server version 5 and client version 3 deployed. You've already tested those. Now you make a change to the shared protocol, and you decide you're going to deploy the server first and then the client (or vice versa). So you bump the server version to 6 in your integration tests, check if they pass, then deploy server version 6 (progressively, but that's fine, you've tested both the combination of client 3 and server 5 and client 3 and server 6). Then you bump the client version to 4 in your integration tests, check if they pass, and deploy client version 4.
Importantly, this is the completely natural way to do every release and deploy; you'd have to go out of your way not to do this. Even if you thought version 6 didn't contain any protocol changes, you'd still deploy it exactly the same way. So you end up testing the real combination of code that's about to get deployed, before you deploy it, every time.
> monorepo lets you magically do atomic API migrations
I'd argue that there are many more APIs which just don't go over network. With monorepo you can do cross-component refactor very easily, this is especially handy when you are working on a widely-used library.
Imagine that you are working on your language's standard library and you want to fix a small API mis-design.
I think having a single repo for a library that's versioned and released as a single component makes a lot of sense, but combining separate services that are deployed separately into a single repository is not a good idea. The latter is what people generally mean by "monorepo" AFAICS.
Well, you said language standard library, so that would mean keeping every program written in that language in a single repo, which seems obviously absurd.
As soon as you discuss network traffic (or really anything that crosses an RPC boundary), things get more complicated, but none of that has anything to do with a monorepo, and monorepos still sometimes simplify things.
So there's a few tools that are common: feature flags, 3 stage-rollouts, and probably more that are relevant, but let's dive into those first two.
Feature "flags" are often dynamically scoped and runtime-modifiable. You can change a feature flag via an RPC, without restarting the binary running. This is done by having something along the lines of
A/B testing tools like optimizely and co provide this, and there are generic frameworks too. `condition_that_enables_feature()`, here is a dynamic function that may change value based on the time of day, the user, etc. (think something like `hash(user.username).startswith(b'00') and user.locale == 'EN'`). The tools allow you to modify these conditions and push and change the conditions all without restarts. That's how you get per-user opt-in to certain behaviors. Fundamentally, you might have an app that is capable of serving two completely different UIs for the same user journey.Then you have "3-phase" updates. In this process, you have a client and server. You want to update them to use "v2" of some api, that's totally incompatible with v1. You start by updating the server to accept requests in either v1 or v2 format. That's stage one. Then you update the clients to sent requests in v2 format. That's stage two. Then you remove all support for v1. That's stage three.
When you canary a new version of a binary, you'll have the old version that only supports v1, and the canary version that supports v1 and v2. If it's the server, none of the clients use v2 yet, so this is fine. If it's the client, you've already updated the server to support v2, so it works fine.
Note again that all of this happens whether or not you use a monorepo.