Airbnb has a version of this internally that was pretty awesome called Evergreen, based on the Uber paper as well.
I wish more companies would open source their monorepo infra. A well done monorepo is a huge force multiplier on a large organization, but the OSS world is lacking a lot of the infra so everyone starts from a painful place and works up or has a bad impression of monorepos. Google's Piper is another example where open sourcing or selling it would have done wonders for the industry. In an age of agents landing code very quickly piper has scaled very well since it was already at unimaginable commit velocity, meanwhile everyone else is trying to rebuild source control to keep up.
Keeping your trunk consistently green at scale is probably too costly. Not even Google can keep its google3 monorepo consistently buildable let alone green. It’s more worthwhile to keep trunk mostly green, stop chasing the final 0.1%, and instead develop tooling to quickly identify culprits to be rolled back automatically.
Discussion of earlier work by Uber in this same vein, back when it was solely an internal product (to be clear, I don’t know how much the system has evolved internally since these posts, and I don’t know for sure that this system includes all the same components).
We have a monorepo at work and the CI only runs a minority of tests prior to merging, and then reverts bad commits later. It can take a few hours until every test has run including your commit. I don't know enough about the setup to know how it decides to run what tests though
How does it handle conflicts during reverting? If another merge depends on a change, and that change causes a failing/broken test, does it revert both?
Yeah that's true. It becomes O(N^2) with one factor of N being the number of commits to the repo, and another factor being the amount of CI stuff you do every time the repo changes.
You can do better than O(N^2) if your changes are monotonic /independent (or you're able to cache for basically the same reason). This usually requires either luck, a huge amount of programmer discipline, or very different programming languages than we're used to like unison.
Monorepo benefits maybe a small set of developers that manages it for the whole company, at the cost of most other engineers who develop in it. Unnecessary waste of time.
Case in point another new merge queue has to be created to somehow manage the messiness
It's really the case that Git is terrible at scale. Honestly most source control is.
When you have teams of 10,000+ people working on code, if you wanted to shard it, it will be lots of little projects, then you're going to have dependency hell. The core in for teams will either have to spend most of their time upgrading everyone's packages, our other teams are going to have to deal with those upgrades.
The core infra teams are either supporting lots of versions all at once, or spending extra time on source control.
I don't know about Uber, but at Google the monorepo is absolutely amazing.
I will the one big issues that is really hard in a monorepo, if you want to upgrade some external library not built by Google, when that library has a large breaking API change, it will never get upgraded.
I think many shops made the mistake of sharding per deployment unit or compilation unit of a larger whole. My first exposure was sharding per executable file of a firmware image. That's just terrible. You can't coordinate changes that you want to be atomic in the firmware image.
In response I think some people went to monorepo to get atomic change sets but now they're too big. If you have an android app team and a backend team, and they don't deploy together or build together or do anything together besides agree on an API, which has to be backwards compatible anyway because apps don't instantly self-update, there's no need to put those things together if it causes headaches. It's probably better to have too many things together than too few, but neither is ideal.
I wish more companies would open source their monorepo infra. A well done monorepo is a huge force multiplier on a large organization, but the OSS world is lacking a lot of the infra so everyone starts from a painful place and works up or has a bad impression of monorepos. Google's Piper is another example where open sourcing or selling it would have done wonders for the industry. In an age of agents landing code very quickly piper has scaled very well since it was already at unimaginable commit velocity, meanwhile everyone else is trying to rebuild source control to keep up.
https://news.ycombinator.com/item?id=19692820.
I've always thought if you're pushing changes at the sort of velocity that requires it, you're doing something wrong, anyway.
We have a monorepo at work and the CI only runs a minority of tests prior to merging, and then reverts bad commits later. It can take a few hours until every test has run including your commit. I don't know enough about the setup to know how it decides to run what tests though
Case in point another new merge queue has to be created to somehow manage the messiness
When you have teams of 10,000+ people working on code, if you wanted to shard it, it will be lots of little projects, then you're going to have dependency hell. The core in for teams will either have to spend most of their time upgrading everyone's packages, our other teams are going to have to deal with those upgrades.
The core infra teams are either supporting lots of versions all at once, or spending extra time on source control.
I don't know about Uber, but at Google the monorepo is absolutely amazing.
I will the one big issues that is really hard in a monorepo, if you want to upgrade some external library not built by Google, when that library has a large breaking API change, it will never get upgraded.
I think many shops made the mistake of sharding per deployment unit or compilation unit of a larger whole. My first exposure was sharding per executable file of a firmware image. That's just terrible. You can't coordinate changes that you want to be atomic in the firmware image.
In response I think some people went to monorepo to get atomic change sets but now they're too big. If you have an android app team and a backend team, and they don't deploy together or build together or do anything together besides agree on an API, which has to be backwards compatible anyway because apps don't instantly self-update, there's no need to put those things together if it causes headaches. It's probably better to have too many things together than too few, but neither is ideal.
it was definitely a case of making many things worse to do than in a monorepo, but more things viable to do.