At Novu, we use a monorepo to manage our 24 libraries and apps. There are many debates over whether you should use a monorepo or a poly-repo. For us, visibility, code sharing, standardization, easier refactoring, and a few other reasons were the critical factors for choosing this approach for our open-source notification infrastructure project.
TLDR;
We migrated from yarn workspaces & lerna to PNPM and nx.dev
The bigger, the slower
With all the advantages, there are a few drawbacks to using monorepos. We noticed a particular drawback when scaling the number of packages and amount of code in each one: The time it takes to bootstrap the project and then build any packages within. So a typical GitHub action for a service would run anywhere between 11 to 30 minutes. And that's for each time a PR was created or a code was pushed to remote.
More than that, installing a package locally with yarn install could take around 5 minutes to install and build all the dependencies.
This amount of time spent bootstrapping and building reduced the developer experience and wasted collectively so much talented people's time. Being an open-source project with a growing number of contributors, this was unacceptable.
Debugging the slowest tasks
Inspecting a typical 12 minutes GitHub action, it was clear that two specific steps took almost 70-80% of the overall time:
- yarn install - takes 5-6 minutes
- yarn build:{package} - could take from 3-6 minutes to build the selected package and its dependencies.
Migrating from yarn workspaces to PNPM
PNPM is a fast, disk space-efficient package manager(as stated on their website), and from some of the benchmarks, there was a massive improvement in install time against yarn workspaces.
Moving from yarn install that took around 6 minutes, the migration to pnpm was effortless: Just adding a pnpm-workspace.yaml to the project's root and running pnpm install, that's all. The symlinks and dependencies for each package we're efficiently installed, in... wait for it... just 1.5 minutes! And that's without any cache at all! After PNPM caches the majority of the dependencies, it takes less than 40 seconds to build and install the dependencies from the cached store.
Reducing ~4 minutes from the bootstrap time for every CI run and locally for first-time contributors is a HUGE win. But wait, we can do even better.
From Lerna to NX.dev
After seeing the Turborepo demo by vercel, I was intrigued by their distributed caching mechanism. With such a mechanism, we can reuse the already built packages by other maintainers and download the dist assets instead of rebuilding them each time.
turborepo vs nx.dev?
After brief research, we decided to go with nx.dev for multiple reasons:
- Maturity - nx was in the market for a while now, and they have a pretty big community around them.
- Performance - Seeing some of the benchmarks nx looks like a faster build system overall.
Our community member nishit-g took over the open GitHub issue and quickly after we had a PR open, the results astonished us: 30 seconds the building step! (Instead of the previous 3-6 minutes building a specific set of packages).
After implementing the nx.cloud for distributed caching, the entire 24 packages take less than 5 seconds when fully cached building. But even without being fully cached due to the intelligent parallelism nx performs and builds our target package in less than 30 seconds.
Summary
Reducing our build times from 12+ minutes to around 3 minutes significantly impacts the developer experience of our maintainers. It also reduces the feedback loop from creating a PR to running our test suite to merging the feature.
HUGE Kudos to nishit-g for migrating us from Lerna to NX. Check him out on his Twitter as well!
You can check the final configuration on our GitHub repository.
Top comments (15)
Cool Stuff
What about yarn berry? I didn't make any benchmark about it, but i'm using it in a lot of project and never got any issue so far. Not sure it's faster than pnpm, but it could be :p
Also, nice article! Thanks you for sharing with us your story!
I used berry for quite some time to manage web apps built in a monorepo style. (not nearly as many as 24 applications but usually two or three between frontend and backend plus a couple APIs) It was a love-hate relationship.
Maybe if I had been able to build a brand new app from scratch it would have been easier to transition, but ultimately I ended up going back to NPM after v7 came out and everyone was talking about how they've almost managed to catch up with yarn.
Not going to lie, it's still a tad slower, likely considerably slower for a sizable monorepo, but the familiarity and improved verbosity is so much better than the headache berry was causing me every other week.
Actually not heard about berry, i'll check it out. Thank you!
We are just migrating to Turborepo, it seems more simple to use then NX and it should be faster as well.
That's cool! Congrats Turborepo is very neat and simple indeed. After doing some readings decided to go with NX. Some of the reasons and data we had based on this and a few other benchmarks: github.com/vsavkin/large-monorepo
Iam a little confused, what is monorepo and polyrepo
A monorepo contains multiple projects in a single git repository, and polyrepo means that you have one project per github repository. Hope this explains it to you :)
yeah nah.
A monorepo is not merely a repo that holds many projects.
A monorepo is a repo that holds two or more separate projects that can define their relationship through a dependancy graph.
If you can't do this, then it's just a repo.
Now i understand, Thank you
Try Turborepo
Actually we tried, decided to go with NX in the end.
Yep, because there's not enough resource available to get started with Turborepo. Anyway build caching feature is amazing I'm love it ❤️
Thanks for sharing!
I highly recommend to check out bilt for monorepo management: github.com/giltayar/bilt - would love to hear what you think of it.
With great love, always trying my best on finding how to give back to this amazing community ❤️