It’s 2:36am on a Sunday morning. You’re on-call and your pager is going off with a critical alert. You flip a light on, roll out of bed, and groggily open your laptop. Maybe it’s nothing and you can go back to bed, addressing whatever it is in the morning. You log on, silence the alert, and start digging into whatever’s going on. Something’s obviously not right: clients don’t seem to be connecting to your databases correctly. Or there’s some problem with the schema, but that wouldn’t make sense since no one should have pushed changes this late at night on a weekend. You start sifting through logs. You feel your pulse pick up as you notice strange logs from the databases. Really strange logs. Connection logs from IP addresses that you don’t recognize and aren’t within your VPC. Clients still aren’t able to connect so you decide to use the “break-glass” service account to investigate what’s going on inside one of your production databases and debug further. Maybe there’s a weird configuration that needs updating or something needs to be hard-reset to start working again.
What you see startles you: every single row of your production database has garbled up messes of data, not the textual data you were expecting. Digging further in, you find a recent change to the schema and pushes from the database root account. One change in particular catches your attention: a new table called “ransom_note”. You pause, shocked, waiting to see if you’ll suddenly wake up from a bad dream. You cautiously begin to inspect the new table: “SELECT COUNT(*) FROM ransom_note” returns only 1 row. “SELECT * FROM ransom_note” reveals your worst suspicions: “all your data has been encrypted, pay us 10 BTC to have the decryption key”.
This is a nightmare scenario of almost every technology business owner, Chief Information Security Officer, and security red-team: a sudden and unexpected attack orchestrated through some unknown means that completely cripples your operations. Maybe it was a well orchestrated social engineering attack. Maybe it was an extremely unfortunate misconfiguration that let some bad actors into your networks. Or maybe it was a sophisticated supply-chain attack from one of the many hundreds of open source dependencies you have within your product’s stack.
Supply-chain attacks have become very popular among nefarious actors for a few reasons: open source software is used nearly everywhere and many open source maintainers are spread incredibly thin. Open source software has become the critical infrastructure of the commons that we all depend on today. But it’s not unlikely to find solo-maintained or completely abandoned projects that have millions of downloads and sit in the critical dependency path within the software-supply-chain of many large enterprise products.
A good example of this is the recent xz supply-chain attack against ssh: a malicious actor was able to inject a backdoor into ssh, a secure way to connect to other computers through a network, by adding nefarious code to the xz library, a lossless data compression library. In theory, if this had not been detected as early as it was, this would have given the nefarious actors a way to remotely execute code or gain access to any affected Linux computer. One thing that stands out in this example, like so many other supply-chain attacks, is the maintenance status of xz: it went relatively untouched with only a few people around to maintain it. Burned out, with no other volunteers, and very few resources to dedicate to the project, the attacker was easily able to slip in malicious code. Because of how burned out the maintainers were, the attacker automatically “inherits trust built up by the original maintainer”, using that good will to make nefarious changes.
For further reading and analysis on the tragedy of the xz attack, I highly recommend this piece from Rob Mensching.
While there’s no one catch-all solution for preventing these kinds of problems in open source, one piece of the bigger puzzle is the “Lottery Factor”: a metric that looks at open source communities and the weight and distribution of work being done by individuals within a project.
The way we at OpenSauced are defining the Lottery Factor is a follows:
The minimum number of team members that have to suddenly disappear from a project (they won the lottery!) before the project stalls due to lack of knowledgeable or competent personnel. If 1 contributor makes over 50% of commits: Very high risk. 2 contributors make over 50% of commits: High risk. 3 to 5 contributors make over 50% of commits: Moderate risk. And over 5 contributors make over 50% of commits: Low risk.
The Lottery Factor can help uncover this sort of burnout and identify projects that need an injection of critical engineering resources. This can begin to give you an idea of how catastrophic it would be if someone who makes the majority of contributions in a project suddenly disappeared (because they won the lottery and went off to live their best life on the beach!). This may happen for any number of reasons and it’s important to note that the Lottery Factor is unique to each individual project: it’s not a hard and fast rule, but rather, another important metric in understanding the full story of a project.
With all that in mind, we are very excited to unveil the inclusion of the Lottery Factor in OpenSauced Repo Pages as an additional metric and insight you can inspect!!
Through the lens of the Lottery Factor, we can begin to look at projects with a better understanding of where the critical “human” links in the secure software supply chain are, where funding resources need to be spent, and where to allocate crucial engineering resources.
In the analogjs/analog example above, we can see that in the last 30 days, about 50% of contributions were made by ~2 contributors, 50% of that being Brandon. This gives the overall Lottery factor as “High” and would start to unveil critical personnel in the Analog and Angular ecosystem.
An example of a project where the lottery factor is critically high is core-js, a widely used JavaScript standards library in use by Amazon, Netflix, and many other Fortune 500 companies across the web:
Over the last 90 days, the core maintainer “zloirock” has made the majority of the contributions. And, because of the wide adoption of core-js, this library could be a good candidate for an injection of critical resources to ensure the good standing and governance of the library.
Now, let’s look at a project with a “Low” Lottery Factor over the last year where there are no single individuals with the majority of the commits, kubernetes/kubernetes:
Because there are so many different people from so many different companies invested in the success of the Kubernetes platform and the cloud-native ecosystem, it makes sense that there are no single critical individuals that would be the sole point of failure if they were no longer working on the project.
The Lottery Factor can help tell a story unique to each individual community and project. And it can help open source project offices, small teams, or individual contributors better understand the landscape of any open source project or piece of technology they depend on.
We at OpenSauced hope this can start to help you understand where the critical human factor is within projects you contribute to and depend on! Make sure to check-out OpenSauced Repo Pages and stay saucey everyone!
Top comments (4)
Thanks for the spotlight on Analog! 😄
I'm more mindful of trying not to be the "lottery factor" case study these days now that I can see it more easily. Open source is better with friends and contributors 🤝
Such a great write up @jpmcb and I'm so glad we don't use the term Bus Factor anymore or that it's becoming less common. 😅
ha @nickytonline I'm also glad it's not the bus factor anymore!
Would love to hear from any of you if you've experienced or been impacted by the lottery factor! 🎟️