As some of the dust around the xz backdoor is slowly starting to settle, we’ve been getting a pretty clear picture of what, exactly, happened, and it’s not pretty. This is a story of the sole maintainer of a crucial building block of the open source stack having mental health issues, which at least partly contributes to a lack of interest in maintaining xz. It seems a coordinated campaign – consensus seems to point to a state actor – is then started to infiltrate xz, with the goal of inserting a backdoor into the project.
Evan Boehs has done the legwork of diving into the mailing lists and commit logs of various projects and the people involved, and it almost reads like the nerd version of a spy novel. It involves seemingly fake users and accounts violently pressuring the original xz maintainer to add a second maintainer; a second maintainer who mysteriously seems to appear at around the same time, like a saviour. This second maintainer manages to gain the original maintainer’s trust, and within months, this mysterious newcomer more or less takes over as the new maintainer.
As the new maintainer, this person starts adding the malicious code in question. Sockpuppet accounts show up to add code to oss-fuzz to try and make sure the backdoor won’t be detected. Once all the code is in place for the backdoor to function, more fake accounts show up to push for the compromised versions of xz to be included in Debian, Red Hat, Ubuntu, and possibly others. Roughly at this point, the backdoor is discovered entirely by chance because Andres Freund noticed his SSH logins felt a fraction of a second slower, and he wanted to know why.
What seems to have happened here is a bad actor – again, most likely a state actor – finding and targeting a vulnerable maintainer, who, through clever social engineering on both a personal level as well as the project level, gained control over a crucial but unexciting building block of the open source stack. Once enough control and trust was gained, the bad actor added a backdoor to do… Well, something. It seems nobody really knows yet what the ultimate goal was, but we can all make some educated guesses and none of them are any good.
When we think of vulnerabilities in computer software, we tend to focus on bugs and mistakes that unintentionally create the conditions wherein someone with malicious intent can do, well, malicious things. We don’t often consider the possibility of maintainers being malicious, secretly adding backdoors for all kinds of nefarious purposes. The problem the xz backdoor highlights is that while we have quite a few ways to prevent, discover, mitigate, and fix unintentional security holes, we seem to have pretty much nothing in place to prevent intentional backdoors placed by trusted maintainers.
And this is a real problem. There are so many utterly crucial but deeply boring building blocks all over the open source stacks pretty much the entire computing world makes use of that it has become a meme, spearheaded by xkcd’s classic comic. The weakness in many of these types of projects is not the code, but the people maintaining that code, most likely through no fault of their own. There are so many things life can throw at you that would make you susceptible to social engineering – money problems, health problems, mental health issues, burnout, relationship problems, god knows what else – and the open source community has nothing in place to help maintainers of obscure but crucial pieces of infrastructure deal with problems like these.
That’s why I’m suggesting the idea of setting up a foundation – or whatever legal entity makes sense – that is dedicated to helping maintainers who face the kinds of problems like the maintainer of xz did. A place where a maintainer who is dealing with problems outside of the code repository can go to for help, advice, maybe even financial and health assistance if needed. Even if all this foundation offers to someone is a person to talk to in confidence, it might mean the difference between burning out completely, or recovering at least enough to then possibly find other ways to improve one’s situation.
If someone is burnt-out or has a mental health crisis, they could contact the foundation, tell their story, and say, hey, I need a few months to recover and deal with my problems, can we put out a call among already trusted members of the open source community to step in for me for a while? Keep the ship steady as she goes without rocking it until I get back or we find someone to take over permanently? This way, the wider community will also know the regular, trusted maintainer is stepping down for a while, and that any new commits should be treated with extra care, solving the problem of some unknown maintainer of an obscure but important package suffering in obscurity, the only hints found in the low-volume mailing list well after something goes wrong.
The financial responsibility for such a safety net should undoubtedly be borne by the long list of ultra-rich megacorporations who profit off the backs of these people toiling away in obscurity. The financial burden for something like this would be pocket change to the likes of Google, Apple, IBM, Microsoft, and so on, but could make a contribution to open source far greater than any code dump. Governments could probably be involved too, but that will most likely open up a whole can of worms, so I’m not sure if that would be a good idea.
I’m not proposing this be some sort of glorified ATM where people can go to get some free money whenever they feel like it. The goal should be to help people who form crucial cogs in the delicate machinery of computing to live healthy, sustainable lives so their code and contributions to the community don’t get compromised. This means not just doling out free money, but also helping people connect to the therapists, doctors, debt restructuring experts and whatever other specialists we all sometimes need in our lives to help us get back on track.
I’m not going to pretend to know how something like this should be set up, organised, or run, and this article and suggestion are more borne out of frustration with how we’re letting these crucial and hard workers hang out to dry and fend for themselves, but it’s obvious the industry needs to do something. If we don’t, we’re going to be seeing a lot more of the kind of orchestrated, sophisticated attacks like the one xz just experienced.
Open source is more than just code, and it’s about damn time we acknowledge that.
Login
30” and a Panasonic-brand electronic toll collection device that you can insert a smart card into. One particularly noteworthy accessory that was available in mid-90s Delicas was a built-in karaoke machine for the rear passengers. Sadly, mine didn’t have that feature.
