How Linux (Almost) Had a Terrible, Horrible, No Good, Very Bad Day 

How Linux (Almost) Had a Terrible, Horrible, No Good, Very Bad Day

If there’s one thing you can say about the people behind the xz supply chain hack, they were certainly willing to play a long con.    For the last two years, a (probable) state-sponsored hacker quietly began integrating themselves into the open source community, particularly with the people responsible for maintaining xz utils (more on what this is and what it does in a minute.)  They began systematically inserting a back door into this core component of the Linux operating system that would have allowed attackers to bypass SSH authentication and remotely access millions of systems.  We were just days away from the biggest supply chain attack in history when they were caught.

What is XZ Utils?

Xz Utils is a program that handles file compression, and it is included as part of several popular Linux distros like Fedora, Debian, and Ubuntu.  There is even a Windows version, although Windows software is usually a zip file rather than an xz file.    Programs like this are crucial because large downloads like software packages need to be compressed, or they would take forever to download even with the highest internet speed.

Open Source, Open to All

To understand how we came so close to disaster, you first have to understand how open source software works.  Open source means that the source code – the building blocks of the software – is available for anyone to see and modify.   Open source software is like buying a box of legos – sure, you can make the robot on the outside of the box, but you can also modify and invent whatever you want.  The same applies to open source software – if you have the requisite programming knowledge, you can contribute bug fixes, work on features, and shape the future of the programs you use every day.  Software like Microsoft Windows and macOS are closed source (although macOS runs on FreeBSD, which is open source, but the user interface and applications are closed source.) With these operating systems, you’re at the mercy of Microsoft and Apple to fix bugs, and as we all know, they often don’t (just take a look at this 40+-year-old bug someone found in Windows in 2018!)  The huge advantage of using an open source OS like Linux is that if you have a bug or a feature request that you want to be implemented, you can just do it yourself.    Of course, just because anyone can technically contribute, does not mean there is just software anarchy.  According to The Linux Foundation, most projects have a structure:

• Leaders

• Someone responsible for making the final decisions about features, releases, and other priorities

• Maintainers

• These people are leaders for specific areas or features; for instance, there is a documentation leader, a leader for developing device drivers, USB, etc. etc.  They are responsible for reviewing code from others before it gets added to their individual area.

• Committers

• Trusted developers who have done enough work for the project that they can make direct code changes rather than be subject to reviews by the maintainers.

• Contributors

• Anyone who contributes, be it code, documentation, or what have you.  Their contributions are reviewed by the maintainer(s) before they’re added to the project.

Foxes in the Hen House

In 2021, someone with the user name JiaT75 opened a GitHub account and made their first commit to an open source project.  They claimed it was just adding clearer error text when an untaring (aka uncompressing) process failed; at the time, it was added without comment, but in retrospect, it appears suspicious.  These changes have since been reverted. In April of 2022, Jia Tan (aka JiaT75) submitted a patch to Xz via the mailing list.  Around the same time, two people began badgering the maintainer of Xz to add another maintainer because patches were not happening fast or often enough.  Neither of these people had any history in the open source community, and after these messages they were never seen again.  Over the course of 2022, JiaT75 becomes the second most active contributor to the xz project.  In January of 2023 JiaT75 merges their first direct code change, which means they have now achieved a level of trust that allows them to implement the code for the back door.  Over the course of 2023, changes were regularly made as JiaT75 implemented the back door one piece at a time.  In February of 2024, the last few files were completed.     While this was happening, the hacker was contacting the leads of all the major Linux distributions to get them to install the updated version of xz utils.  Richard WM Jones from Redhat wrote about his contact with the hacker and Redhat’s scramble to remove the backdoor once they found it, and Ubuntu has also made public the post from Jia Tan asking them to include it.  This is an overview of the timeline, you can find an excellent detailed version with links to the GitHub submissions and e-mails here.    An Unlikely Discovery  With all the careful measures taken to make this look legit, how did they get caught?  Purely by a stroke of luck. Andres Freund, a developer working at Microsoft, was troubleshooting a performance issue on a Debian Linux system.  When you remember that no stable version of Debian was released with the vulnerability, and therefore he was working on an experimental version, the sheer luck behind this discovery is astounding.  He noticed that SSH logins were using too much CPU and recalled an error he had seen in Valgrind (a program used to monitor computer memory), so he put the pieces together.  Thanks to his keen eye and serious investigative skills, he traced the problem to xz utils and sent a missive to the Open Source Security List to describe the problem.    Most people never dig this deep into performance issues, and even if they do, it takes a lot of system knowledge to be able to trace them to the specific cause the way Freund did.

We’re Safe Now, Right?…..Right?

Supply chain attacks are obviously not limited to open source software. After all, the reason most people know the term “Supply chain attack” is because of SolarWinds in 2020, which was most certainly not open source.  But still, this shows that open source software may be more vulnerable than others.  When the fake accounts began badgering Lasse Collin about lack of updates,  his response showed that the open source developers are subject to limited time, burnout, and other struggles just like closed source developers, and adding this on top of the fact that open source development is not paid, well…it’s easy to see how someone could make themselves popular very quickly, and how maybe new code is not always tested as thoroughly as it should be.  Again, this definitely isn’t a problem specific to open source, but it’s perhaps easier to exploit.  Regardless of the development method, we need to ramp up supply chain security across the board before the next attack is successful.