Zero trust: How the ‘Jia Tan’ hack complicated open-source software

Matteo Collina has written software that’s on your computer. You probably aren’t aware of it, but it’s definitely there, maybe even being used to read this very article. 

He also considers himself a vampire hunter. 

Not the Van Helsing type, mind you. In Collina’s world of open-source software, he considers “vampires” to be anyone that wants those responsible for operating and maintaining open-source projects — known as maintainers — to provide “one-on-one support … without being willing to give anything” in return. 

Collina, the co-founder of backend development tool Platformatic, has been a key maintainer in the open-source software community for more than a decade. He is one among many responsible for NodeJS, a tool that allows developers to use JavaScript in their web applications. Collina is responsible for writing, updating and securing code that is used in millions of projects around the world. 

Collina is well-attuned to the pressures that come from maintaining a vital open-source project. He has missed numerous holidays and gotten into disputes with giant tech companies over his ability to support the code base without receiving any compensation for his labor. He has done this all while watching the once-thriving open-source community struggle to remain sustainable without burning through developers.

That was all before April, when it was discovered that a lone maintainer, suspected to actually be a nation-state-backed hacker, nearly turned a widely used open-source software component into a dangerous Trojan horse. A developer noticed unusual performance in a beta version of the compression utility XZ Utils and discovered that code inserted by the maintainer ‘Jia Tan’ was actually a back door that would have granted hackers administrative privileges to anyone running the code. The back door was caught before it was distributed in an update, preventing a catastrophic outcome for many common Linux distributions.

The Jia Tan incident preyed upon the fragility of the software supply chain, which is largely powered by open-source software maintained by volunteers like Collina. The trust in those maintaining that software, which was already fraught, has been damaged significantly, according to experts who spoke to CyberScoop for this story. 

“The community model of just trusting [the code] because it’s open source was never a great model. You should trust it because you trust the people who wrote it or who reviewed it,” said Aeva Black, the open-source security lead at the Department of Homeland Security’s Cybersecurity and Infrastructure Security Agency. “The [Jia Tan] event did break some of that trust.”

Cracks in the foundation

A week after the Jia Tan incident came to light, the OpenJS Foundation and the Open Source Security Foundation issued a joint release revealing that a similar campaign to gain access to the OpenJS software library was foiled in November 2023. OpenJS powers JavaScript projects, which are used by billions of websites worldwide.

“JavaScript is essentially in 99% of the world’s websites because you need it to make your websites interactive. It’s sort of the front door for most companies,” said Robin Ginn, the executive director at OpenJS Foundation, a nonprofit that supports the Javascript ecosystem. 

Ginn said that the incident response team “saw a very similar pattern to XZ.”  Someone — very likely state-backed actors — targeted a single maintainer using multiple personas with the goal of obtaining administrator rights. The maintainer who was targeted only realized what happened after reading about the XZ Utils attempt. 

OpenJS did not reveal the specific project that was targeted, but Ginn said the particular package has tens of millions of monthly downloads. The incident lacked the sophistication of the Jia Tan operation. Ginn said malicious actors never contributed to the project and were sloppy in hiding their tracks. During the XZ Utils case, Jia Tan first contributed legitimate code in December 2021 before being given maintainer access in September 2022.

“The maintainer already had security measures in place and was not going to hand off a project to an unknown email,” Ginn said of the OpenJS incident.

There was also a separate attack on a different open-source JavaScript library around the same time. A Chinese company called Funnull bought the domain and GitHub account rights of Polyfill, a JavaScript code package that allows modern functionality on older browsers that do not natively support it. The package was soon updated to inject malicious code, leading users to scam sites like sports betting pages, according to the cybersecurity firm Sansec. The operation affected more than 100,000 websites before the domain was taken offline.

The resource question

The open-source software that has become a bedrock of the digital economy often starts as personal passion projects. But popularity and use by governments and corporations rarely accompany additional resources, only more demands for features, fixes, and labor.

“Open source has grown so much in the last 10-20 years, where everybody’s using it and a lot of people are not contributing back,” Ginn said.

Even though groups like the OpenJS Foundation help with managing some of the most widely known software libraries on the internet, the Javascript ecosystem has been running on fumes for years. 

Like many large open-source projects, this one is mostly run by volunteers. OpenJS has two full-time employees dedicated to sustaining and supporting the 35 open-source projects related to Javascript, most of which are maintained and supported by a network of 200 volunteers. Those volunteers are then responsible for dealing with the contributions made to these projects from around 150,000 contributors.

Funding is often an issue. OpenJS secured a grant of over €800,000 from Germany’s Sovereign Tech Fund in March 2023. This grant, one of the few dedicated to open-source security and resilience improvements, will be used to implement security recommendations specifically for open-source development.

“That almost doubled our budget,” Ginn said. “So that’s how underfunded we are.” 

“We are hoping more governments will step in so we can provide more support on infrastructure and security,” she added.

Ginn knows that relying on a smattering of grants is not a sustainable option. In comments submitted to the Office of the National Cyber Director, OpenJS said that the organization exists almost on a year-by-year basis based on the grant money it receives. 

“I’m competing against the bright, shiny object of the day, whereas all of these technologies are being used by companies to build billion-dollar businesses,” Ginn said. 

A dry well

Even with organizations like the OpenJS Foundation, protecting and securing open-source software is not straightforward.

Open-source projects don’t come with IT teams or corporate networks. There is no requirement or mandate that maintainers go through security awareness training. Identity management, microsegmentation, or access control is not something maintainers think about, let alone attach to their projects. 

“Software, and computer systems ultimately, are all built by people and maintained by people. With open source, the difference is that it’s inspectable by anybody, rather than only trusting the vendor,” said Black, who in addition to their role at DHS is also an open-source advocate that has spent decades contributing to projects.

The extremely federated nature of open-source code is problematic from a security perspective because it makes consistent security oversight and vulnerability management across numerous independent contributors and projects difficult to enforce.

“People like to think open source is just one big community and it’s really not,” said Brian Fox, co-founder and chief technology officer at the open-source supply chain cybersecurity firm Sonatype.“It’s hundreds of thousands of individuals. And everybody is different and every project is different. So there isn’t going to be a one-size-fits-all solution.”

Fox, who also helps run Apache Maven, a build automation tool used primarily for Java projects, said even open-source projects with healthy financial backing operate on budgets that pale in comparison to corporations that depend on the software maintainers to oversee. 

“The amount of investment it actually takes to actually prop up what is already a massively distributed and basically free workforce in a way that ensures the infrastructure and the ecosystem can function is like basically spreadsheet dust on most of our government’s budgets,” Fox said. “We just have to have the will to do it.”

Even so, Fox and other experts caution that endless cash will not automatically improve open-source projects. Software development is as much about the process itself, with good projects needing a stable, sustainable process.

Collina, who has made a living as an open-source developer for more than 10 years, said the juxtaposition of what’s needed for security and how that fits into the development process may never be one that comes together.

“How can we make open source safe without destroying open source itself?” he asked.

Fix bugs and burn out

The lack of money or corporate support isn’t the only thing that’s hampering open-source projects. Sometimes it’s the security industry itself. 

Collina said vulnerability researchers are a huge source of frustration. According to him, many appear to care more about earning a bug bounty or building clout than fixing the bugs they discover. He said he’s wasted countless hours on low-quality reports that misunderstand the basic functions of code. When there is a report worth taking seriously, it’s extremely hard to follow the industry norm of a 90-day public disclosure, especially if he is the only person doing the legwork to find and fix the issue.  

“More than half of the security researchers are just abusive — like in the language that they use, the comments they make,” he said. “I don’t give a damn about your bounty.” 

In order to survive, Collina, and many other developers have automated their way around the deluge of bug hunters: auto replies for repeatable issues, ignoring poorly written CVE reports and giving maintainers a break from security issues if they feel burned out. Collina himself will only share certain security updates and repository access to those with whom he’s “broken bread.”

Black said that starting about a decade ago, they began seeing “a large increase in single maintainer projects. There’s just one person who’s writing the code and no one’s reviewing it. That’s the risk.”

All of the security issues Collina has dealt with have been related to actual, technical bugs in the software. With security issues now targeting the people who maintain that software, he worries it will be the last straw for people working on open-source projects. 

“Either you structure your life in which those things do not affect you and you have a very strong mind, or you do not [and] you crash, you burn out,” Collina said. “This amount of pressure pushes people to say, ‘F–k it, I’m not doing it anymore.’“

Here come the feds

One of the biggest consumers of open-source software is, of course, the U.S. federal government.

Black joined CISA to help the agency understand the full scope of the open-source community, from the tech giants using the software to the foundations that financially support the software, to the hobbyists that maintain it all. Black previously led open-source security strategy at Microsoft Azure’s CTO office and was also the vice chair at OpenSSF’s Technical Advisory Committee.

Black is trying to bridge the divide between all three factions. Participating in a transparent, global, and disparate community is a hard ask for the federal government. But the goal is improving the security culture around the open-source ecosystem.

“CISA is approaching this community as a participant, not as a regulator. We’re not telling them what to do, because it’s just free speech,” Black said. “It’s just people sharing their ideas and sometimes collaborating in ways that result in products.”

CISA is not the only agency reaching out to the open-source community. The White House has held several summits with corporations, organizations and foundations that represent some of the more commercially successful open-source projects, like the Linux Foundation and RedHat, along with GitHub and Microsoft. 

Additionally, the Office of the National Cyber Director has established the Open Source Software Security Initiative interagency working group, which produced a strategy for the federal government’s use and contribution to the open-source community. The road map is largely focused on collaboration and engagement, like improving security for developers, training and resources. 

The White House announced a new office aimed at studying open-source components and vulnerabilities in critical infrastructure called the Open-Source Software Prevalence Initiative.

The administration’s approach toward open-source security mirrors its broader push to shift the security burden from end-users and the resource-poor to bigger vendors and companies that have the resources but often lack the will to contribute to open-source projects. CISA has spent much of the past few years advocating for additional involvement and assistance from the companies that benefit from open-source software.

“We are encouraging companies to be responsible consumers of open source,” Black said. “Most companies include open-source software in their commercial products, I think that’s … the most important place for the burden to be shouldered … is when companies are taking open-source [code] from the wild and putting it in their products. They should also be a sustainable contributor back to the communities who are producing or taking care of the software they depend on.”

In the meantime, Black had words of advice for the overworked, single builder looking to outlast burnout and stay secure:

“Find collaborators that you trust, lean much more into trust relationships and make visible to the consumers of your project — other Linux distributions that depend on it — any change in the trust relationships of who’s maintaining the project,” Black said. “That’s what keeps open source safe.”