It Depends: Software Supply Chain Security - TOB Podcast
This transcript is of an episode of the Trail of Bits Podcast. It was written by Nick Selby and Emily Haavik, and story edited by Chris Julin. You can listen to the original, 20-minute podcast, at Trail Of Bits Audio.
NARRATOR (NICK SELBY): We’ve been thinking a lot about supply chains lately. Because they’ve been breaking. Coffee. Surgical masks. Computer chips. Condoms. Lunchables. They’ve all fallen victim to supply chain problems.
And that got me thinking about my friend Clint Bruce.
CLINT BRUCE [FROM VIDEO CALL]: - Hey buddy, how are you?
NICK SELBY [FROM VIDEO CALL]: - I’m good, man, it’s really good to see you. Hey, Clint, thanks so much for doing this …
NARRATOR: Clint’s a former Lieutenant Commander in Naval Special Warfare - a Navy SEAL- and I was remembering something he told me a while back, so I got him on a video call.
NICK SELBY [FROM VIDEO CALL]: - Tell me why supply chains matter to Navy SEALs.
CLINT BRUCE [FROM VIDEO CALL]: Basically, you put my team in front of a door anywhere in the world and our job is to get through it. People know about that from the news and movies, which is kinda cool, but what they may not think about is how we get to the door in the first place. Because some Marines drove us in a truck, after they pick us up from the point where the Army dropped us off after meeting us on shore, because the Navy brought us to the shore from the aircraft carrier, that the Air Force transport brought us to from the base, and each of those things has to work from the people being healthy to the vehicles starting when they’re supposed to, so basically, if we don’t have that entire supply chain up and running, my guys don’t kick in a door and be heroes and get to do shaving commercials.
NARRATOR: So yeah, the SEALs have a supply chain.
Anybody who makes or assembles or, really, does anything maintains what’s known in the industry as a “BOM” - a Bill of Materials, like the one Clint was describing. At an industrial chemical plant, or television factory, or tire maker, wherever: every thing you buy has a BOM, and each component of every BOM in turn has its own BOM - and so on. String all these together? You have a supply chain.
MUSIC - Everybody Get Up - No Vocals & FX - Ian Post
NARRATOR: Modern software has a supply chain.
Most people have a quaint vision of well-paid software engineers tapping keyboards in a giant, fern, foosball-table- and- kombucha-keg filled room, modern software actually isn’t written.
It’s assembled.
Sure, developers still write code – but for the core functions of most software, they don’t start from scratch. They use open-source code and libraries, developed by a community, and available to everyone. And those open-source building blocks are themselves dependent on OTHER pieces of open-source software, which are built atop yet others, and so on. So the dependencies are recursive, or ‘nested’ like a Russian Matryoshka doll.
And, as it’s become clear to anyone who’s tried to buy a car recently, anything that depends on a supply chain can suffer a break in that chain.
So when you ask whether your software is safe, well … It depends.
MUSIC OUT
NARRATOR: In 2021, talk about the software supply chain moved from the pocket-protector crowd to the mainstream after the high-profile hack of Solar Winds - software used by tens of thousands of companies around the world, ironically, to improve their security.
PATRICK GRAY: If you had to point to one single event that made people care about supply chain security, it was the SolarWinds thing. No question.
NARRATOR: Patrick Gray is a journalist covering the world of information security, and he’s run the Risky Business podcast since 2007.
PATRICK GRAY: it does make sense because as soon as you start [00:00:45] scratching the surface of what software supply chains actually look like just under [00:00:50] that surface, it is a horror show
NARRATOR: Even though companies and security experts were sounding the alarm for years, until Solar Winds, the damage wasn’t scary enough or clear and present enough to get noticed in the boardroom. Here’s one reason for that. The software supply chain is highly complex. So some business people simplify it to make it easier to talk about.
ERIC OLSON: On my tombstone it’s going to read, ‘Nuance matters.”
NARRATOR: Eric Olson is the Director of Threat Intelligence and Incident Response for Jet Blue Airways.
ERIC OLSON: So-called supply chain attacks are like ‘cyber security:’ people talk about it like it’s a monolithic thing, and it’s just not. Defenders work for business people – where the nuance is (properly) lost. Attack details don’t really matter from their perspective. the attackers understand that nuance matters, because they care about the one, minute opening that lets them do what they do.
MUSIC JD Scavenger by Randy Sharp
NARRATOR: The nuance Eric mentions is based on the components that are in your software - the kind that Solar Winds highlighted for us: what is in your software supply chain? The only way to find that out is to get a read on all of your dependencies.
To do this, you can generate something called a software bill of materials - an “SBOM.”
ALLAN FRIEDMAN: So if you go to the store today and buy a Twinkie, it’s going to come with a list of ingredients. Why don’t we expect that same level of transparency in software, right?
NARRATOR: That’s Allan Friedman, senior advisor and strategist at CISA - the Cybersecurity and Infrastructure Security Agency. Allan’s been talking about SBOM for years now - a lot longer than most of us have been.
ALLAN FRIEDMAN: Software isn’t hewn out of alabaster marble by tonsured monks on Greek islands. It’s something that’s assembled of known components. An SBOM is the enumeration of those components. This product uses this component which in turn uses these other components …
NARRATOR: But SBOMs remained the realm of frustrated policy wonks, gobsmacked that everyone kept not listening - until one day in May 2021, just months after the SolarWinds debacle was first reported, the White House put them center stage: in the new Executive Order on Improving the Nation’s Cybersecurity. The Biden Administration announced that it would require SBOMs for all software vendors used by the federal government.
ALLAN FRIEDMAN: This is the highest levels of government trying to figure out how do we react and make legitimate progress on both the security of our software, the security of the software supply chain …
NARRATOR: With that one Executive Order, SBOM went from a nice-to-have, to something you need if you want to do business with the federal government.
And in the end, the work at Trail of Bits to develop It Depends was based upon work supported by DARPA - the Defense Advanced Research Projects Agency.
MUSIC OUT
NARRATOR: Now, there are already commercial tools that will deliver an SBOM.
But none of these tools went a step further, and collected the nested dependencies. So, even though all software is dependent upon something - that’s dependent upon something else - that “something else” wouldn’t show up. That’s a big problem.
And it’s a problem that Trail of Bits wanted to solve.
EVAN SULTANIK: Hi, I’m Evan Sultanik. I’m a principal security engineer at trail of bits … The original problem we were trying to solve was to create a tool that could automatically analyze and compare open source code bases to see how similar they were to each other with respect to the upstream dependencies that they used. We wanted to be able to see what’s the probability that if an upstream library were vulnerable or compromised, how would that trickle down into the software we actually care about?
NARRATOR: To start off, they did what most software engineers do:
EVAN SULTANIK: Initially, we wanted to leverage as much existing tooling as we could to solve this problem.
MUSIC: Ripples by Tamuz Dekel
NARRATOR: The team initially homed in on Dependabot - a tool that was available in GitHub, one of the most popular code repository applications in the world, Dependabot is a tool that can automatically scan your codebase, highlight out-of-date dependency libraries, then insert suggested updates to them within the development workflow.
EVAN SULTANIK: Dependabot seemed like a great choice because it’s open source, and it’s … integrated with GitHub. It seemed like a great idea. It can already scan code bases, and … it can enumerate all of the libraries and dependencies in a codebase.
EVAN SULTANIK: my colleague, Alex Remie, quickly made the first proof of concept of a tool that could extract SBOMs using dependabot.
MUSIC OUT
NARRATOR: But Alex was super busy on another project, so …
EVAN SULTANIK: He transferred that to Eric Kilmer, who was going to polish it … and he got a fully functioning proof of concept. However, from that exercise, we quickly realized that one, it would be very cumbersome to extend dependabot to create the superset of dependencies that we were really interested in. Also dependabot, it was really slow. And we thought that it shouldn’t have to be that slow.
NARRATOR: Then the showstopper:
EVAN SULTANIK: another problem is dependabot doesn’t even support common languages like C and C++.
MUSIC: Future Perfect, by Evgeny Bardyuzha
NARRATOR: Dependabot just wasn’t working. We had to get that full universe of nested dependencies the team was looking for, and we had to make it applicable to a range of common languages and package managers. Otherwise the project was just another SBOM generator.
To make matters more complex, to deploy software, you have to deal with package managers. You’ve seen their ilk when you download a Windows exe file or a Mac installer. The problem with package managers is, they - might … just … grab any one of several qualifying versions to satisfy the requirement, based on what else is running on that box.
EVAN SULTANIK: There’s almost never just one dependency resolution for a piece of software. The dependencies that the package manager will choose to satisfy its requirements are going to depend on the environment in which it’s being deployed. It might have drastically different libraries that get installed on staging than it will in production.
MUSIC OUT
NARRATOR: That’s big, Evan says. And it makes it clear what we need to do.
EVAN SULTANIK: ,we want to enumerate all possible dependency resolutions … not just the single one on the system on which Dependabot is run.
NARRATOR: So the unique proposition of It Depends is that it recursively builds a project’s dependency graph starting from either a source code repository or a package specification, enumerating every dependency it could possibly have, not just a single resolution.
If Trail of Bits could create a tool that could capture those dependency nuances, and could support common languages like C and C++, this could be a really big advance in dependency management. And of course, like all our tools, this needed to be free:
EVAN SULTANIK: From day one, we thought that if a tool like this existed, it would be incredibly beneficial to the open source community. So it was always our intention to release it for free to the public as an open source tool.
NARRATOR: The path to get there wasn’t easy. But with all the risks that come with not knowing what’s actually running in your software, this achievement begged the question: why had no one made a tool like this before?
WILLIAM WOODRUFF: My name is William Woodruff, and I’m a senior security engineer at Trail of Bits.
NARRATOR: Will has a guess at why.
WILLIAM WOODRUFF: I think the single reason why we haven’t seen a lot of other tools attempt to create a sort of whole universe software bill of material is because it is just on its face, it’s a very difficult problem to generalize …
NARRATOR: Actually, it’s a mess; a dog’s dinner of possibilities. This flies in the face of why it feels so easy to click that install wizard and watch an installation get magically handled for you. So we’d like to give you an idea of what is happening behind the scenes when you click that Go button - it’s actually … Well … It’s crazy.
WILLIAM WOODRUFF: there are three major operating systems in current use. Two of which are commonly used for server deployments. On those you can have any number of system level dependencies, versions of operating system vendor dependencies, depending on how up to date the system is, you can have hot fixes, patches that are applied by the company, at runtime, because you have specific requirements.
NARRATOR: Will has spent a lot of time in his career thinking about dependencies - for six years, he’s been a maintainer of Homebrew - a free and open-source software package management system that makes it easier to install software on Apple machines - and that’s where he got his first taste of dependency management. More recently, he’s been a professional contributor to the packaging ecosystem for the Python language, adding features to Py P I - the Python Package Index, and adjacent tooling.
Sometimes, rather than just depending on an open source library and loading it when they need to, software developers will actually copy the files of that library directly into their code, and begin to maintain it separately from the open source project. That’s called Vendoring, and it throws another monkey into this wrench:
WILLIAM WOODRUFF: You can have versions of Python packages that are spoofed because the Python package has been vendored in by the operating systems package manager instead of the Python package index installer. So all of these things are very frustrating exceptions to the general process of dependency resolution and dependency analysis. It’s one of these things where the 90% case is extremely easy. And then the 10% case requires an extraordinary amount of long term detailed effort to model everything correctly.
MUSIC: The Swindler by The Original Orchestra
NARRATOR: One of the things that sets It-Depends apart is the fact that it doesn’t assume you did everything right.
WILLIAM WOODRUFF: a lot of security tools assume that people are going to take the happy path. That means that they assume that all the dependencies and sub dependencies … are fully resolved locally. And that you know exactly, if you haven’t already fetched those dependencies and sub dependencies, which exact versions … you’re going to fetch. This is a very conservative assumption to make when you’re doing analyses
NARRATOR: Just to be clear, the assumption that a user took the Happy Path? It’s almost never correct.
MUSIC OUT
NARRATOR: That leaves the Sad Path.
MUSIC: BLUE - ALTERNATIVE - INSTRUMENTAL VERSION by Faith Richards
WILLIAM WOODRUFF: The sad path is basically that degree of uncertainty. When you install Python packages to the Python packaging ecosystem, each package can run arbitrary code, which at runtime can make dependency decisions for you, depending on the kind of operating system you’re running on, for example.
NARRATOR: This means that package managers and the packages you are installing will actually make different decisions about which dependencies to give you when you click that Go button, depending on, say, whether you are on a Linux machine or a Windows machine. They don’t ask.
WILLIAM WOODRUFF: Because of this, there is all kinds of sadness.
NARRATOR: The sad path involves a lot of uncertainty.
WILLIAM WOODRUFF: when you’re on the sad path … you have to work from the outside in, is the way that I think about it. So your view of the system is what’s happened after you’ve finished setting up all your dependencies and setting up your runtime, you know, you only know at the end, what’s what it’s going to look like, you don’t have a sort of determinate view from from the from the outset. as a result, you have to do a lot more work if you want to approach or approximate that end view, without actually having to do the entire setup.
NARRATOR: It-Depends doesn’t work for everything, so with support from Google, Trail of Bits developed another tool called pip-audit. Pip-audit looks through Python dependency trees for known security vulnerabilities - things that are publicly known to the open source community to be a weakness in a popular package. It works symbiotically with It-Depends:
WILLIAM WOODRUFF: The way that I think of pip audit and it-depends is that they cover two halves of the same whole. Pip audit is a developer-focused tool that helps individual Python application developers identify the vulnerable dependencies in their stack, patch them, get them updated, and move on with their lives. It-depends is a sort of swiss army knife that security engineers like myself and others might use to generate an entire manifest of a project, determine perhaps what the more structural issues within that package might be, or project might be.
MUSIC: OU ALLONS NOUS D’ICI - INSTRUMENTAL by Dan Zeitune
NARRATOR: A tool like it-depends can be used in many different ways. One of the specific goals at Trail of Bits - was to support the company’s assurance practice. So when a Trail of Bits security engineer begins to assess a customer’s code base, they can run it-depends to get a better idea of what the dependencies are, and what vulnerabilities might exist.
EVAN SULTANIK: software development teams can integrate it-depends into their CI pipeline, and be emailed anytime that the SBOM of their software tool changes. Companies undergoing due diligence or auditing can also run it-depends to generate an SBOM. And in general, companies who are concerned about supply chain issues can set their minds at ease by at least understanding the current requirements of their software.
NARRATOR: A former Facebook engineer, Charity Majors, once said that trying to fix software without seeing what it’s doing is like trying to repair an engine by placing your ear against the car’s fender.
At its core, the goal of it-depends was to get underneath the surface. To give all developers a tool to check their work, look under the hood, and make sure what they’re building is safe. Meanwhile, Evan and his team at Trail of Bits continue to work on it-depends, to make it a stronger, more efficient tool.
EVAN SULTANIK: Development of it-depends isn’t over, there’s still lots of work to do. More languages to support and more package managers to support.
NARRATOR: For example, one problem is that despite an effectively global standardization of known security bugs into a library called Common Vulnerabilities and Exposures - CVE - the names used by people who discover vulnerabilities are often marketed differently from the official CVE name.
EVAN SULTANIK: It’s actually a really difficult problem to go from those names to the CVEs that affect it. Because it’s effectively a string matching problem. For many pieces of software. Some package managers do already have a vulnerability database that we can leverage … . But for C and C++ code, it’s actually really hard, in many cases, to map CVEs to the libraries that they’re using.
NARRATOR: And Trail of Bits is now getting around to detection of that vendored code:
EVAN SULTANIK: It-depends currently doesn’t detect that. It-depends is only looking at external dependencies of a code base. But if you copy-paste vulnerable code, you are going to have that vulnerability in your code base. So that’s still an open problem.
MUSIC:
NARRATOR: Trail of Bits is strategically committed to open-source work - with an equally strong commitment to both development and continuous updates of the work done by ourselves and others. In 2021, Trail of Bits employees submitted more than 190 contributions to non-Trail of Bits repositories.
EVAN SULTANIK: We’ve already had some excellent open source contributions. It looks like we’re getting a lot of traction and usage in the community and we’re really excited about the future of it-depends.
NARRATOR: There are links in the show notes to these articles and more:
The original blog post announcing the availability of It Depends describes the history you just heard with more technical specificity, and also of course links to the GitHub repository where you can download It Depends and try it for yourself.
That blog post also links to the repository where you can download pip-audit, and give that a whirl.
The show notes also link to the 2021 Executive Order on Improving the Nation’s Cybersecurity,
If you’re interested in the catalog of open source projects Trail of Bits participates in and contributes to, please read the blog post Celebrating our 2021 Open Source Contributions. The post links to contributions our engineer consultants have made to a huge range of open source projects from assert-rs to ZenGo-X.
Season One of Trail of Bits is available for download now, wherever you get your podcasts.
MUSIC OUT
MUSIC IN Scapes - Gray North
NARRATOR: The people who worked on this podcast are Emily Haavik, Chris Julin, Dan Guido, Clint Bruce, Patrick Gray, Eric Olson, Allan Friedman, Evan Sultanik, Will Woodruff, Trent Brunson, and hi, I’m Nick Selby, I’m the Director of the Software Assurance Practice here at Trail of Bits.
Chris Julin made our theme music.
Trail of Bits helps secure some of the world’s most targeted organizations and devices. We combine high-end security research with a real-world attacker mentality to reduce risk and fortify code. We believe the most meaningful security gains hide at the intersection of human intellect and computational power. Learn more at trailofbits dot com; on Twitter we are AT trailofbits; Dan Guido’s Twitter is AT Dguido, and I’m AT fuzztech.
You can listen to this transcript in its original form, a 20-minute podcast, at Trail Of Bits Audio.