All Day DevOps 2021: Securing the software supply chain with ephemerality and the least-privilege principle

‘The new boundary for systems engineering is how ephemeral can you make any given process with a privilege’

“Every program and every privileged user of the system should operate using the least amount of privilege necessary to complete the job,” said computer scientist Jerry Saltzer back in 1974.

Nearly half a century later, and the principle of least privilege remains paramount in software development, James Watters, CTO for modern application platforms at VMware, told attendees at this year’s All Day DevOps.

While managing risk is increasingly difficult to manage in diverse, multi-cloud environments, DevOps teams now have powerful technical means to adhere to the least-privilege model, he suggested.

Automation and “fundamentally changing how runtime behaves” are all reducing overall system privilege in ways that were impossible back in the 1970s, said Watters, one of three speakers in the keynote session.

Kit Colbert, VMware’s senior VP and CTO, said the benefits extend beyond barring entry to bad actors, because well-intentioned developers inevitably make mistakes.

The more you “structure [access control] in such a way that they can’t accidentally do something that exposes them to risk, the better”, he said.

Colbert also noted how the “notion of user identity in the OS” does not really exist on the open internet (IP addresses don’t really count) – something the YubiKey addressed by retrofitting web programming with a “hardbound identity in a physical key”.

Ephemeral builds

Ephemerality was another recurring theme during the presentation.

“The new boundary for systems engineering is how ephemeral can you make any given process with a privilege,” said Watters.

“After Equifax everyone suddenly got even more hyper-aware of long-lived shared secrets that might be present in their system,” he added.

Ashok Banerjee, VP of engineering for security, compliance, and privacy at VMware, said ephemeral builds are particularly critical “because in production systems the actors typically do not persist malicious activities in the VM because we don’t patch yesterday’s VM”.

He added: “We throw it away and build a new VM. So, they have to drift the configuration in some way. And ephemeral environments that get built from infrastructure-as-code do not allow that – so it’s a powerful tool”.

Attacking the build pipeline

From dependency confusion attacks against open source code to package mirror attacks against the software delivery phase, every layer of the CI/CD pipeline is under assault, said Bannerjee.

But attacks in the intermediate phase – the build pipeline – might be more egregious than those targeting the source code system “because malicious actors do not like version history”.

BACKGROUND Software supply chain attacks – everything you need to know

The SolarWinds assailants, for instance, were better able to disguise their activities by swapping files on the build system.

Attackers typically wait two weeks after initial incursions, he continued, “so you won’t correlate the new behavior with the product installation”.

Observed Colbert: “Even if you repave weekly, you are still able to catch those and get them out of there before they can do any damage”.

The butterfly effect

Asked what they were most concerned about with the DevSecOps status quo, Bannerjee cited “patch flat-footedness – waiting 12 hours might be too long”.

Watters was exercised by the downside of a one-to-many paradigm of code changes.

“We’ve created this universe of cloud automation where small changes roll out to enormous numbers of machines that are not back-office systems they literally run our economy today,” he said.

“The butterfly that flaps its wings in our software supply chain can cause hurricanes.”

Fortunately, thanks to the Equifax and SolarWinds attacks in particular, the information security industry was now “making it one of the core engineering disciplines of modern software operations”.

Watters spotlighted another promising development: pipelines’ “decomposition into personaes”, which “will democratize the DevOps approach” for organizations that otherwise would lack the human resources.

Colbert, meanwhile, raised the prospect of architecturally eliminating whole classes of potential risks or vulnerabilities”. Do it right and it’s a “fundamental level of security” rather than just “playing whack a mole” against myriad threats, he said.