How the build pipeline was compromised

Popular DevOps platform CircleCI has blamed an attack that successfully planted malware on an internal engineer’s laptop for a recent security breach.

The attack, acknowledged on January 4, prompted CircleCI to advise software developers that relied on its platform to rotate secrets and API tokens.

In a post-mortem on the breach, published on Friday (January 13), the San Francisco-based company offered a detailed description of what went wrong.


BACKGROUND Devs urged to rotate secrets after CircleCI suffers security breach


CircleCI said it first got wind that something was wrong on December 29 when one of its customers reported “suspicious GitHub OAuth activity”, prompting an investigation involving CircleCI’s security team and GitHub.

The security team at the continuous integration and continuous delivery (CI/CD) vendor then discovered that an “unauthorized third party leveraged malware deployed to a CircleCI engineer’s laptop in order to steal a valid, 2FA-backed SSO [single sign-on] session” on or around December 16.

The malware enabled unnamed attackers to steal session cookie data before impersonating the targeted employee in order to access a “subset of CircleCI’s production systems” using forged access tokens.

“Because the targeted employee had privileges to generate production access tokens as part of the employee’s regular duties, the unauthorized third party was able to access and exfiltrate data from a subset of databases and stores, including customer environment variables, tokens, and keys,” CircleCI’s write-up of its post-mortem explains.

Miscreants were subsequentially able to extract “encryption keys from a running process, enabling them to potentially access the encrypted data” on or around December 22.

Incident response

In response to the attack, CircleCI restricted employee access to its production systems as a temporary measure before rebuilding its production environment with clean hosts and revoking project API tokens.

Bitbucket and GitHub OAuth tokens were rotated in the days after the attack as customers were urged to refresh and change their secrets and API tokens. In addition, CircleCI worked with AWS to notify customers of potentially affected AWS tokens.


CHECK IT OUT Potentially win swag by telling us how to improve The Daily Swig


In the wake of the attack, CircleCI has worked with external incident response experts alongside rolling out a suite of measures designed to harden the security of its platform and help its customers to better secure any secrets.

CircleCI’s writeup goes on to sketch out the tactics and techniques of its attackers alongside the publication of indications of compromise (IOCs), data that will help security defenders at other companies in both identifying and blocking similar attacks.

Although some in the security community faulted CircleCI for failing to follow best practice, particular in a failure to apply adequate access controls to production systems, others praised its “transparent and detailed incident disclosure” and the overall response to the post-mortem from infosec Twitter was positive.


DON’T MISS Deserialized web security roundup – Slack, Okta breaches; lax US gov passwords report; and more