‘Suspicious activity’ caused devs to pull the plug pending further investigation
UPDATED OpenDev.org’s Gerrit deployment has been restored after being taken offline following the detection of malicious activity on its repositories.
The repositories were disabled two hours after project maintainers were alerted to a suspected security breach on Tuesday morning (October 20).
“We believe an admin account in Gerrit was compromised allowing an attacker to escalate privileges within Gerrit,” said Clark Boylan in a service announcement issued later that day.
“Around 02:00 UTC October 20 suspicious review activity was noticed, and we were made aware of it shortly afterwards.
“The involved account was disabled and removed from privileged Gerrit groups. After further investigation we decided that we needed to stop the service, this happened at about 04:00 UTC.”
The service was fully restored around 20 hours later, at 00:10 UTC, “once we were satisfied we had contained the breach,” Jeremy Stanley, OpenDev systems administrator, told The Daily Swig.
YOU MIGHT ALSO LIKE GitHub Gist: Account takeover vulnerability patched in code-sharing web service
The suspicious activity was traced to the compromise, on October 6, of “at least two” Ubuntu One accounts, one of which had administrator privileges. Boylan said both accounts have now been secured.
Said Stanley: “We've been working closely with the operators of UbuntuOne SSO [OpenDev’s OpenID provider] to help them identify any compromised accounts and related malicious activity, however we're uncertain yet as to the means by which control was initially gained.”
SSH key clues
Hunting for evidence of privilege escalation, Gerrit maintainers identified 97 accounts whose SSH keys had been updated since a system back-up on October 1.
These SSH keys have now been deleted, so users who have updated their SSH keys since October 1 must generate new keys.
Boylan said OpenID URLs and group membership changes were also being screened.
Commits submitted to every Git repo branch since October 1 will be scrutinized too.
“We will verify that the latest commit can reach the last known good commit in the git DAG,” explained Boylan. “For non merge commits we will also correlate these to Gerrit changes.
“We will then ask that you help us by verifying the commits on your projects are as reviewed and not malicious."
He added: “We will also need to check git tags which should all be signed and can be verified that way.”
Unclear motives
Jeremy Stanley said the attacker’s “motivations remain unclear” and that “no lasting alteration or tampering of data has been identified. It's entirely possible this incident was merely a precursor to further attempts at compromising the service.”
He added that there was no indication that the breach related to "defects in the Gerrit software".
API users are advised to generate new HTTP API tokens as all existing tokens will be deleted.
Clark Boylan said his team would send emails with further information to affected users soon.
OpenDev hosts open source repositories for OpenStack, among other codebases, which developers can review and continuously integrate through Gerrit, as well as Git and Gitea.
Stanley said OpenDev has around 30,000 user accounts but that not all users interact with the service regularly.
OpenDev’s infrastructure status log provides a rough incident response timeline.
This article was updated on October 22 to correct a technical inaccuracy regarding OpenDev’s Gerrit deployment, and on October 23 to incorporate new comments from OpenDev.