GitHub releases post-mortem on race condition vulnerability that forced global user sign-out

Issue resulted in some users gaining access to other users’ sessions

GitHub has released further details of a rare security vulnerability that potentially allowed users to log into other user sessions.

Last week, as reported by The Daily Swig, GitHub users were forcibly logged out of their accounts to protect against what the open source platform called a “potentially serious” race condition security bug.

In a blog post yesterday (March 18), GitHub’s Dirkjan Bussink gave a post-mortem analysis of what caused the bug and how it was fixed.

The issue was first discovered on March 2, Bussink wrote, after a GitHub user reported that they had logged in to their account, but then had been authenticated as another user. A second report from another user followed.

“After reviewing request and audit logs, we could validate the external reports received – a user’s session was indeed seen being suddenly shared across two IP addresses around the time of the reports,” wrote Bussink.

An investigation determined that this race condition vulnerability was caused by a thread handling issue.

Thread handling issues

GitHub browser interactions are mainly handled by a Ruby on Rails application that’s known to have components that were not written to run in multiple threads, Bussink explained.

The new background thread produced a “novel and unforeseen interaction” with GitHub’s exception handling routines, meaning that when exceptions were reported from a background thread – such as a query timeout – the error log would contain information from both the background thread and the currently running request, “showing that the data was being pulled across threads”.

GitHub initially believed this would only be a problem internally, however it was later determined that it was also affecting user sessions.

BACKGROUND GitHub users forcibly logged out of accounts to patch ‘potentially serious’ security bug

The blog post reads: “Unicorn, the underlying Rack HTTP server used in our Rails application, does not create a new and separate env object for each request.

“It instead allocates one single Ruby Hash that is then cleared (using Hash#clear) between each request that it handles.

“This led to the realization that the thread safety issue in our exception logging could result in not only inconsistent data being logged in these exceptions, but the sharing of request data on GitHub.com.”

If an exception occurred at “just the right time” and if concurrent request processing happened in “just the right sequence” across multiple requests, the session in one response was replaced with a session from an earlier response.

“Returning the incorrect cookie only happened for the session cookie header and as we noticed before, the rest of the response body, such as the HTML, was all still based on the user who was previously authenticated,” wrote Bussink.

The bug needed very specific conditions to work – it required a background thread, shared exception context between the main thread and the background thread, callbacks in the exception context, reusing the env object between requests, and GitHub’s particular authentication system.

More information on the vulnerability and a technical analysis can be found in Bussink’s blog post.

Clean up

GitHub was able to fix the issue and identify all user sessions that were affected.

As a precaution, all user sessions on GitHub were revoked, despite the fact that the rare bug had a “very small chance of occurring”.

Bussink wrote: “This was not a risk we were willing to take, given the potential impact of even one of these incorrectly returned sessions being used.”