Billion-dollar blue team: MSRC spearheads Microsoft’s security defense

Mechele Gruhn provides a behind-the-scenes view into the Microsoft Security Response Center

The Microsoft Security Response Center (MSRC) is at the front line of defense for the world’s biggest software company.

The center is responsible for handling every incoming security vulnerability impacting Microsoft products, along running blue team defense exercises, bug bounties, and coordinating Redmond’s monthly Patch Tuesday efforts.

“There are many thousands of people at Microsoft who have ‘security’ in their job title,” said MSRC principal security manager, Mechele Gruhn. “However, it’s really the MSRC that has the purview across all of Microsoft, in conjunction with a few other groups.”

Incident response

In an insightful webinar hosted by the Ethical Hacker Network last week, Gruhn outlined the MSRC’s wide range of responsibilities when it comes to protecting an organization that reported revenues of $110 billion last year.

“I run a team of 19 people who are responsible for the end-to-end security vulnerability and response,” said Gruhn.

“We are responsible for every incoming security vulnerability for all products, software, services and hardware for all of Microsoft’s offerings. We are responsible for triage, coordinating the information, and publishing on Patch Tuesday.”

Prior to the MSRC’s formation more than 20 years ago, Gruhn said, there was no Patch Tuesday, there were no bug bounty programs, and Platform-as-a-Service hadn’t been invented – much less cloud computing.

Fast forward to 2019, the MSRC is at the forefront of technological change. And in order to ensure maximum efficiency and flexibility, the center has adopted a modern, federated security operations model.

“We are embracing the DecSecOps model,” said Gruhn. “We have not only our product security incident response team (PSIRT), but also our computer security incident response team (CSIRT). We are also responsible for Azure and AI security at Microsoft.”

According to Gruhn, the MSRC’s software security incident response (SSIRP) team is the “conductor of the symphony” when it comes to incident response at Microsoft. And given the vast scope of some security issues, the center plays a vital role in coordinating separate teams around the world.

“We practice our communication, we practice who is on point, we appoint specific people to coordinate all of the communications,” she said. “This may sound like overkill, but it certainly isn’t, particularly when a single large-scale incident response for one of our items can involve well over 600 people in various capacities.”

A day in the life…

She added: “A day in the life of an incident is really a good way to describe what happens. When an incident is reported to us it goes through initial triage. Whether this is a zero-day vulnerability that is being exploited in the wild, or whether it is an attack against networks, it all starts with an initial report.”

“We tend to follow the FIRST guidance that is given for incident response (Microsoft helped to author that guidance). We coordinate with our legal and customer support teams, and bring in red teams and blue teams as needed.”

“Depending on the incident, we may have a continual bridge or regular meetings. But we will also coordinate across industry. Each individual incident is unique. The conductor of the orchestra is still there, but the individual pieces or each incident change based on the specifics of the situation.”

External security efforts

In addition to dealing with vulnerabilities coming in from outside Microsoft, the MSRC also handles flaws that Microsoft employees may have found in other organizations’ products.

“We have the Microsoft Active Protection (MAP) program, where we work with partners to help them provide guidance so they can update their products and help customer protect their products,” said Gruhn.

“We also have a government program, where we work with several governments to help make the world more secure.”

Pwn2Own was taking place in Vancouver at the same time as last week’s webinar. For Gruhn, bug bounties and hackathons are another important way of helping improve the security of the Microsoft ecosystem.

“We love competitions like Pwn2Own,” Gruhn said. “We absolutely love it when teams go up against our products and find vulnerabilities… so that we can fix them. We want to work, and continue to work, to make our software the most secure software that we can.”

Patch me if you can

While the MSRC continues to work with external organizations for coordinated disclosure, the center is perhaps best known for its role in coordinating the monthly Patch Tuesday security updates.

“Patch Tuesday has been in place since 2003 and has happened without fail every single second Tuesday of the month, with one exception,” Gruhn explained. “It’s always an interesting journey.

“We continually will check our fixes for regression issues before they go out the door, and this can lead to us occasionally having to pull a patch at the last minute – or we may need to add something in because a new vulnerability came in that has a very high severity and high impact.

“We try to create a situation where all of these things happen seamlessly because we realize very deeply that this is a very important event that requires a lot of effort from a lot of people around the globe in order to make sure systems are patched.”

Of course, sysadmins around the world will be well aware of the out-of-band updates that are sometimes issued for Microsoft products and services.

As she wrapped up her discussion, Gruhn said the MSRC’s decision to issue updates outside of Patch Tuesday is never taken lightly.

“We are very aware of our customers as we are making these changes,” she said. “Having been outside of Microsoft, I can tell you that random acts of security patching are expensive.

“When we do this, it’s based on risk analysis and doing the best thing for the customer. That’s something to keep in mind when you see an out-of-band release and wonder why Microsoft did that.”

UPDATE (28/03) An earlier version of this article stated that the MSRC was comprised of just 19 people. Microsoft recently got back to us with a statement that reads: “We wanted to clarify that Mechele’s team of 19 people is just one part of the MSRC, and the organization has more than 100 people worldwide who work across vulnerability response, cyber defense operations, engineering, and security community outreach.” We are happy to set the record straight.