Inside Microsoft’s rush to get back online after a Global Outage

Late into the night on Nov 12, hundreds of Microsoft engineers and top executives were sitting in their darkened offices or home basements, logged in to a videoconference triaging a nightmare scenario for a technology company.The world still runs on email, and much of that is processed through Microsoft Outlook. For hours, a company outage affected hospitals, schools, businesses and congressional offices in Washington. Even Microsoft’s own staff couldn’t use email.On the videoconference, Scott Guthrie, the senior executive overseeing the company’s computing infrastructure, was proposing possible fixes, according to a person familiar with the call who was not authorised to speak publicly. People chewed on fingernails, rubbed their foreheads and debated how to turn off other services to prevent a “stampeding herd” so that as systems come back online, they would not be overwhelmed with new demands and shut back down.By the morning of Nov 13, the company had apologised at least twice. It said it determined that the problem stemmed from a recent upgrade to its network, which shut down some servers, and that the systems were steadily improving. “Given the scale of demand, we restored services gradually, validating stability at each stage,” the company said in a statement. “While this took longer, it helped ensure a stable restoration.”Finally, they had kept the herd at bay.The prolonged and widespread outage was just the most recent reminder that the technological underpinnings of modern life and business are run through just a handful of major companies. And while cyberattacks and malicious actors can target their systems, sometimes the companies learn the hard way that they can introduce devastating flaws when updating or maintaining their own global systems.For Microsoft, the problems from this week’s outage are not done yet. The short-term financial fallout should be “manageable,” analysts at investment bank William Blair wrote in a research note on Nov 13. But they said this prolonged outage is a reputational hit that could make companies and other customers reconsider having so much riding on one technology provider.“We believe the outage could prompt enterprises to reassess their dependence on Microsoft’s infrastructure,” they wrote.Outlook and other services affected, including Teams and its artificial intelligence assistant Copilot, are part of Microsoft 365’s cloud subscription business, which is used by more than 450 million people and brought in almost US$28 billion (S$36 billion) in its last quarter, which ended Sept 30. NYTIMESThis article originally appeared in The New York Times.