Microsoft reveals hidden bug behind massive Outlook outage, dormant since July

Microsoft reveals hidden bug behind massive Outlook outage, dormant since July

Last week, millions of users around the world faced major disruption when popular Microsoft 365 apps, including Outlook, Exchange Online, and Teams, suddenly stopped working. The global blackout lasted for more than 36 hours, leaving offices, schools, and government departments unable to send emails or access their daily work tools.Now, Microsoft has shared a detailed explanation of what caused the massive crash and how its engineers managed to bring everything back online. In a preliminary post-incident report published on November 15, along with a statement on X from Charles Lamanna, Microsoft’s Executive Vice President of Copilot, Agents, and Platform, the tech giant apologised and admitted that a rare combination of three internal issues caused the meltdown.A hidden bug from July triggered the chaosMicrosoft confirmed that the outage was not caused by a cyberattack. Instead, the primary culprit was a hidden software bug that had been sitting inside Microsoft’s networking gateway servers since July 2026. The company had rolled out a specific update for these gateway servers over three months ago, and it seemed to work fine at the time.However, Microsoft admitted that its initial testing had a major blind spot. The company tested the update on older, weaker hardware with fewer processor cores. When the same code ran on newer, more powerful servers with more cores under heavy traffic, it caused a resource jam, known technically as lock contention that exhausted the servers' CPU and memory, causing them to freeze completely. Three problems hit at the exact same timeFor more than three months, the July bug remained hidden. That changed on Thursday, November 12, when two other routine events happened at the exact same moment.About 15 hours before the crash, Microsoft Teams shifted its traffic to the primary network gateway that also supports Exchange Online. While this kind of traffic shift is part of normal operations, it significantly increased the workload on that gateway. At the very same time, nearly 14 per cent of the gateway’s server capacity was offline due to an unrelated maintenance issue from the previous day.On their own, neither the extra Teams traffic nor the missing 14 per cent capacity would have caused a problem. Together, however, they pushed the remaining servers past their limit during peak US working hours, waking up the hidden July bug.Why more than 80 per cent of servers crashedOnce the first few servers froze, Microsoft’s automated traffic system tried to help by quickly moving users to other healthy servers. Unfortunately, this sudden rush of extra traffic triggered the same bug on those healthy servers, causing them to crash too.Normally, Microsoft’s network is divided into isolated sections so that a problem in one area does not spread to others. Due to a separate issue, those safety barriers failed to contain the problem.Within two hours, the domino effect spread across the globe, leaving more than 80 per cent of Microsoft’s gateway servers completely unresponsive. On November 12, Microsoft’s monitoring tools flagged the spike in connection failures, and customer complaints began pouring in thirty minutes later.Why Outlook stayed down much longer than TeamsFixing the problem took over 36 hours because standard recovery methods failed. Every time engineers restarted a crashed server, a massive backlog of waiting internet traffic rushed in all at once, immediately overwhelming the machine and making it crash again.Microsoft was able to restore apps like Teams, OneDrive, SharePoint, and Office much earlier because those services could be shifted away from the broken network gateway to a backup system. However, Exchange Online, which powers Outlook emails is too massive to make that switch, leaving millions of users locked out of their emails for over a day.How Microsoft finally fixed the blackoutBecause the defective code was over three months old, Microsoft’s automated systems could not simply undo the update with a quick rollback.To solve it, the team had to build a brand-new patch that restored the software to its older, healthy state before the July update. Rather than turning everything back on at once, engineers took large groups of servers offline, sometimes more than half of a region at a time installed the new fix, and slowly brought them back online in phases.Microsoft confirmed that services were running smoothly again, before issuing its final update to customers on November 14 and a final, complete report on the incident will be released on November 20.- Ends

Original Source

Read the full article at Indiatoday →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.