Microsoft outage saw 80% of servers fail – here’s what happened

Microsoft outage saw 80% of servers fail – here’s what happened

Last week saw a truly epic outage to Microsoft 365, with a huge number of users unable to send and receive emails. The company has now admitted that at the worst point, more than 80% of its servers were unresponsive. Microsoft has now explained what went wrong, stating that the mistake that led to the failure was actually made all the way back in July without being detected … The company said at the time that it was a technical issue and not a cyber attack, and that it was working hard to fix the problem, admitting that it was taking longer than expected. Microsoft also denied that it was related to the thousands of layoffs announced the previous month. Users were understandably unimpressed at the scale of the outage and the time taken to fix it, although meme creators were having fun. Microsoft’s explanation – two factors combined Platform head Charles Lamanna admitted the severity of the problem. In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive. He said it resulted from a combination of two issues. The company maintains two independent network gateway systems, one of which failed, meaning that the remaining gateway took the full load. That wouldn’t have been a problem on its own, Microsoft stating that each one has enough capacity to handle all traffic. However, a mistake made back in July led to a cascading failure. A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. This was due to lock contention inside an internal process which caused the machine to become unavailable. This defect was the result of a fix that was prepared specifically for Microsoft 365 networking gateway servers […] The initial change in behavior first started deployment in July of this year. The update went through safe deployment and saturated globally over several weeks, during which time no issues were observed. For more than three months, the change was deployed worldwide without impact. Our normal load testing did not detect the latent issue because the different traffic and SKU mix used in our test labs. The contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on our weaker hardware SKUs with fewer cores. In other words, the update wasn’t tested under the conditions in which it was most likely to fail. 9to5Mac’s Take While the company claims that the incident is unrelated to laying off staff and replacing them with AI systems, I find myself extremely skeptical. It seems inconceivable that a company of Microsoft’s size could make what appears to be a schoolboy error of this magnitude unless there were insufficient engineers working on the update. Please share your own thoughts in the comments. Apple TV 4K (latest model) Apple AirPods Pro 3 Apple HomeView meross Smart Garage Door Opener Logitech MX Master 4 for Mac FTC: We use income earning auto affiliate links. More.

Original Source

Read the full article at 9to5mac →

KhanList aggregates and links to publicly available news content. We do not host full articles from third-party sources. Always verify important information with original sources.