Microsoft Finally Reveals the Cause of the Complete Microsoft 365 Service Outage

Microsoft Finally Announces the Issue That Affected Its Services and the Complete Outage of Microsoft 365

microsoft - Microsoft Corporation — **Microsoft Finally Announces the Issue That Affected Its Services and the Complete Outage of Microsoft 365microsoft-Microsoft Corporation**

Based on Microsoft's Statement:

Microsoft revealed the cause of the service outage in the Microsoft 365 application, which affected users worldwide and lasted approximately five hours last Wednesday.

The American tech giant said the outage was due to a change in the router's Internet Protocol (IP) address, which in turn caused packet routing issues between other routers in its wide-area network.

Microsoft said at the time that the outage occurred after a WAN update caused problems with the Domain Name System (DNS) network configuration, and that Wide Area Network (WAN) users in all regions affected by the infrastructure were experiencing issues accessing Microsoft 365 services.

The service experienced wave after wave of outages, peaking approximately every 30 minutes, according to the Microsoft Azure service status page, which was also affected as it intermittently displayed the error: (504 Gateway Timeout).

The list of services affected by the outage includes: (Microsoft Teams), (Exchange Online), (Outlook), (SharePoint), (OneDrive for Business), (PowerBi), (Microsoft 365 Admin Centre), (Microsoft Graph), (Microsoft Intune), (Microsoft Defender for Cloud Apps), and (Microsoft Defender for Identity).

Also:

In total, Microsoft spent more than five hours resolving the issue, between 7:05 AM GMT (when the investigation began) and when service was restored (12:43 AM GMT).

Now, Microsoft has come forward and stated that the cause of the problem was a change to a router's IP address in the WAN using a command that was not thoroughly vetted and that behaves differently across different network devices.

As part of the planned change to update IP addresses in the (WAN) routers,

a command directed at a router caused messages to be sent to all other routers in the (WAN),

which caused the neighbor table to be recalculated and private routing to be reset.

During this recalculation process, the routers were unable to correctly forward the packets passing through them, as the company added.

Following the incident, Microsoft said it will now block the execution of high-impact commands and will also require all commands to follow specific instructions for security configuration changes.

In total,

Microsoft spent more than five hours resolving the issue, between 7:05 AM GMT (when the investigation began) and when service was restored (12:43 AM GMT).

Now,

Microsoft has come forward and stated that the cause of the problem was a change to a router's IP address in the WAN,

using a command that was not thoroughly vetted and that behaves differently across different network devices.

As part of the planned change to update IP addresses in the WAN routers,

a command directed at a router caused messages to be sent to all other routers in the (WAN),

which caused the neighbor table to be recalculated and private routing to be reset,

during this recalculation process, the routers were unable to correctly forward the packets passing through them.

It also added:

Following the incident,

Microsoft said it will now block the execution of high-impact commands and will also require all commands to follow specific instructions for security configuration changes.