December 2021
16
Summary of impact: Between 01:00 UTC and 02:25 UTC on 16 December 2021, Azure Active Directory (Azure AD) users may have experienced impact when accessing Microsoft 365, Dynamics 365 and Azure services.
Customers using desktop and mobile applications, such as Microsoft Teams, with their work or school accounts experienced minimal disruption as they were automatically routed to the Azure AD Backup Authentication service.
Some Azure AD B2C users may have experienced impact between 01:00 UTC and 02:25 UTC.
Some Microsoft Account users, using Outlook on iOS to access their email, may have experienced impact between 00:11 UTC and 02:25 UTC.
Root Cause: A regular operating system (OS) update was deployed to endpoints of a backend service in the Azure AD authentication stack, which interacted with the service in an unexpected way, making the service on the updated endpoints unresponsive. As the update rollout progressed, all redundant endpoints were impacted, at which point the service became unavailable, on 16 December 2021 at 00:11 UTC.
All changes, with this one included, follow the safe deployment process (SDP) with automated health monitoring in place meant to stop the rollout in the event of an issue. In this case, due to a gap in the backend service’s health monitoring, the update rollout was not stopped until all redundant endpoints were impacted. As the backend service became unavailable, this issue started to manifest as sign-in failures for a subset of Microsoft Accounts (personal accounts).
The Azure AD authentication stack is designed with circuit-breakers that isolate failures in service dependencies , and in this case, limiting impact to Microsoft Accounts . However, due to a second latent issue in one of the circuit-breakers, requests that were queuing up as a result of sign-in failures were not limited by this circuit breaker, leading to sign-in failures for a subset of users of Azure AD and Azure AD B2C.
During this outage, we failed to communicate on the specific impact to Azure AD B2C, in particular marking it as impacted on the Azure Status Page, due to a coordination issue. In addition, our first notification to impacted customers was delayed.
Mitigation:
01:00 UTC – The backup authentication service was automatically activated and started taking traffic.
02:25 UTC – The backend service was restored .
04:34 UTC – Traffic to the backup authentication service was routed back to the primary authentication infrastructure.
05:44 UTC – The primary authentication infrastructure was monitored to ensure services stability. Further changes to the backend service were disabled and the incident was declared fully mitigated.
Next steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):
• Disable all infrastructure changes including OS updates to the active capacity of the backend service – Complete
• Perform changes to the backend service to help reduce time to detect and recover in similar scenarios – Complete
• Improve the communications process to help account for coverage of all impacted services and scenarios - Complete
• Deploy a change to the circuit breakers to fix the discovered latent issue and help minimize impact caused by this backend service – Complete
• Investigate and solidify a plan for additional process and tooling improvements to help reduce the notification time for impacted customers. - January 2022
• Continue to expand coverage of the Backup authentication service – July 2022