February 2022
2
Summary of Impact: Between 19:50 UTC and 22:06 UTC on Feb 2, 2022, customers using Azure Active Directory (Azure AD) may have experienced failures when performing any service management operations.
During this incident, the Azure AD REST API service experienced some availability impact globally. However, the overall availability was at 99.996% through the duration of the incident. This means that retries had a high probability of success and customer impact would have been minimal or unnoticed.
After investigation, it was determined that there was no impact to Azure AD B2C, as previously reported.
Even though customer impact may have been minimal or unnoticed, the global nature of the incident led us to attend to this issue at a high severity and overcommunicate to ensure the potentially broad impact was notified.
Root Cause: As part of planned maintenance, a change to a dependency was rolled out which affected the availability of the Azure AD REST API service. Resiliency measures initially delayed the impact by relying on caches. When said measures were exhausted, the first request every 10 seconds resulted in a call to the dependency, causing a timeout and the request to fail.
Mitigation: Engineers rolled back the change to the dependency, mitigating the incident.
Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):
- Improve monitoring of the dependency to help ensure that its availability is more accurately observed regardless of underlying resiliency measures, which would help prevent customer impact much earlier.
- The dependency on the service that caused the impact is being re-evaluated to determine if it can be removed from the Azure AD REST API.
Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey: https://aka.ms/AzurePIRSurvey