August 2022
18
What happened?
Between 16:30 UTC on 18 Aug 2022 and 02:22 UTC on 19 Aug 2022, a platform issue caused Azure offerings such as Bastion, ExpressRoute, Azure Container Apps, Azure ML, Azure Managed HSM, Azure Confidential VMs, Azure Database Services (MySQL - Flexible Server, Postgres- Flexible Server, PostgreSQL – Hyperscale) to experience provisioning failures globally. This issue impacted customers that relied on provisioning of certificates as part of provisioning of an Azure resource. This write-up is a Post Incident Review (PIR) we are providing to summarize what went wrong, how we responded, and the steps Microsoft is taking to learn from this and improve.
What went wrong, and why?
The requesting authority for Azure Key Vault (the underlying platform, on which all the described services rely for the creation of certificate resources) was experiencing high latency and volume of requests. This resulted in provisioning failures for the impacted services, as those services were not able to acquire certificates within the expected time. During the incident, a backend service that Azure Key Vault relies on, became unhealthy due to an unexpected spike in traffic during scheduled hardware maintenance, which caused a build-up of requests in the queue, resulting in high latencies to fulfil new certificate creation requests.
How did Microsoft respond?
We developed and deployed a hotfix to increase the throughput, created new queues for request processing, and drained the queue of accumulated requests to alleviate the overall latency and process requests as expected.
How is Microsoft making incidents like this less likely, or at least less impactful?
• In the short-term, we are implementing request caps and partitioning the request queues, to help prevent lasting failures in the service in similar scenarios.
• We are also reviewing the backend capacity and gaps in the maintenance process that led to the loss of availability during this maintenance operation.
• Based on our learning from this incident, we are implementing improvements to our health monitoring and operational guidance that would help reduce the time to detect similar issues and allow us to address similar issues before customers experience impact.
• In the longer term, we are working to add fine-grained distributed throttling and portioning, to add additional isolation layers to the backend of this service, which will minimize impact in similar scenarios.
• Finally, we will work to add more Availability Zones and fault domains in all layers of the stack, along with automatic failover to the service, to help prevent disruption to customer workloads.
How can we make our incident communication more useful?
We are piloting this "PIR" format as a potential replacement for our "RCA" (Root Cause Analysis) format.
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/YLBJ-790