Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/8GCS-858
What happened?
Between 23:20 UTC on 9 March and 19:32 UTC on 10 March 2026, a platform issue resulted in impact to the Azure OpenAI Service. Impacted customers experienced HTTP 400 and HTTP 429 error responses, specifically for the GPT-5.2 model. All other GPT models were unaffected during this time.
This incident impacted customer resources and queries in the following regions: Australia East, Central US, East US 2, Korea Central, Norway East, Sweden Central, and UK South.
What went wrong and why?
The Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at: https://azure.microsoft.com/blog/enterprise-trust-in-azure-openai-service-strengthened-with-data-zones.
A recent update to the Azure OpenAI GPT 5.2 model introduced a configuration change that was not compatible with the version of the model engine code running in production. As part of this update, certain feature settings were enabled to improve service efficiency and resilience – however, the deployed engine version did not yet support those settings. As a result, when customer requests were routed to engines with this mismatch, the service was unable to process these requests correctly.
Generally speaking, any updates to the service are rolled out in line with our Safe Deployment Practices (SDP) which deploys to different regions gradually, in stages. During this update, the earlier stages of the rollout did not include sufficient backend model instances for this issue to surface before the update progressed to additional regions. As a result, the rollout had completed its deployment across our fleet before we were able to determine customer impact.
During mitigation of the primary issue, we identified a secondary issue that affected service recovery. Azure OpenAI relies on internal telemetry to understand real-time service capacity across regions and to route traffic accordingly. At the time recovery actions were underway, an unrelated issue in this internal telemetry system led to incomplete capacity information being leveraged. As a result, traffic routes were temporarily being determined using incomplete data, which led to a disproportionate amount of traffic being directed to a limited set of available regions. This created additional resource pressure in those regions and resulted in continued intermittent request failures (HTTP 429 errors) for some customers, even as the rollback of the configuration issue was progressing and other regions were actually available to receive requests. Once the routing updates were successfully completed and full capacity information was restored across regions, traffic distribution normalized and service recovery progressed as expected.
During the incident, we also identified a monitoring gap related to anomalous HTTP 400 error patterns. While HTTP 400 responses do occur during normal service usage, our monitoring was not configured for service‑side anomalies, only for client-side errors – which are typically caused by incorrect parameters in user requests. This miss in monitoring during the initial stages of the incident delayed detection and response.
How did we respond?
How are we making incidents like this less likely or less impactful?
How can customers make incidents like this less impactful?
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/8GCS-858
Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/_SVS-5_G
What happened?
Between 07:58 UTC on 07 February 2026 and 04:24 UTC on 08 February 2026, impacted customers experienced intermittent service unavailability, timeouts and/or higher than normal latency for services in the West US region. The event began following a power interruption affecting one of the datacenters within the region, after which impact manifested as infrastructure availability loss and service disruptions across multiple dependent workloads in the region.
As power stabilization progressed, recovery proceeded in phases – however, a subset of storage and compute infrastructure components did not return to a healthy state immediately after power restoration, which slowed recovery for dependent components, and contributed to ongoing symptoms such as delayed telemetry and resource recovery.
Impacted services included:
What went wrong and why?
Under normal operating conditions, Azure datacenter infrastructure is designed to tolerate utility power disturbances through redundant electrical systems and automated failover to on-site generators. During this incident, although utility power was still active, an electrical failure in an onsite transformer resulted in the loss of utility power to the datacenter. Although generators started as designed, a cascading failure within a control system prevented the automated transfer of load from utility power to generator power. As a result, Uninterruptible Power Supply (UPS) batteries carried the load for several minutes – until they were fully depleted, leading to customer impact from this power loss as early as 07:58 UTC.
Once power loss occurred, recovery was not uniform across all dependent infrastructure. Our datacenter operations team restored power to IT racks by leveraging our onsite generators, with 90% of IT racks powered by 09:31 UTC on 07 February. While infrastructure generally came back online as expected after power restoration, subsets of networking, storage and compute infrastructure required further electrical control system troubleshooting, before power could be restored. We were fully restored and running generator-backed power by 11:29 UTC. Power restoration was sequenced to avoid destabilizing power that had already been brought back online, and due to the inconsistent state of the control system.
Following power restoration at the datacenter, recovery of customer workloads progressed in stages based on dependencies. Network and Storage recovery were critical requirements, as compute hosts must be able to access persistent storage to complete boot, validate system state, and reattach customer data before workloads can resume. Network devices and services recovered as expected once power was restored.
The power loss affected six storage scale units within the datacenter, of which four recovered as expected once power was restored. Two storage scale units experienced prolonged recovery, which became a primary factor contributing to extended impact as these contained a large subset of storage nodes that failed to complete the boot process. During the boot process, critical artifacts need to be downloaded, but these were delayed or timed out – resulting in these nodes entering a state for which there was not an automated recovery action. Manual operations were attempted, and eventually succeeded in recovering the necessary nodes to bring these storage scale units back online. Due to the dependencies that many compute and platform services have on these storage scale units, there was a delay in overall service restoration. Storage recovery completed by 19:05 UTC.
Compute recovery began once power and storage dependencies were partially restored. Automated recovery systems were available and functioning, but the scale of the event resulted in a surge of required recovery activities across the datacenter. This placed sustained pressure on regional shared infrastructure components responsible for coordinating health checks, power state validation, and repair actions. During the initial recovery window, this elevated load reduced the speed at which individual compute nodes could be recovered. While many nodes returned to service as expected, some nodes required repeated recovery attempts, due to insufficient prioritization across concurrent fault categories during the surge of recovery activities. We introduced optimizations to accelerate recovery, including deprioritizing non-critical recovery activities. As pressure on these shared systems decreased, compute recovery progressed more predictably and remaining nodes were gradually restored. Compute recovery completed by 23:30 UTC.
How did we respond?
How are we making incidents like this less likely or less impactful?
How can customers make incidents like this less impactful?
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/_SVS-5_G
Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/FNJ8-VQZ
What happened?
Between 18:03 UTC on 02 February 2026 and approximately 00:30 UTC on 03 February 2026, a platform issue caused some customers to experience degraded performance and control plane failures, for multiple Azure services across multiple regions. Impacted services included:
As part of our recovery, between 00:08 UTC and 06:05 UTC on 03 February 2026, customers using the 'Managed identities for Azure resources' service (formerly Managed Service Identity) may have experienced failures when attempting to create, update, or delete Azure resources, or acquire Managed Identity tokens – in the East US and West US regions. Originally communicated on incident tracking ID _M5B-9RZ, this impacted the creation and management of Azure resources with assigned managed identities – including but not limited to Azure AI Video Indexer, Azure Chaos Studio, Azure Container Apps, Azure Database for PostgreSQL Flexible Servers, Azure Databricks, Azure Kubernetes Service, Azure Stream Analytics, Azure Synapse Analytics, and Microsoft Copilot Studio. Newly created resources of these types may have also failed to authenticate using managed identity.
What went wrong, and why?
On 02 February 2026, a remediation workflow executed based on a resource policy, intended to disable anonymous access on Microsoft-managed storage accounts. The purpose of this policy is to reduce unintended exposure by ensuring that anonymous access is not enabled unless explicitly required.
Due to a data synchronization problem in the targeting logic, the policy was incorrectly applied to a subset of storage accounts that are intentionally configured to allow anonymous read access for platform functionality. These storage accounts form part of the VM extension package storage layer. VM extensions are small applications used during provisioning and lifecycle operations to configure, secure, and manage VMs. Extension artifacts are retrieved directly from storage accounts, during early-stage provisioning and scaling workflows.
When anonymous access was disabled across these storage accounts, VMs and dependent services were unable to retrieve required extension artifacts. This resulted in control plane failures and degraded performance across affected services.
Although the policy process followed safe deployment practices and was initially rolled out region by region, the health assessment defect caused the incorrect targeting state to propagate broadly across public cloud regions in a short time window. This accelerated the impact beyond the intended blast radius.
Once identified, the policy job was immediately disabled to prevent further changes. Rollback procedures to restore anonymous access were validated in a test region and then executed region by region across affected storage accounts. As access was restored, control plane operations gradually recovered and service management queues began to clear.
As storage access was restored, a secondary issue emerged. The re-enablement of the VM extension package storage layer allowed multiple previously blocked workflows to resume concurrently, across infrastructure orchestration systems and dependent services.
The resulting surge in resource management operations generated a significant and sudden increase in requests to the backend service supporting Managed Identities for Azure resources. As a result, starting at 00:08 UTC on 03 February 2026, customers in East US and West US experienced failures when attempting to create, update, or delete Azure resources that use managed identities, as well as when acquiring Managed Identity tokens.
Automatic retry behaviors in upstream components, designed to provide resiliency under transient failure conditions, amplified traffic against the managed identity backend. The cumulative effect exceeded service limits in East US. Resilient routing mechanisms subsequently directed retry traffic to West US, resulting in similar saturation.
Although the managed identity infrastructure was scaled out, additional capacity alone did not immediately mitigate impact – because retry amplification continued to generate load at a rate that exceeded recovery capacity. Active load shedding and controlled reintroduction of traffic were required to stabilize the service and allow backlog processing while maintaining normal operations, with full recovery by 06:05 UTC.
How did we respond?
How are we making incidents like this less likely or less impactful?
How can customers make incidents like this less impactful?
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/FNJ8-VQZ