Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/7Q30-010
What happened?
Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 1 October 2026, customers using several Azure networking services – including Azure ExpressRoute Gateway, Azure Application Gateway, Azure Firewall, Azure Bastion, Azure Virtual WAN, and Azure VPN Gateway – experienced intermittent failures when performing service management operations. In addition, a subset of upgraded Azure ExpressRoute gateways could not retrieve their required configuration from the control plane service, resulting in temporary connectivity loss. Customers using Azure VMware Solution may also have temporarily lost network connectivity to their software-defined datacenter environments.
The issue affected multiple regions including Australia Central 2, Australia Southeast, Austria East, Belgium Central, Canada East, Central US, Chile Central, Denmark East, East Asia, France Central, Germany North, Germany West Central, India South Central, Indonesia Central, Japan West, Jio India Central, Korea Central, Malaysia West, North Europe, Norway West, South Africa North, South Central US, South India, Southeast Asia, Sweden South, Switzerland North, Taiwan North, UAE North, UK South, West Europe, West US, West US 2, and West US 3.
The issue began when a platform-triggered gateway update resumed across multiple regions, increasing request volume to the regional gateway control plane service, thereby affecting its ability to process management operations reliably. Impact times varied region by region, as the issue was triggered by a rolling update moving through the regions. We paused this update while applying a configuration change in phases to restore the control plane service. The last wave of customer impact was mitigated at 02:15 UTC on 1 October 2026.
What went wrong and why?
Azure regularly applies operating system and security updates to networking gateway services. These updates are designed to roll out gradually across deployment stages and redundant gateway instances, allowing service health to be evaluated before updates proceed. Before this incident, a temporary deployment restriction had paused operating system updates for a group of networking gateways. When this restriction expired, a large number of gateway updates resumed within a short period, generating a surge in gateway restarts and configuration-retrieval requests. At the same time, a previously deployed change had increased the processing time required for configuration operations handled by a shared regional networking control plane service. The combination of increased request volume and longer processing times overwhelmed the control plane, affecting its ability to process requests reliably. As a result, management operations for several Azure networking services failed intermittently. Existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing under these conditions.
For Azure ExpressRoute Gateway, an additional issue prolonged connectivity impact. Gateway instances restarting during the update process were marked healthy before their configuration and routing state had been fully restored. This allowed updates to proceed to another redundant instance before the first instance was ready. Consequently, some gateways temporarily lost connectivity to connected on-premises networks because neither instance could retrieve the required routing configuration. After approximately 30 minutes of unsuccessful configuration-retrieval attempts, the affected gateways automatically applied their locally stored last known good (LKG) configurations and restored connectivity.
Automated monitoring detected ExpressRoute connectivity degradation at 20:43 UTC on 30 September 2026, followed by broader control plane degradation at 20:58 UTC. We pursued several recovery approaches in parallel, including pausing further gateway updates at 21:45 UTC, shifting control plane traffic, and attempting to add resources. Although these actions provided temporary relief, they did not fully resolve the issue. Further investigation identified the combination of elevated request volume and increased processing time per operation as the primary contributing factors.
Once this was understood, we began rolling back the change that had increased control plane processing time, starting at 01:15 UTC on 1 October 2026. The rollback was applied across the affected regions, and customer impact ended at 02:15 UTC.
How did we respond?
How are we making incidents like this less likely or less impactful?
How can customers make Incidents like this less impactful?
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/7Q30-010
Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/air/7QL5-Z50
What happened?
Between 10:03 UTC and 15:58 UTC on 29 September 2026, a platform issue resulted in impact to Azure OpenAI Service, Foundry Agent Service, Foundry Models, and Azure AI services in the Sweden Central region. Impacted customers experienced intermittent request failures, increased latency, and/or HTTP 500 and 503 errors when submitting requests to affected models and data plane APIs hosted in this region.
Note that the Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at: https://azure.microsoft.com/blog/enterprise-trust-in-azure-openai-service-strengthened-with-data-zones/
What went wrong and why?
When requests are made to Azure OpenAI Service and related Azure AI services in a region, the service relies on a regional backend service to retrieve resource metadata and configuration information for the account making the request. This backend service reads from a caching layer and a database layer. Other internal services that call this backend service maintain their own caches of this information, to reduce load on the backend service.
At approximately 09:38 UTC on 29 September 2026, a planned deployment of one of these internal services began in Sweden Central. As part of this deployment, the internal service's caches were cleared and - because the deployment coincided with peak traffic hours in the region - this produced a large number of cache misses, where requests were made for information no longer stored in the cache. Two other defects in how that service read from its caches amplified this effect, causing most lookups to bypass the cache and fall back to calling the backend service directly. By approximately 10:00 UTC, traffic to the backend service had increased to approximately 20 times its normal level.
This surge drove a sharp increase in requests from the backend service to its caching layer, which reached full utilization, thereby increasing latency. This higher latency caused connections to be held open longer, while client retries added further requests. As a result, the network connection capacity on the infrastructure node instances was exhausted, so new outbound connections to both the caching layer and the database layer could not be established.
While this traffic surge was a trigger, the extended duration of the incident was primarily caused by how the backend service determined whether its instances were healthy. Automated health checks for each instance depended on successfully connecting to the database layer. When instances could not establish new outbound connections, they failed this health check and were automatically restarted by design. As more instances entered this restart loop, fewer healthy instances remained to serve requests, and retries from dependent services further increased the load on the remaining instances.
Multiple factors also extended the duration of the incident. Existing network metrics and diagnostic metrics did not effectively indicate the connection exhaustion issue in this scenario. It took an extended amount of time to correlate the failures with what was causing this specific connection exhaustion scenario. As a result, different mitigation attempts - including scaling the number of service instances, and increasing compute and memory resources - could not resolve the incident, until the health check issue was identified and mitigated.
How did we respond?
How are we making incidents like this less likely or less impactful?
How can customers make incidents like this less impactful?
How can we make our incident communications more useful?
You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/7QL5-Z50