Naar hoofdinhoud gaan

Product:

Regio:

Datum:

september 2026

30

Incorrect Region: correction: Impacted region(s) within a previous communication has been corrected.

What happened?

Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 01 October 2026, a subset of customers using gateway services in multiple regions experienced degraded or interrupted network connectivity. Impacted customers may have observed gateways failing to load in the Azure Portal, along with failures or delays in network management operations across the following impacted services.

  • Azure ExpressRoute Gateway
  • Azure Firewall
  • Azure Application Gateway and Web Application Firewall
  • Azure VPN Gateway
  • Azure VMware Solution

What do we know so far?

Our investigation identified that a recent change to a regional gateway management service triggered a higher-than-expected load when an unrelated operating system servicing maintenance proceeded gradually through multiple regions. Normally, this gateway management service would auto scale as needed. Demand on dependent services increased due to the increased workload, preventing these regional services from scaling as expected. The operating system servicing maintenance was paused as a precaution.

We reverted the contributing regional gateway manager change, which reduced gateway manager load and allowed affected services to recover. We are continuing to investigate the scaling behavior and the safeguards needed to help prevent recurrence.

How did we respond?

  • 20:30 UTC on 30 September 2026 – Customer impact began.
  • 21:29 UTC on 30 September 2026 – We began investigating connectivity issues affecting Azure ExpressRoute Gateway in UK South.
  • 22:27 UTC on 30 September 2026 – We identified that the issue was affecting multiple Azure regions and expanded our investigation.
  • 23:05 UTC on 30 September 2026 – We identified a correlation with operating system servicing activity and paused further servicing while we worked to restore affected services.
  • 01:36 UTC on 01 October 2026 – Recovery progressed across most affected regions, while configuration changes were applied to remaining impacted regions, including France Central, North Europe, Southeast Asia, UK South, and UK West.
  • 02:15 UTC on 01 October 2026 – We confirmed that the issue was mitigated.

What happens next?

  • Our team will be completing an internal retrospective to understand the incident in more detail. Once that is completed, generally within 14 days, we will publish a Post Incident Review (PIR) to all impacted customers.
  • To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: .
  • For more information on Post Incident Reviews, refer to .
  • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact:
  • Finally, for broader guidance on preparing for cloud incidents, refer to .

29

Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube):


What happened?

Between 10:03 UTC and 15:58 UTC on 29 September 2026, a platform issue resulted in impact to Azure OpenAI Service, Foundry Agent Service, Foundry Models, and Azure AI services in the Sweden Central region. Impacted customers experienced intermittent request failures, increased latency, and/or HTTP 500 and 503 errors when submitting requests to affected models and data plane APIs hosted in this region. 

Note that the Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at:

What went wrong and why?

When requests are made to Azure OpenAI Service and related Azure AI services in a region, the service relies on a regional backend service to retrieve resource metadata and configuration information for the account making the request. This backend service reads from a caching layer and a database layer. Other internal services that call this backend service maintain their own caches of this information, to reduce load on the backend service.

At approximately 09:38 UTC on 29 September 2026, a planned deployment of one of these internal services began in Sweden Central. As part of this deployment, the internal service's caches were cleared and - because the deployment coincided with peak traffic hours in the region - this produced a large number of cache misses, where requests were made for information no longer stored in the cache. Two other defects in how that service read from its caches amplified this effect, causing most lookups to bypass the cache and fall back to calling the backend service directly. By approximately 10:00 UTC, traffic to the backend service had increased to approximately 20 times its normal level.

This surge drove a sharp increase in requests from the backend service to its caching layer, which reached full utilization, thereby increasing latency. This higher latency caused connections to be held open longer, while client retries added further requests. As a result, the network connection capacity on the infrastructure node instances was exhausted, so new outbound connections to both the caching layer and the database layer could not be established.

While this traffic surge was a trigger, the extended duration of the incident was primarily caused by how the backend service determined whether its instances were healthy. Automated health checks for each instance depended on successfully connecting to the database layer. When instances could not establish new outbound connections, they failed this health check and were automatically restarted by design. As more instances entered this restart loop, fewer healthy instances remained to serve requests, and retries from dependent services further increased the load on the remaining instances.

Multiple factors also extended the duration of the incident. Existing network metrics and diagnostic metrics did not effectively indicate the connection exhaustion issue in this scenario. It took an extended amount of time to correlate the failures with what was causing this specific connection exhaustion scenario. As a result, different mitigation attempts - including scaling the number of service instances, and increasing compute and memory resources - could not resolve the incident, until the health check issue was identified and mitigated.

How did we respond?

  • 09:38 UTC on 29 September 2026 – A deployment of an internal service in Sweden Central began, which cleared its caches.
  • 10:00 UTC on 29 September 2026 – Traffic to the backend service began increasing sharply.
  • 10:03 UTC on 29 September 2026 – Customer impact began because of service instance instability.
  • 10:17 UTC on 29 September 2026 – Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region, and alerted our engineering teams.
  • 10:28 UTC on 29 September 2026 – Engineers were engaged and investigating the issue.
  • 11:21 UTC on 29 September 2026 – Additional engineering teams were engaged, ongoing deployments were reviewed for the Azure OpenAI Service in Sweden Central to determine whether any recent change correlated with the issue.
  • 11:30 UTC on 29 September 2026 – We identified that infrastructure node service instances were experiencing timeouts when attempting to connect to the backend service's caching and database layers.
  • 13:55 UTC on 29 September 2026 – We confirmed that the database layer was healthy. We made attempts to mitigate through different scaling adjustments of service instances to lower the number of outbound connections, which did not result in significant improvements.
  • 15:00 UTC on 29 September 2026 – We further increased compute resources for the backend service and increased its resource limits. In parallel, we investigated infrastructure health for affected service instances and restarted service instances.
  • 15:40 UTC on 29 September 2026 – We reconfigured the automated health check to no longer depend on remote connectivity, which stopped the restart loop. Service availability began to recover.
  • 15:58 UTC on 29 September 2026 – Service availability had recovered, and customer impact was mitigated.
  • 16:30 UTC on 29 September 2026 – We identified the source of the increased traffic and correlated it with the internal service deployment, while monitoring confirmed that services were stable.

How are we making incidents like this less likely or less impactful?

  • We have reconfigured the automated health check for the backend service so that it no longer depends on connectivity to its database layer, preventing instances from being restarted unnecessarily during dependency issues. (Completed)
  • We have added monitoring and alerting on the load and latency of the caching layer used by the backend service. (Completed)
  • We have updated our troubleshooting and procedural guidance for such connection exhaustion scenarios, to assist on-call engineers in mitigating similar scenarios. (Completed)
  • We are adding additional throttling and rate limiting for the backend service, to protect it from sudden increases in traffic. (Estimated completion: October 2026)
  • We are increasing the resource capacity of the caching layer used by the backend service across regions. (Estimated completion: October 2026)
  • We are improving diagnostic metrics and automated detection, to help identify such issues more quickly. (Estimated completion: October 2026)
  • In the longer term, we are partitioning the caching layer used by the backend service to increase its throughput. (Estimated completion: December 2026)

How can customers make incidents like this less impactful?

  • Consider reviewing our guidance and best practices related to Business Continuity and Disaster Recovery (BCDR) scenarios for Microsoft Foundry:
  • Consider reviewing and implementing our best practices surrounding retry patterns, especially with exponential backoff, to improve workload resiliency during intermittent issues:
  • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review:
  • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact:
  • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

juli 2026

23

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?

Between 14:44 and 19:41 UTC on 23 July 2026, a subset of customers experienced connectivity failures, increased latency, and/or difficulty accessing Azure services hosted within the West US region. Impact was limited to network traffic entering or exiting the region, traffic that remained entirely within the region was not affected.

Impacted services included: App Service, Application Insights, Azure AD B2C, Azure AI Bot Service, Azure AI Search, Azure API Management, Azure Application Gateway, Azure Bastion, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure ExpressRoute, Azure Firewall, Azure Kubernetes Service (AKS), Azure Monitor, Azure Speech in Foundry Tools, Azure Virtual Desktop, Azure Virtual WAN, Azure VMware Solution, Azure VPN Gateway, as well as Microsoft Sentinel and Power BI Embedded. Downstream impact to Microsoft 365 services has been communicated via MO1437424.

What went wrong and why?

Azure datacenters connect through redundant network paths, ensuring that traffic can continue flowing even when one path is unavailable for maintenance or repair. When unplanned maintenance – including ‘break-fix’ hardware repair – is needed on an optical device along one of these paths, the repair process starts by creating system-readable instructions, verifying that the redundant path remains healthy, and performing safety checks to confirm that the work will not affect customer traffic. Because break-fix activities on a single device are generally expected to be impactless, these requests are processed automatically without human engagement or triage.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk. A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe. The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter's worth of devices in a single request. As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption. The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure. Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

Meanwhile, strong signals indicating WAN impact drew much of the investigative focus. Two parallel investigation streams developed: one focused on the WAN routing anomaly, and another examining network device health from within the datacenter. It took additional time to work past these misleading signals and eventually identify the break-fix activity as the source. It was not until 17:19 UTC that our engineers were able to correlate the routing anomaly with the break-fix activity, at which point we had identified the trigger event and began planning a rollback.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful. After our engineers correlated the routing anomaly with the break-fix activity, we identified that the incomplete preparation activities established the trigger conditions. We initiated a manual rollback at 17:45 UTC, then datacenter-to-WAN connectivity was restored when the rollback completed at 18:26 UTC, and all impacted services fully recovered by 19:41 UTC.

How did we respond?

  • 14:44 UTC on 23 July 2026 – Device break-fix began. Earliest potential customer impact time. 
  • 14:45 UTC on 23 July 2026 – Service alerts fired and multiple Azure service teams started investigating the traffic anomalies.
  • 14:54 UTC on 23 July 2026 – Automated recovery system started performing multiple retries to recover the devices that were affected by the over scoping that happened during the break-fix issue.
  • 15:26 UTC on 23 July 2026 – Routing anomaly alert incorrectly triaged to WAN issue and was sent to engineers to investigate further. 
  • 16:58 UTC on 23 July 2026 – After multiple retries, automated recovery system raised an alert to engineers to investigate. Engineers tried to recover the devices manually. 
  • 17:19 UTC on 23 July 2026 – Engineers correlated the routing anomaly with the initial break-fix trigger event and began attempting auto-rollback.
  • 17:45 UTC on 23 July 2026 – Engineers initiated manual rollback of the break-fix.
  • 18:26 UTC on 23 July 2026 – Rollback was successful on all affected devices, engineers continued monitoring recovery of affected services.
  • 19:41 UTC on 23 July 2026 – Customer impact confirmed as mitigated, after all services returned to expected operating levels.

How are we making incidents like this less likely or less impactful?

  • We have fixed the defect in the blast radius analysis system that incorrectly expanded the scope of this ‘break-fix’ unplanned maintenance to include additional devices. Input validation now ensures that both diverse paths for a datacenter cannot undergo break-fix requests at the same time. (Completed)
  • We have scanned all existing break-fix requests in the system to identify similar conditions that could have allowed the same defect to produce similar impact elsewhere. (Completed)
  • We have enhanced safety checks to block any break-fix or maintenance activity that would affect all diverse paths for a datacenter, to de-risk this failure mode. (Completed)
  • We are improving change visibility, break-fix requests will be recorded in the standard Azure change ledger, to reduce the time required to correlate maintenance activity with network issues during future investigations. (Estimated completion: August 2026)
  • We are enhancing our datacenter traffic monitoring to detect abnormalities in inbound and outbound traffic more quickly, and surface these directly to engineers for evaluation – enabling faster customer notification, and mitigation efforts. (Estimated completion: September 2026)
  • We are improving change tooling guardrails so that multiple devices cannot be isolated simultaneously during maintenance. (Estimated completion: September 2026)
  • In the longer term, we are improving our automated recovery system to fail more quickly and escalate to engineers more promptly when retries are unsuccessful, rather than continuing to retry against disrupted connectivity. (Estimated completion: October 2026)
  • Finally, we are improving our automated rollback so it can recover more robustly, even when the underlying connectivity on which it depends is degraded. (Estimated completion: October 2026)

How can customers make incidents like this less impactful?

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey: