Ugrás a fő tartalomra

Termék:

Régió:

Dátum:

2026. szeptember

30

Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube): 


What happened? 

Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 1 October 2026, customers using several Azure networking services – including Azure ExpressRoute Gateway, Azure Application Gateway, Azure Firewall, Azure Bastion, Azure Virtual WAN, and Azure VPN Gateway – experienced intermittent failures when performing service management operations. In addition, a subset of upgraded Azure ExpressRoute gateways could not retrieve their required configuration from the control plane service, resulting in temporary connectivity loss. Customers using Azure VMware Solution may also have temporarily lost network connectivity to their software-defined datacenter environments. 

The issue affected multiple regions including Australia Central 2, Australia Southeast, Austria East, Belgium Central, Canada East, Central US, Chile Central, Denmark East, East Asia, France Central, Germany North, Germany West Central, India South Central, Indonesia Central, Japan West, Jio India Central, Korea Central, Malaysia West, North Europe, Norway West, South Africa North, South Central US, South India, Southeast Asia, Sweden South, Switzerland North, Taiwan North, UAE North, UK South, West Europe, West US, West US 2, and West US 3.  

The issue began when a platform-triggered gateway update resumed across multiple regions, increasing request volume to the regional gateway control plane service, thereby affecting its ability to process management operations reliably. Impact times varied region by region, as the issue was triggered by a rolling update moving through the regions. We paused this update while applying a configuration change in phases to restore the control plane service. The last wave of customer impact was mitigated at 02:15 UTC on 1 October 2026.  

What went wrong and why? 

Azure regularly applies operating system and security updates to networking gateway services. These updates are designed to roll out gradually across deployment stages and redundant gateway instances, allowing service health to be evaluated before updates proceed. Before this incident, a temporary deployment restriction had paused operating system updates for a group of networking gateways. When this restriction expired, a large number of gateway updates resumed within a short period, generating a surge in gateway restarts and configuration-retrieval requests. At the same time, a previously deployed change had increased the processing time required for configuration operations handled by a shared regional networking control plane service. The combination of increased request volume and longer processing times overwhelmed the control plane, affecting its ability to process requests reliably. As a result, management operations for several Azure networking services failed intermittently. Existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing under these conditions. 

For Azure ExpressRoute Gateway, an additional issue prolonged connectivity impact. Gateway instances restarting during the update process were marked healthy before their configuration and routing state had been fully restored. This allowed updates to proceed to another redundant instance before the first instance was ready. Consequently, some gateways temporarily lost connectivity to connected on-premises networks because neither instance could retrieve the required routing configuration. After approximately 30 minutes of unsuccessful configuration-retrieval attempts, the affected gateways automatically applied their locally stored last known good (LKG) configurations and restored connectivity. 

Automated monitoring detected ExpressRoute connectivity degradation at 20:43 UTC on 30 September 2026, followed by broader control plane degradation at 20:58 UTC. We pursued several recovery approaches in parallel, including pausing further gateway updates at 21:45 UTC, shifting control plane traffic, and attempting to add resources. Although these actions provided temporary relief, they did not fully resolve the issue. Further investigation identified the combination of elevated request volume and increased processing time per operation as the primary contributing factors. 

Once this was understood, we began rolling back the change that had increased control plane processing time, starting at 01:15 UTC on 1 October 2026. The rollback was applied across the affected regions, and customer impact ended at 02:15 UTC. 

How did we respond? 

  • 20:30 UTC on 30 September 2026 – Customer impact began, with intermittent failures affecting management operations across multiple Azure networking services. 
  • 20:43 UTC on 30 September 2026 – ExpressRoute monitoring detected connectivity degradation. 
  • 20:58 UTC on 30 September 2026 – Control plane monitoring detected health failures across additional regions. 
  • 21:12 UTC on 30 September 2026 – Multiple teams were engaged to investigate the issue and coordinate response efforts. 
  • 21:45 UTC on 30 September 2026 – Further gateway operating system updates were paused to prevent additional update activity. 
  • 22:09 UTC on 30 September 2026 – Traffic shifting actions provided temporary relief, but elevated resource utilization returned. Further traffic shifts were stopped. 
  • 22:15 UTC on 30 September 2026 – Attempts to add control plane resources were investigated, and constraints involving dependent platform services were addressed. 
  • 22:36 UTC on 30 September 2026 – Affected ExpressRoute gateways automatically restored connectivity using their last known good configurations. 
  • 23:55 UTC on 30 September 2026 – We identified increased processing time for gateway configuration operations were associated with a previously deployed performance change, so we determined that a rollback would be a relevant mitigation. 
  • 01:15 UTC on 1 October 2026 – The rollback began in an initial region, where the results were validated before proceeding. 
  • 02:15 UTC on 1 October 2026 – The rollback was completed across the affected regions, and customer impact ended. 

How are we making incidents like this less likely or less impactful? 

  • We have rolled back the change that introduced the performance regression, to de-risk this failure mode immediately. (Completed) 
  • We are improving how paused operating system updates resume, to ensure they follow staged rollout practices rather than resuming across large numbers of gateways at once. (Estimated completion: October 2026) 
  • We are strengthening ExpressRoute gateway health checks, to ensure that redundant instances are updated only after the initial instances have fully restored its configuration and routing state. (Estimated completion: March 2027) 
  • We are addressing platform issues that prevented the control plane service from scaling out successfully during the incident. (Estimated completion: March 2027) 
  • In the longer term, we are improving throttling and load-shedding mechanisms in the control plane service to help protect critical request processing during unexpected increases in demand. (Estimated completion: June 2027) 

How can customers make Incidents like this less impactful? 

  • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. 
  • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review:
  • The impact times above represent the overall incident duration and are not specific to any individual customer. Actual impact varied by service, region, customer, and resource. For guidance on implementing monitoring to understand granular impact:
  • Finally, consider ensuring that the right people in your organization are notified about any future service issues by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

29

Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube):


What happened?

Between 10:03 UTC and 15:58 UTC on 29 September 2026, a platform issue resulted in impact to Azure OpenAI Service, Foundry Agent Service, Foundry Models, and Azure AI services in the Sweden Central region. Impacted customers experienced intermittent request failures, increased latency, and/or HTTP 500 and 503 errors when submitting requests to affected models and data plane APIs hosted in this region. 

Note that the Azure OpenAI Service processes customer requests through model engines deployed across multiple Azure regions and supported by traffic routing systems. Depending on the selected deployment model (Global, Data Zone, or Regional), note that requests may be routed across multiple Azure regions within defined geographic boundaries to support availability and resilience, while customer data remains stored at rest in the selected Azure region. Learn more at:

What went wrong and why?

When requests are made to Azure OpenAI Service and related Azure AI services in a region, the service relies on a regional backend service to retrieve resource metadata and configuration information for the account making the request. This backend service reads from a caching layer and a database layer. Other internal services that call this backend service maintain their own caches of this information, to reduce load on the backend service.

At approximately 09:38 UTC on 29 September 2026, a planned deployment of one of these internal services began in Sweden Central. As part of this deployment, the internal service's caches were cleared and - because the deployment coincided with peak traffic hours in the region - this produced a large number of cache misses, where requests were made for information no longer stored in the cache. Two other defects in how that service read from its caches amplified this effect, causing most lookups to bypass the cache and fall back to calling the backend service directly. By approximately 10:00 UTC, traffic to the backend service had increased to approximately 20 times its normal level.

This surge drove a sharp increase in requests from the backend service to its caching layer, which reached full utilization, thereby increasing latency. This higher latency caused connections to be held open longer, while client retries added further requests. As a result, the network connection capacity on the infrastructure node instances was exhausted, so new outbound connections to both the caching layer and the database layer could not be established.

While this traffic surge was a trigger, the extended duration of the incident was primarily caused by how the backend service determined whether its instances were healthy. Automated health checks for each instance depended on successfully connecting to the database layer. When instances could not establish new outbound connections, they failed this health check and were automatically restarted by design. As more instances entered this restart loop, fewer healthy instances remained to serve requests, and retries from dependent services further increased the load on the remaining instances.

Multiple factors also extended the duration of the incident. Existing network metrics and diagnostic metrics did not effectively indicate the connection exhaustion issue in this scenario. It took an extended amount of time to correlate the failures with what was causing this specific connection exhaustion scenario. As a result, different mitigation attempts - including scaling the number of service instances, and increasing compute and memory resources - could not resolve the incident, until the health check issue was identified and mitigated.

How did we respond?

  • 09:38 UTC on 29 September 2026 – A deployment of an internal service in Sweden Central began, which cleared its caches.
  • 10:00 UTC on 29 September 2026 – Traffic to the backend service began increasing sharply.
  • 10:03 UTC on 29 September 2026 – Customer impact began because of service instance instability.
  • 10:17 UTC on 29 September 2026 – Service monitoring detected elevated failure rates for Azure OpenAI Service requests in the region, and alerted our engineering teams.
  • 10:28 UTC on 29 September 2026 – Engineers were engaged and investigating the issue.
  • 11:21 UTC on 29 September 2026 – Additional engineering teams were engaged, ongoing deployments were reviewed for the Azure OpenAI Service in Sweden Central to determine whether any recent change correlated with the issue.
  • 11:30 UTC on 29 September 2026 – We identified that infrastructure node service instances were experiencing timeouts when attempting to connect to the backend service's caching and database layers.
  • 13:55 UTC on 29 September 2026 – We confirmed that the database layer was healthy. We made attempts to mitigate through different scaling adjustments of service instances to lower the number of outbound connections, which did not result in significant improvements.
  • 15:00 UTC on 29 September 2026 – We further increased compute resources for the backend service and increased its resource limits. In parallel, we investigated infrastructure health for affected service instances and restarted service instances.
  • 15:40 UTC on 29 September 2026 – We reconfigured the automated health check to no longer depend on remote connectivity, which stopped the restart loop. Service availability began to recover.
  • 15:58 UTC on 29 September 2026 – Service availability had recovered, and customer impact was mitigated.
  • 16:30 UTC on 29 September 2026 – We identified the source of the increased traffic and correlated it with the internal service deployment, while monitoring confirmed that services were stable.

How are we making incidents like this less likely or less impactful?

  • We have reconfigured the automated health check for the backend service so that it no longer depends on connectivity to its database layer, preventing instances from being restarted unnecessarily during dependency issues. (Completed)
  • We have added monitoring and alerting on the load and latency of the caching layer used by the backend service. (Completed)
  • We have updated our troubleshooting and procedural guidance for such connection exhaustion scenarios, to assist on-call engineers in mitigating similar scenarios. (Completed)
  • We are adding additional throttling and rate limiting for the backend service, to protect it from sudden increases in traffic. (Estimated completion: October 2026)
  • We are increasing the resource capacity of the caching layer used by the backend service across regions. (Estimated completion: October 2026)
  • We are improving diagnostic metrics and automated detection, to help identify such issues more quickly. (Estimated completion: October 2026)
  • In the longer term, we are partitioning the caching layer used by the backend service to increase its throughput. (Estimated completion: December 2026)

How can customers make incidents like this less impactful?

  • Consider reviewing our guidance and best practices related to Business Continuity and Disaster Recovery (BCDR) scenarios for Microsoft Foundry:
  • Consider reviewing and implementing our best practices surrounding retry patterns, especially with exponential backoff, to improve workload resiliency during intermittent issues:
  • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review:
  • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact:
  • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

2026. július

23

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?

Between 14:44 and 19:41 UTC on 23 July 2026, a subset of customers experienced connectivity failures, increased latency, and/or difficulty accessing Azure services hosted within the West US region. Impact was limited to network traffic entering or exiting the region, traffic that remained entirely within the region was not affected.

Impacted services included: App Service, Application Insights, Azure AD B2C, Azure AI Bot Service, Azure AI Search, Azure API Management, Azure Application Gateway, Azure Bastion, Azure Cosmos DB, Azure Data Explorer, Azure Database for PostgreSQL, Azure Databricks, Azure ExpressRoute, Azure Firewall, Azure Kubernetes Service (AKS), Azure Monitor, Azure Speech in Foundry Tools, Azure Virtual Desktop, Azure Virtual WAN, Azure VMware Solution, Azure VPN Gateway, as well as Microsoft Sentinel and Power BI Embedded. Downstream impact to Microsoft 365 services has been communicated via MO1437424.

What went wrong and why?

Azure datacenters connect through redundant network paths, ensuring that traffic can continue flowing even when one path is unavailable for maintenance or repair. When unplanned maintenance – including ‘break-fix’ hardware repair – is needed on an optical device along one of these paths, the repair process starts by creating system-readable instructions, verifying that the redundant path remains healthy, and performing safety checks to confirm that the work will not affect customer traffic. Because break-fix activities on a single device are generally expected to be impactless, these requests are processed automatically without human engagement or triage.

On 23 July 2026, a break-fix repair was initiated on an optical device to address a network reliability risk. A defect in our blast radius analysis system incorrectly expanded the scope of the repair event to include all optical devices egressing a specific datacenter. The safety validation step, which is designed to confirm that at least one of the two redundant datacenter paths remains available, ran but incorrectly concluded the operation was safe. The checks validated each device individually rather than evaluating the aggregate effect of isolating all devices at once, a scenario that was not accounted for because the system was never designed to process a full datacenter's worth of devices in a single request. As a result, routes were withdrawn from multiple devices simultaneously, disrupting connectivity between the datacenter and the WAN – therefore impacting traffic entering or leaving the West US region.

Once the route withdrawals took effect at 14:44 UTC, physical links and routing adjacencies continued to appear healthy, which initially masked the correlation between the break-fix activity and the connectivity disruption. The impact presented as a WAN routing anomaly, as third-party networks could not reach Azure in the region, rather than as a datacenter connectivity failure. Although all physical work in the region was stopped, our engineers could not correlate to this recent change because the preparation activities in advance of the break-fix did not succeed, so the physical layer and traffic appeared healthy.

Meanwhile, strong signals indicating WAN impact drew much of the investigative focus. Two parallel investigation streams developed: one focused on the WAN routing anomaly, and another examining network device health from within the datacenter. It took additional time to work past these misleading signals and eventually identify the break-fix activity as the source. It was not until 17:19 UTC that our engineers were able to correlate the routing anomaly with the break-fix activity, at which point we had identified the trigger event and began planning a rollback.

Our automated recovery and rollback system detected the device failures, and attempted multiple retries to restore the affected devices. However, because that system depended on the same datacenter connectivity that had been disrupted, its automated rollback attempts were unsuccessful. After our engineers correlated the routing anomaly with the break-fix activity, we identified that the incomplete preparation activities established the trigger conditions. We initiated a manual rollback at 17:45 UTC, then datacenter-to-WAN connectivity was restored when the rollback completed at 18:26 UTC, and all impacted services fully recovered by 19:41 UTC.

How did we respond?

  • 14:44 UTC on 23 July 2026 – Device break-fix began. Earliest potential customer impact time. 
  • 14:45 UTC on 23 July 2026 – Service alerts fired and multiple Azure service teams started investigating the traffic anomalies.
  • 14:54 UTC on 23 July 2026 – Automated recovery system started performing multiple retries to recover the devices that were affected by the over scoping that happened during the break-fix issue.
  • 15:26 UTC on 23 July 2026 – Routing anomaly alert incorrectly triaged to WAN issue and was sent to engineers to investigate further. 
  • 16:58 UTC on 23 July 2026 – After multiple retries, automated recovery system raised an alert to engineers to investigate. Engineers tried to recover the devices manually. 
  • 17:19 UTC on 23 July 2026 – Engineers correlated the routing anomaly with the initial break-fix trigger event and began attempting auto-rollback.
  • 17:45 UTC on 23 July 2026 – Engineers initiated manual rollback of the break-fix.
  • 18:26 UTC on 23 July 2026 – Rollback was successful on all affected devices, engineers continued monitoring recovery of affected services.
  • 19:41 UTC on 23 July 2026 – Customer impact confirmed as mitigated, after all services returned to expected operating levels.

How are we making incidents like this less likely or less impactful?

  • We have fixed the defect in the blast radius analysis system that incorrectly expanded the scope of this ‘break-fix’ unplanned maintenance to include additional devices. Input validation now ensures that both diverse paths for a datacenter cannot undergo break-fix requests at the same time. (Completed)
  • We have scanned all existing break-fix requests in the system to identify similar conditions that could have allowed the same defect to produce similar impact elsewhere. (Completed)
  • We have enhanced safety checks to block any break-fix or maintenance activity that would affect all diverse paths for a datacenter, to de-risk this failure mode. (Completed)
  • We are improving change visibility, break-fix requests will be recorded in the standard Azure change ledger, to reduce the time required to correlate maintenance activity with network issues during future investigations. (Estimated completion: August 2026)
  • We are enhancing our datacenter traffic monitoring to detect abnormalities in inbound and outbound traffic more quickly, and surface these directly to engineers for evaluation – enabling faster customer notification, and mitigation efforts. (Estimated completion: September 2026)
  • We are improving change tooling guardrails so that multiple devices cannot be isolated simultaneously during maintenance. (Estimated completion: September 2026)
  • In the longer term, we are improving our automated recovery system to fail more quickly and escalate to engineers more promptly when retries are unsuccessful, rather than continuing to retry against disrupted connectivity. (Estimated completion: October 2026)
  • Finally, we are improving our automated rollback so it can recover more robustly, even when the underlying connectivity on which it depends is degraded. (Estimated completion: October 2026)

How can customers make incidents like this less impactful?

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey: