Skip to Main Content

September 2026

30

Join either of our upcoming 'Azure Incident Retrospective' livestreams discussing this incident (to hear from our engineering leaders, and to get any questions answered by our experts) or watch a recording of the livestream (available the following week, on YouTube): 


What happened? 

Between 20:30 UTC on 30 September 2026 and 02:15 UTC on 1 October 2026, customers using several Azure networking services – including Azure ExpressRoute Gateway, Azure Application Gateway, Azure Firewall, Azure Bastion, Azure Virtual WAN, and Azure VPN Gateway – experienced intermittent failures when performing service management operations. In addition, a subset of upgraded Azure ExpressRoute gateways could not retrieve their required configuration from the control plane service, resulting in temporary connectivity loss. Customers using Azure VMware Solution may also have temporarily lost network connectivity to their software-defined datacenter environments. 

The issue affected multiple regions including Australia Central 2, Australia Southeast, Austria East, Belgium Central, Canada East, Central US, Chile Central, Denmark East, East Asia, France Central, Germany North, Germany West Central, India South Central, Indonesia Central, Japan West, Jio India Central, Korea Central, Malaysia West, North Europe, Norway West, South Africa North, South Central US, South India, Southeast Asia, Sweden South, Switzerland North, Taiwan North, UAE North, UK South, West Europe, West US, West US 2, and West US 3.  

The issue began when a platform-triggered gateway update resumed across multiple regions, increasing request volume to the regional gateway control plane service, thereby affecting its ability to process management operations reliably. Impact times varied region by region, as the issue was triggered by a rolling update moving through the regions. We paused this update while applying a configuration change in phases to restore the control plane service. The last wave of customer impact was mitigated at 02:15 UTC on 1 October 2026.  

What went wrong and why? 

Azure regularly applies operating system and security updates to networking gateway services. These updates are designed to roll out gradually across deployment stages and redundant gateway instances, allowing service health to be evaluated before updates proceed. Before this incident, a temporary deployment restriction had paused operating system updates for a group of networking gateways. When this restriction expired, a large number of gateway updates resumed within a short period, generating a surge in gateway restarts and configuration-retrieval requests. At the same time, a previously deployed change had increased the processing time required for configuration operations handled by a shared regional networking control plane service. The combination of increased request volume and longer processing times overwhelmed the control plane, affecting its ability to process requests reliably. As a result, management operations for several Azure networking services failed intermittently. Existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing under these conditions. 

For Azure ExpressRoute Gateway, an additional issue prolonged connectivity impact. Gateway instances restarting during the update process were marked healthy before their configuration and routing state had been fully restored. This allowed updates to proceed to another redundant instance before the first instance was ready. Consequently, some gateways temporarily lost connectivity to connected on-premises networks because neither instance could retrieve the required routing configuration. After approximately 30 minutes of unsuccessful configuration-retrieval attempts, the affected gateways automatically applied their locally stored last known good (LKG) configurations and restored connectivity. 

Automated monitoring detected ExpressRoute connectivity degradation at 20:43 UTC on 30 September 2026, followed by broader control plane degradation at 20:58 UTC. We pursued several recovery approaches in parallel, including pausing further gateway updates at 21:45 UTC, shifting control plane traffic, and attempting to add resources. Although these actions provided temporary relief, they did not fully resolve the issue. Further investigation identified the combination of elevated request volume and increased processing time per operation as the primary contributing factors. 

Once this was understood, we began rolling back the change that had increased control plane processing time, starting at 01:15 UTC on 1 October 2026. The rollback was applied across the affected regions, and customer impact ended at 02:15 UTC. 

How did we respond? 

  • 20:30 UTC on 30 September 2026 – Customer impact began, with intermittent failures affecting management operations across multiple Azure networking services. 
  • 20:43 UTC on 30 September 2026 – ExpressRoute monitoring detected connectivity degradation. 
  • 20:58 UTC on 30 September 2026 – Control plane monitoring detected health failures across additional regions. 
  • 21:12 UTC on 30 September 2026 – Multiple teams were engaged to investigate the issue and coordinate response efforts. 
  • 21:45 UTC on 30 September 2026 – Further gateway operating system updates were paused to prevent additional update activity. 
  • 22:09 UTC on 30 September 2026 – Traffic shifting actions provided temporary relief, but elevated resource utilization returned. Further traffic shifts were stopped. 
  • 22:15 UTC on 30 September 2026 – Attempts to add control plane resources were investigated, and constraints involving dependent platform services were addressed. 
  • 22:36 UTC on 30 September 2026 – Affected ExpressRoute gateways automatically restored connectivity using their last known good configurations. 
  • 23:55 UTC on 30 September 2026 – We identified increased processing time for gateway configuration operations were associated with a previously deployed performance change, so we determined that a rollback would be a relevant mitigation. 
  • 01:15 UTC on 1 October 2026 – The rollback began in an initial region, where the results were validated before proceeding. 
  • 02:15 UTC on 1 October 2026 – The rollback was completed across the affected regions, and customer impact ended. 

How are we making incidents like this less likely or less impactful? 

  • We have rolled back the change that introduced the performance regression, to de-risk this failure mode immediately. (Completed) 
  • We are improving how paused operating system updates resume, to ensure they follow staged rollout practices rather than resuming across large numbers of gateways at once. (Estimated completion: October 2026) 
  • We are strengthening ExpressRoute gateway health checks, to ensure that redundant instances are updated only after the initial instances have fully restored its configuration and routing state. (Estimated completion: March 2027) 
  • We are addressing platform issues that prevented the control plane service from scaling out successfully during the incident. (Estimated completion: March 2027) 
  • In the longer term, we are improving throttling and load-shedding mechanisms in the control plane service to help protect critical request processing during unexpected increases in demand. (Estimated completion: June 2027) 

How can customers make Incidents like this less impactful? 

  • There was nothing that customers could have done to avoid or minimize impact from this specific service incident. 
  • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review:
  • The impact times above represent the overall incident duration and are not specific to any individual customer. Actual impact varied by service, region, customer, and resource. For guidance on implementing monitoring to understand granular impact:
  • Finally, consider ensuring that the right people in your organization are notified about any future service issues by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

< View full status history