Azure status history

Watch our 'Azure Incident Retrospective' video about this incident: https://aka.ms/AIR/VLB8-1Z0

What happened?

Between 02:34 UTC and 07:25 UTC on 16 June 2023, a network issue caused excessive packet loss that affected traffic entering or leaving the West Europe region and, to a lesser extent, traffic between Availability Zones inside the West Europe region. Resources hosted in this region may have experienced availability failures, low throughput, or increased latencies. The traffic loss rate peaked at 10% for short periods of time during the incident.

Downstream Azure services dependent on this network connectivity were also impacted – including Azure API Management, Azure App Services, Azure Application Insights, Azure Automation, Azure Container Apps, Azure Container Registry, Azure Cosmos DB, Azure Data Explorer, Azure Data Factory, Azure Database for MySQL, Azure Database for PostgreSQL, Azure Databricks (which also experienced impact in North Europe, as a result of a control plane dependency), Azure DevOps, Azure Digital Twins, Azure Event Grids, Azure HDInsight, Azure IoT Hub, Azure Key Vault, Azure Kubernetes Service, Azure Logic Apps, Azure Monitor, Azure NetApp Files, Azure Resource Graph, Azure Resource Manager, Azure Service Bus, Azure Site Recovery, Azure SQL Database, Azure Spatial Anchors, Azure Synapse, Azure Update Management Center, Azure Virtual Desktop, Azure Virtual Machines, and Azure Web PubSub – as well as subsets of Microsoft 365, Microsoft Intune, Microsoft Purview, Microsoft Sentinel, and Microsoft Power Platform services.

What went wrong and why?

The Azure network is comprised of millions of links between routers, with many redundant links to cope with failures. As a faulty link can cause packet loss or corruption, Azure has network automation systems that continually monitor these links for health. When our automation systems detect a link is unhealthy, the system confirms it is safe to remove the link from service; issues commands to the routers to shut down the link; and then issues a request to datacenter staff to repair or replace the faulty link.

A network architecture detail relevant to this incident is that the routers carrying traffic between Availability Zones and the Wide Area Network in West Europe are connected by groups of multiple physical links bundled together into what are called Link Aggregation Groups (LAGs or port-channels). To keep traffic balanced across the physical links, if one or more links in a LAG fail it is desirable to turn off the entire LAG.

Shortly before the start time of this incident, a new network topology description was added into the network automation systems that manage the West Europe region. This new topology included links that were about to be added to the network to increase the capacity in West Europe, but the physical links had not yet been connected or turned on. This network topology description incorrectly listed these links as being ‘Production’. The network automation systems detected LAGs that contained working physical links as well as the non-working ‘Production’ links. Then, as designed, network automation began issuing commands consistent with standard mitigations to turn off the LAGs, to prevent impact due to potential link imbalance.

Although regions are built with extensive redundancy, enough links were taken out of service to cause congestion and impact to customer traffic. As soon as the network automation systems reached our operational safety thresholds, they determined it was unsafe to continue taking links out of service and stopped their activity, as designed. This safety mechanism prevented the impact from worsening.

How did we respond?

Packet loss detection alerts fired at 03:03 UTC and notified our on-call engineers that help was needed to recover the network. Engineers scanned the network to identify links that were incorrectly removed from service and distinguish them from links that were correctly removed from service for being unhealthy. Although these mitigation efforts were in progress by 04:45 UTC, the packet loss worsened from 02:34 to 05:45 as traffic in the region increased with the workday starting in the EMEA region.

Engineers placed the healthy links back into service and ensured that the network automation would not remove them again. From 05:45 UTC, the network recovered progressively throughout the remaining incident period, as the links were restored. The network was substantially recovered by 06:12 UTC, and loss rates returned to normal by 07:25 UTC. Other services dependent on the network recovered shortly thereafter. This capacity augment project in West Europe is halted and will only be resumed after a thorough review and update of the current process. 

How are we making incidents like this less likely or less impactful?

The two key factors that contributed to this incident were (1) a capacity augment project carried out where the state of the new to-be-added links was set to “Production” within our network automation system, and our system detected these links as inoperative and initiated mitigation actions; and (2) the system that acts as a safety gate for the auto-mitigation actions estimated the degree of impact incorrectly, and allowed the auto-mitigation systems to take down too much capacity on the network, which led to congestion and packet drops. As a result of these learnings, our repair items include the following:

• We have enhanced our alerts to accelerate packet loss detection and decrease time to detect (Completed)
• We are enhancing our roll-back capability for our on-call engineers to speed up recovery (Estimated completion: August 2023).
• We will auto-declare regional incidents to ensure customers get notified more quickly (Estimated completion: August 2023).
• We are making improvements to the procedure that was used for this specific capacity augment scenario to ensure non-working links are not set to an In-Production state (Estimated completion: September 2023).
• We are making the safety gates used by our auto-mitigations more stringent with respect to network utilization and capacity headroom (Estimated completion: September 2023).
• We are making improvements to the procedure that was used for this specific capacity augment scenario to ensure non-working links are not set to a Production state (Estimated completion: October 2023).

How can customers make incidents like this less impactful?

• Consider a multi-region geodiversity strategy for your mission-critical workloads, to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/training/modules/design-a-geographically-distributed-application/ and https://learn.microsoft.com/azure/architecture/patterns/geodes
• Applications that use exponential-backoff in their retry strategy may have seen success, as an immediate retry during the short intervals of high packet loss may have also seen high packet loss, but a retry conducted during periods of lower loss would have likely succeeded. For more details on retry patterns, refer to https://learn.microsoft.com/azure/architecture/patterns/retry
• More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://docs.microsoft.com/azure/architecture/framework/resiliency
• Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey: https://aka.ms/AzPIR/VLB8-1Z0