Skip to Main Content

February 2023

7

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?  

Between 20:19 UTC on 7 February 2023 and 04:30 UTC on 9 February 2023, a subset of customers with workloads hosted in the Southeast Asia and East Asia regions experienced difficulties accessing and managing resources deployed in these regions.

One Availability Zone (AZ) in Southeast Asia experienced cooling failure. Infrastructure in that zone was shutdown to protect data and infrastructure. This led to failures in accessing resources and services hosted in that zone. However, this zone failure also resulted in two further unexpected failures. Firstly, regional degradation for some services and secondly, services designed to support failover to other regions or zones did not work reliably. 

The services which experienced degradation in the region included App Services, Azure Kubernetes Service, Databricks, API Management, Application Insights, Backup, Cognitive Services, Container Apps, Container Registry, Container Service, Cosmos DB, NetApp Files, Network Watcher, Notification Hubs, Purview, Redis Cache, Search, Service Bus, SignalR Service, Site Recovery, SQL Data Warehouse, SQL Database, Storage, Synapse Analytics, Universal Print, Update Management Center, Virtual Machines, Virtual WAN, Web PubSub – as well as the Azure portal itself, and subsets of the Azure Resource Manager (ARM) control plane services.

BCDR Services that were designed to support regional or zonal failover that didn’t work as expected included Azure Site Recovery, Azure Backup and Storage accounts leveraging Geo Redundant Storage (GRS).

 

What went wrong and why? 

In this PIR we will break down what went wrong into 3 parts. The first part will focus on the failures that led to the loss of an availability zone. The second part will focus on the unexpected impact to the ARM control plane, and BCDR services. Lastly, we will focus on the extended recovery for the SQL Database service.

Chiller failures in a single Availability Zone

At 15:17 UTC on 7 February 2023, a utility voltage dip event occurred on the power grid, affecting a single Availability Zone in the Southeast Asia region. Power management systems managed the voltage dip as designed, however a subset of chiller units that provide cooling to the datacenter tripped and shut down. Emergency Operational Procedures (EOP) were performed as documented for impacted chillers but were not successful. Cooling capacity was reduced in the facility for a prolonged time, and despite efforts to stabilize by shutting down non-critical infrastructure, temperatures continued to rise in the impacted datacenter. At 20:19 UTC on 7 February 2023, infrastructure thermal warnings from components in the datacenter directed a shutdown on critical compute, network and storage infrastructure to protect data durability and infrastructure health. This resulted in loss of resource and service availability in the impacted zone in Southeast Asia.

Maximum cooling capacity for this facility incorporates 8 chillers from 2 different suppliers. Chillers 1 to 5 are from supplier A providing 47% of the cooling capacity, chillers 6, 7 and 8 from supplier B make up the remaining 53%. Chiller 5 was offline for planned maintenance. After the voltage dip, chiller 4 continued to run and 1, 2 and 3 were started, but could not provide the cooling necessary to stabilize the temperatures which had already increased during the restart period. These 4 units were operating as expected, but 6, 7, and 8 were unable to be started. This was not the expected behavior of these chillers, which should automatically restart when tripped by a power dip. The chillers also failed a manual restart as part of the EOP (emergency operating procedure) execution.

This fault condition is currently being investigated by the OEM chiller vendor. Preliminary investigations point to the compressor control card which ceased to respond after the power dip, inhibiting the chillers from successfully restarting. From the chiller HMI alarm log, a “phase sequence alarm” was detected. To normalize this alarm, the compressor control card had to be hard reset requiring a shut down, leaving it powered off for at least 5 minutes to drain the internal capacitors before powering the unit on. The current EOP did not list out these steps. This is addressed later in the PIR.

Technicians from Supplier B were dispatched onsite to assist. By the time the chiller compressor control card was reset with their assistance, the chilled water loop temperatures exceeded the threshold of 28 degrees Celsius, causing a lock out of the restart function to protect the chiller from damage. This condition is referred to as a thermal lockout. To reduce the chilled water loop temperature, augmented cooling was required to be brought online and thermal loads had to be reduced by shutting down infrastructure, as previously stated. This successfully reduced the chilled water loop temperature below the thermal lockout threshold and enabled the restart function for chillers 6, 7, and 8. Once the chillers were restarted, temperatures progressed toward expected levels and, with all units recovered, by 14:00 UTC on 8 February 2023, temperatures were back within our standard operational thresholds.

With confidence that temperatures were stable, we then began to restore power to the affected infrastructure, starting a phased process to first bring storage scale units back online. Once storage infrastructure was verified healthy and online, compute scale units were then powered up. As the compute scale units became healthy, virtual machines and other dependent Azure services recovered. Most customers should have seen platform service recovery by 16:30 UTC on 08 February, however a small number of services took an extended time to fully recover, completing by 04:30 UTC on 9 February 2023. 

Unexpected impact ARM and BCDR services.

While infrastructure from the impacted AZ was powered down, services deployed and running in a zone resilient configuration were largely unimpacted by the shutdown of power in the zone. However multiple services may have experienced some form of degradation, these services are listed above. This was due to dependencies on ARM, which did experience unexpected impact from the loss of a single AZ.

In addition, three services that are purpose-built for business continuity and disaster recovery (BCDR) planning and execution, in both zonal and regional incidents, also experienced issues, and will be discussed in detail:

Azure Site Recovery (ASR) - Helps ensure business continuity by keeping business apps and workloads running during outages. Site Recovery replicates workloads running on physical and virtual machines (VMs) from a primary site to a secondary location. When an incident occurs at a customer’s primary site, a customer can fail over to a secondary location. After the primary location is running again, customers can fail back.

Azure Backup – Allows customers to make backups that keeps data safe and recoverable.

Azure Storage – Storage accounts configured for Geo Redundant Storage (GRS) to provide resiliency for data replication.

Impact to the ARM service:

As the infrastructure of the impacted zone was shutdown, the ARM control plane service (which is responsible for processing all service management operations) in Southeast Asia lost access to a portion of its metadata store hosted in a Cosmos DB instance that was in the impacted zone. It was expected that this instance was zone resilient, but due to a configuration error, local to the Southeast Asia region, it was not. Inability to access this metadata resulted in failures and contention across the ARM control plane in that region. To mitigate impact, the ARM endpoints in Southeast Asia were manually removed from their global service Traffic Manager profile. This was completed at 04:10 UTC on 8 February 2023. Shifting the ARM traffic away from Southeast Asia increased traffic to nearby APAC regions, in particular East Asia, which quickly exceeded provisioned capacity. This exacerbated the impact to include East Asia where customers would have experienced issues managing their resources via ARM. The ARM service in East Asia was scaled-up and by 09:30 UTC on the 8 of February the service was fully recovered. 

 Impact to BCDR services

 A subset of Azure Site Recovery customers started experiencing failures with respect to zonal failovers from the Southeast Asia region and regional failovers to the East Asia region.

 Initial failures were caused by a recent update to a workflow that leveraged an instance of Service Bus as the communication backbone, which was non-zonally resilient and was in the impacted zone. Failures related to workflow dependencies on Service Bus were completely mitigated by 04:45AM UTC on 8 February 2023 by disabling the newly added update in the Southeast Asia region.

 Unfortunately, further failures continued for a few customers due to the above ARM control plane issues that were ongoing. At 09:15 UTC on 8 February 2023, once the ARM control plane issues were mitigated, all subsequent ASR failovers worked reliably.

 The ARM impact caused failures and delays in restoring applications and data using Azure Backup cross region restore to East Asia. A portion of Azure Backup customers in the Southeast Asia region had enabled the cross region restore (CRR) capability which allows them restore their application and data in the East Asia region during regional outages. At 09:00 UTC on the 08 Feb, we started proactively enabling CRR capability for all Azure Backup customers with GRS Recovery Services Vaults (RSV). This was a proactive mitigation action to enable customers had not enabled this capability to recover their application and data in the East Asia region. Observing customer activity, we detected a configuration issue in one of the microservices specific to the Southeast Asia region which caused CRR calls to fail for a subset of customers. This issue was mitigated for this subset, through a configuration update and the cross region application and data restores were fully functioning by 15:10 UTC on 8 February 2023.

 Most customers using Azure Storage with GRS were able to execute customer controlled failovers without any issues. A subset of customers encountered unexpected delays or failures. There were two underlying causes for this. The first was an integrity check that the service performs prior to a failover. In some instances, this check was too aggressive and blocked the failover from completing. The second cause was that storage accounts using hierarchical namespaces could not complete a failover.

Extended Recovery for SQL Database

The impact to Azure SQL Database customers throughout this incident would have fallen into the following categories.

Customers using Multi-AZ Azure SQL were not impacted, except those using proxy mode connection due to one connectivity capacity unit not being configured with zone-resilience.

Customer using active geo-replication with automatic failover were failed out of region and failed back when recovery was complete. These customers had ~1.5 hours of downtime prior to automatic failover completing.

Self-help geo-failover or geo-restore: Some customers who had active geo-replication self-initiated failover, and some performed geo-restore to other regions.  

The rest of the impacted SQL Database customers were impacted from the point of power down and recovered as compute infrastructure was brought back online and Tenant Rings (TR) reformed automatically. One Tenant Ring of capacity for SQL Database required manual intervention to properly reform. Azure SQL Database capacity in a region is divided into Tenant rings. In Southeast Asia there are hundreds of rings. Each ring consists of a group of VMs (10-200) hosting a set of databases. Rings are managed by Azure Service Fabric (SF) to provide availability when we have VM, network or storage failures. For one of the TRs, when power was restored to the compute infrastructure hosting the supporting group of VMs, the ring did not reform and thus was unable to make any of the databases on that ring available. Infrastructure hosting these VMs did not recover automatically due to a BIOS bug. After the compute infrastructure was successfully started, a second issue in heap memory management in SF also required manual intervention to reform the TR and make customer databases available. The final TR became available at 04:30 UTC on the 09 February 2023.

How are we making incidents like this less likely or less impactful?

Firstly, we wanted to provide a status update of the review of facilities readiness in response to the chiller issues.

Our datacenter team engaged with the OEM vendor to fully understand the results and any additional follow-ups to prevent, shorten the duration, and reliably recover after these types of events. (Completed)

We're also implementing updates to existing procedures to include manual (cold) restarting of the chillers as recommended by the OEM vendor. (Completed)

We are also conducting additional training from the OEM to our Datacenter operations personnel to be familiar with the updated procedures. (Completed)

When it comes to executing BCDR plans in the event of single AZ impact, we discussed in the earlier PIR that the impact to ARM in multiple regions inhibited customers' ability to reliably self-mitigate. We will complete a validation of ARM service configuration in the Southeast Asia region to prevent this class of impact from repeating. (Completed)

We will also validate that all instances of Azure Resource Manager service are automatically region and/or zone redundant. (Completed)

We will ensure that there is sufficient capacity for ARM in all regions to manage additional traffic caused by zonal and/or regional failures. (Completed)

We will be conducting a thorough review of all Foundational services for gaps in resiliency models. This work will be ongoing, however the first round will be completed for ARM, Software Load Balancer, SQL DB, Cosmos DB, and Redis Cache by April 2023. You can learn more about our Foundational services here ().

We will be conducting a dependency assessment for all zonal services with an automated daily walk of the dependency graph, documenting points of failures and closing any gaps. [Ongoing]

BCDR critical services will take the following remediations:

We will verify that the configuration issue with Azure Backup cross region restores will be corrected in the impacted region and across all other regions. (Completed)

We will verify that Azure Site Recovery will revert to a zonally resilient workflow model. (Completed)

We will be adjusting safety check thresholds for customer-initiated Storage failover for GRS accounts to ensure failovers aren’t unnecessarily blocked. (Completed)

Beyond VMSS, VMs, Service Fabric and Azure Kubernetes Services, Chaos Studio will be expanding its scenarios so that customers and Azure services can run further service simulations where it is possible to inject faults that emulate zonal failures. (Dec 2023)

We will ensure that our documentation represents the SQL behavior that honors setting changes made by customers when it comes to the SQL services failing back. More information can be found here (). (Completed)

Our communications aim to provide insights into mitigation workstreams for incidents. We understand that for some scenarios this information is critical for customers to make decisions about invoking BCDR plans. We will strive to provide a recoverability expectation and timeline to assist customers in making these decisions for all incidents moving forward. (Completed)

We acknowledge that rapid correlation to impacted or potentially impacted resources in customers' subscriptions was difficult to translate into customer impact and action plans. Work is planned to be completed by December 2023 that addresses improvements in this ability. The below article provides current steps for customers to understand AZ logical mappings across multiple subscriptions ().

If customers have questions about Compliance, Regulations & Standards visit the Service Trust Portal ().

How can we make our incident communications more useful?

You can rate this Post Incident Review (PIR) and provide any feedback using our quick 3-question survey 

< View full status history