Skip to Main Content

January 2023

31

What happened?

Between 05:55 UTC on 31 January 2023 and 00:58 UTC on 1 February 2023, a subset of customers using Virtual Machines (VMs) in the East US 2 region may have received error notifications when performing service management operations – such as create, delete, update, scale, start, stop – for resource hosted in this region.

The impact was limited to a subset of resources in one of the region’s three Availability Zones, (physical AZ-02) and there was no impact to Virtual Machines in the other two zones. This incident did not cause availability issues for any VMs already provisioned and running, across any of the three Availability Zones (AZs) – impact was limited to service management operations described above.

Downstream Azure services with dependencies in this AZ were also impacted, including Azure Backup, Azure Batch, Azure Data Factory, Azure Database for MySQL, Azure Database for PostgreSQL, Azure Databricks, and Azure Kubernetes Service.

What went wrong and why?

The East US 2 region is designed and implemented with three Availability Zones. Each of these AZs are further internally sliced into partitions, to ensure that a failure in one partition does not affect the processing in other partitions. Every VM deployment request is processed by one of the partitions in the AZ, and a gateway service is responsible for routing traffic to the right partition. During this incident, one of the partitions associated with physical AZ-02 experienced data access issues at 05:55 UTC on 31 January 2023, because the underlying data store had exhausted an internal resource limit.

While the investigation on the first partition was in progress, at around 11:43 UTC a second partition within the same AZ experienced a similar issue and became unavailable, leading to further impact. Even though this partition became unavailable, the cache service was keeping the partition data and so new deployments in the cluster were succeeding. At 12:04 UTC, the cache service restarted and couldn't retrieve the data, as the target partition was down. Due to a resource creation policy configuration issue, all partitions were required to create a new resource in the gateway service. When the cache service didn’t have the data, this policy resulted in blocking all new VM creations in the AZ. This resulted in additional failures and slowness during the impact timeline, causing failures to downstream services.

How did we respond?

Automated monitoring alerts were triggered, and engineers were immediately engaged to assess the issue. As the issue was being investigated, attempts to auto-recover the partition failed to succeed. By 16:45 UTC, engineers had determined and started implementing mitigation steps on the first partition.

At 17:49 UTC, the first partition was successively progressing with recovery, and engineers decided to implement the recovery steps on the second partition. During this time, the number of incoming requests for VM operations continued to grow. To avoid further impact and failures to additional partitions, at 19:15 UTC Availability Zone AZ-02 was taken out of service for new VM creation requests. This ensured that all new VM creation requests would succeed, as they were automatically redirected to the other two Availability Zones.

By 23:45 UTC on 31 January 2023, both partitions had completed a successful recovery. Engineers continued to monitor the system and, when no further failures were recorded beyond 00:48 UTC on 1 February 2023, the incident was declared mitigated and Availability Zone AZ-02 was fully opened for new VM creations.

How are we making incidents like this less likely or less impactful?

  • As an immediate measure, we scanned all partitions across the three Availability Zones to successfully confirm that no other partitions were at risk of the same issue (Completed).
  • We have improved our telemetry to increase incident severity automatically if the failure rate increases beyond a set threshold (Completed).
  • We have also added automated preventive measures to avoid data store resource exhaustion issues from happening again (Completed).
  • We are working towards removing the cross-partition dependencies for VM creation (Estimated completion: April 2023).
  • We are adding a new capability to redirect deployments to other AZs based on known faults from a specific AZ (Estimated completion: May 2023).

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey: 

< View full status history