Skip to Main Content

March 2023

6

Watch our 'Azure Incident Retrospective' video about this incident:

What happened?

Between 03:50 UTC and 17:55 UTC on 6 March 2023, a subset of customers using Azure Storage may have experienced greater than expected throttling when performing requests against Storage resources located in the West Europe region. Azure services dependent on Azure Storage may also have experienced intermittent failures and degraded performance due to this issue. These included Azure Automation, Azure Arc enabled Kubernetes, Azure Bastion, Azure Batch, Azure Container Apps, Azure Data Factory (ADF), Azure ExpressRoute \ ExpressRoute Gateways, Azure HDInsight, Azure Key Vault (AKV), Azure Logic Apps, Azure Monitor, and Azure Synapse Analytics. 

What went wrong and why?

Azure Storage employs a throttling mechanism that ensures storage account usage remains within the published storage account limits (for more details, refer to ). This throttling mechanism is also used to protect a storage service scale unit from exceeding the overall resources available to the scale unit. In the event that a scale unit reaches such limits, the scale unit employs a safety protection mechanism to throttle accounts that are deemed to contribute to the overload, while also balancing load across other scale units.

Configuration settings are utilized to control how the throttling services monitor usage and apply throttling. A new configuration was rolled out to improve the throttling algorithm. While this configuration change followed the usual Safe Deployment Practices (for more details, refer to: ), this issue was found mid-way through the deployment. The change had adverse effects due to the specific load and characteristics on a small set of scale units where the mechanism unexpectedly throttled some storage accounts in an attempt to bring the scale unit to a healthy state.

How did we respond?

Automated monitoring alerts were triggered, and engineers were immediately engaged to assess issues that the service-specific alert reported. This section will provide a timeline for how we responded. Batch started investigating these specific alerts at 04:08 UTC. This investigation showed that there were storage request failures and service availability impact. At 04:40 UTC Storage engineers were engaged to begin investigating the cause. While storage was engaged and investigating with Batch, at 05:30 UTC, automated monitoring alerts were triggered for ADF. The ADF investigation showed there was an issue with underlying Batch accounts. Batch confirmed with ADF that they were impacted by storage failures and are working with storage engineers to mitigate. At this time, storage engineers diagnosed that one scale unit was operating above normal parameters, this included identifying that Batch storage accounts were throttled due to tenant limits. And the action engineers took was to load balance traffic across different scale units. By 06:34 UTC, storage engineers started to do Batch account migration in efforts to mitigate the ongoing issues. 

At 07:15 UTC, automation detected an issue with AKV requests. The AKV investigation showed there was an issue with the underlying storage accounts. Around 09:10 UTC, engineers performed a failover that mitigated the issue for all existing AKV read operations. However, create, read (for new requests), update and delete operations for AKV were still impacted. Around 10:00 UTC, storage engineers correlated the occurrences of the downstream impacted services with the configuration rollout because the scope of the issue expanded to additional scale units. By 10:15 UTC, storage engineers began reverting the configuration change on select impacted scale units. The Batch storage account migration finished around 11:22 UTC, after that, Batch service became healthy at 11:35 UTC. ADF began to recover after Batch mitigation was completed. ADF was fully recovered around 12:51 UTC after accumulated tasks got consumed.

By 16:34 UTC, impacted resources and services were mitigated. Shortly thereafter, engineers scheduled a rollback of the configuration change on all scale units (even ones that were not impacted), declaring mitigation.  

How are we making incidents like this less likely or less impactful?

  • We are tuning our monitoring to anticipate and quickly detect when a storage scale unit might engage its scale unit protection algorithm and employs throttling to storage accounts. These monitors will proactively alert engineers to take necessary actions (Estimated completion: March 2023).
  • Moving forward, we will rollout throttling improvements in tracking-mode first to assess its impact before getting enabled since throttling improvements may react differently to different workload types and load on a particular scale unit (Estimated completion: March 2023).

The above improvement areas will help to prevent/detect storage-related issues across first-party services that are reliant on Azure storage accounts - for example, Batch and Data Factory.

  • Our AKV team is working on improvements to the current distribution of storage accounts across multiple scale units and update storage implementation to ensure that read and write availability can be decoupled when such incidents happen (Estimated completion: May 2023).
  • We continue to expand our AIOps detection system to provide better downstream impact detection and correlation - to notify customers more quickly, and to identify/mitigate impact more quickly (Ongoing).

How can customers make incidents like this less impactful?

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

< View full status history