Skip to Main Content

July 2023

6

Watch our 'Azure Incident Retrospective' video about this incident:

What happened? 

Between 23:15 UTC on 6 July 2023 and 09:00 UTC on 7 July 2023, a subset of data for Azure Monitor Log Analytics and Microsoft Sentinel failed to ingest. Additionally, platform logs gathered via Diagnostic Settings failed to route some data to customer destinations such as Log Analytics, Storage, Event Hub and Marketplace. These failures were caused by a deployment of a service within Microsoft, with a bug that caused a much higher than expected call volume that overwhelmed the telemetry management control plane. Customers in all regions experienced impact.

Security Operations Center (SOC) functionality in Sentinel may have been impacted. Queries against impacted tables with date range listed above, inclusive of the logs data that we failed to ingest, might have returned partial or empty results. This includes analytics (detections), hunting queries, workbooks with custom queries, and notebooks. In cases where Event or Security Event tables were impacted, incident investigations of a correlated incident may have showed partial or empty results. 

What went wrong and why? 

A code deployment for the Azure Container Apps service was started on 3 July 2023 via the normal Safe Deployment Practices (SDP), first rolling out to Azure canary and staging regions. This version contained a misconfiguration that blocked the service from starting normally. Due to the misconfiguration, the service bootstrap code threw an exception, and was automatically restarted. This caused the bootstrap service to be stuck in a loop where it was being restarted every 5 to 10 seconds. Each time the bootstrap service was restarted, it provided configuration information to the telemetry agents also installed on the service hosts. Each time the configuration information was sent to the telemetry hosts, they interpreted this as a configuration change, and therefore they also automatically exited their current process and restarted as well. Three separate instances of the agent telemetry host, per application host, were now also restarting every 5 to 10 seconds.

Upon each startup of the telemetry agent, the agent immediately contacted the telemetry control plane to download the latest version of the telemetry configuration. Normally this is an action that would take place one time every several days, as this configuration would be cached on the agent. However, as the deployment of the Container Apps service progressed, several hundred hosts now had their telemetry agents requesting startup configuration information from the telemetry control plane every 5-10 seconds. The Container Apps team detected the fault in their deployment on 6 July 2023, stopped the original deployment before it was released to any production regions, and started a new deployment of their service in the canary and staging regions to correct the misconfiguration.

However, the aggregate rate of requests from the services that received the build with the misconfiguration exhausted capacity on the telemetry control plane. The telemetry control plane is a global service, used by services running in all public regions of Azure. As capacity on the control plane was saturated, other services involved in ingestion of telemetry, such as the ingestion front doors and the pipeline services that route data between services internally, began to fail as their operations against the telemetry control plane were either rejected or timed out. The design of the telemetry control plane as a single point of failure is a known risk, and investment to eliminate this risk has been underway in Azure Monitor to design this risk out of the system.

How did we respond?

The impact on the telemetry control plane grew slowly and did not create problems that were detected until 12:30 UTC on 6 July 2023. When the issues were detected, the source of the additional load against the telemetry control plane was not known, but the team suspected additional load had been created against the control plane and took these actions:

  • 6 July 2023 @ 14:53 UTC – Internal incident bridge created.
  • 6 July 2023 @ 15:56 UTC – ~500 instances of garbage collector service were removed, to reduce load on telemetry control plane
  • 6 July 2023 @ 16:09 UTC – First batch of Node Diagnostics servers were removed, to reduce load on telemetry control plane. This process of removing this type of server continued over the next 10 hours.
  • 6 July 2023 @ 20:20 UTC – Source of anomalously high traffic was identified, and the responsible team was paged to assist.
  • 6 July 2023 @ 23:00 UTC – IP address blocks deployed, to prevent anomalous traffic from hitting telemetry control plane.
  • 6 July 2023 @ 23:15 UTC – External customer impact started, as cached data started to expire.
  • 7 July 2023 @ 01:30 UTC – Three additional clusters were added to telemetry control plane to handle additional load, and we began restarting existing clusters to clear backlogged connections.
  • 7 July 2023 @ 02:19 UTC – Initial customer notification was posted to the Azure Status page, to acknowledge the incident was being investigated while we worked to identify which specific subscriptions were impacted.
  • 7 July 2023 @ 02:45 UTC – An additional three clusters were added to telemetry control plane.
  • 7 July 2023 @ 06:15 UTC – Targeted customer notifications sent via Azure Service Health to customers with impacted subscriptions (sent on Tracking ID XMGF-5Z0).
  • 7 July 2023 @ 09:00 UTC – Incident declared mitigated, as call error rate and call latency against control plane APIs stabilized at typical levels.

How are we making incidents like this less likely or less impactful? 

We know customer trust is earned and must be maintained, not just by saying the right thing but by doing the right thing. Data retention is a fundamental responsibility of the Microsoft cloud, including every engineer working on every cloud service. We have learned from this incident and are committed to the following improvements:

  • We have ensured that our telemetry control plane services are now running with additional capacity (Completed)
  • We have created additional alerting on certain metrics that indicate critical, unusual failure patterns in API calls (Completed)
  • We will be adding new positive caching and negative caching to the control plane, to reduce load on backing store (Estimated completion: September 2023)
  • We are putting in place additional throttling and circuit breaker patterns to our core telemetry control plane APIs (Estimated completion: September 2023)
  • In the longer term, we are creating isolation between internal-facing and external-facing services using the telemetry control plane (Estimated completion: December 2023) 

How can customers make incidents like this less impactful?

Note that the Azure Monitoring Agent (AMA) provides more advanced collection and ingestion resilience capabilities (such as caching, buffering and retries) than the legacy Microsoft Monitoring Agent (MMA). Customers who have not yet completed their migration from MMA to AMA, would benefit from accelerating and completing the migration - for use cases required by them and supported in AMA. For more details:

Customers using Microsoft Sentinel can consider the following compensating steps:

  • Identify high priority assets, log sources that cover those assets, and detection or hunting logic normally applied to those assets and logs.
  • If possible, run one-time queries at the source, for the indicated date range where ingestion was impacted - based on the prioritized assets, logs, and logic. It may also be useful to run the same queries for week prior to the incident, compare, and look for differences.
  • Alternatively, to the extent the collection architecture allows for it, re-ingest data from the source into Sentinel, and run those one-time queries in Sentinel.

Finally, consider ensuring that the right people in your organization will be notified about any future service issues - by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more:

How can we make our incident communications more useful?

You can rate this PIR and provide any feedback using our quick 3-question survey:

< View full status history