Skip to Main Content

February 2022

12

Summary of Impact: Between 11:45 UTC on 12 Feb 2022 and 11:43 UTC on 15 Feb 2022, a limited subset of customers using SQL Databases or Cosmos DB experienced database unavailability and may have seen errors when connecting to their database instances. This issue affected a specific generation of hardware hosting SQL and Cosmos DB resources in six regions.

Root Cause: Our telemetry has shown that a subset of nodes running our newest hardware generation hosting SQL DB and Cosmos DB experienced a loss of connectivity to the network control plane starting on 12 Feb 2022. Our newest hardware generation uses a new and optimized network control plane and data plane designed to improve performance and reduce irregularities and latency, with a new control plane secured by TLS authentication. The TLS certificates that secure this channel are rotated regularly and expire within a short timespan for security purposes. A race condition in the underlying Remote Procedure Call (RPC) mechanism caused the network control plane channel in some cases to not pick up the rotated certificate, leading to connectivity failures once the certificate expired. 

Routine maintenance in the general Azure compute fleet had fixed this issue with a code update in January. However, based on telemetry, we did not believe this was a significant risk to SQL and Cosmos DB environments, which were scheduled to receive the code update later in February. On 12 Feb 2022, a significant number of previously rotated certificates expired simultaneously, causing impact. We first mitigated the impacted nodes and DBs, and then pushed the code update using an emergency process through the SQL and Cosmos DB fleets to make sure the impact would not recur.

Mitigation: Once impact was identified, we recovered nodes with network control plane connectivity loss and brought DBs back to a healthy state over the course of 12 Feb - 15 Feb. While most DBs recovered earlier in the incident, running a full update to get the code fix to all nodes had to proceed slowly to ensure uptime and data integrity, so it took until 14 Feb for Cosmos DB and 15 Feb for SQL DB to complete the rollout.

Next Steps: We apologize for the impact to affected customers. We are continuously taking steps to improve the Microsoft Azure Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to):

  • Improved procedures to help determine if code updates for Azure servers are critical, so we can patch critical data services like SQL and Cosmos DB sooner, instead of taking them at a normally scheduled pace.
  • Faster automated recovery and deployment for critical fixes to SQL and Cosmos DB, to help reduce impact time for critical node fixes.
  • Improved telemetry to help detect certificate expiration risk.

Provide Feedback: Please help us improve the Azure customer communications experience by taking our survey:

< View full status history