Skip to Main Content

August 2022

30

What happened?

Between 06:00 UTC on 30 Aug 2022 and 16:00 UTC on 31 Aug 2022, customers running Ubuntu 18.04 (bionic) Virtual Machines (VMs) who had Ubuntu Unattended-Upgrades enabled received a systemd version that resulted in Domain Name System (DNS) resolution errors. This issue was confined to Ubuntu version 18.04, but impacted all Azure regions including public and sovereign clouds.

Downstream Azure services that rely on impacted Ubuntu VMs also experienced impact during this window – including Azure Kubernetes Service (AKS), Azure Monitor, Application Insights, Log Analytics and Microsoft Sentinel. AKS customers could have experienced pod creation errors such as ImagePullBackoff as kubelet was unable to resolve DNS names of container registry. Customers may have experienced an inability to access Azure Monitor, Application Insights, Log Analytics, and/or Microsoft Sentinel log data, and may have noticed missed or delayed Log Search alerts and/or Microsoft Sentinel alerts.

What went wrong, and why?

At 06:00 UTC on 30 August 2022, a Canonical Ubuntu security update was published – so Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrade enabled started to download and install the new packages, including systemd version 237-3ubuntu10.54. This led to a loss of their DNS configurations due to a race-condition bug: .

The manifestation of this bug was triggered due to the combination of this and a previous update. This bug only affects systems using a driver name to identify the proper Network Interface Card (NIC) in their network configuration, which is why this issue impacted Azure uniquely and not other major cloud providers. This resulted in DNS resolution failures and network connectivity issues for Azure VMs running Ubuntu 18.04 (bionic). As a result, other services dependent on these VMs were impacted by the same DNS resolution issues.

When unattended-upgrades are enabled, security updates are automatically downloaded and applied once per day by default. Considering their criticality, security updates like these do not go through our Safe Deployment Practices (SDP) process. However, we are reviewing this process to ensure that we minimize customer impact during incidents like these.

How did we respond?

Multiple Azure teams detected the issue shortly after the packages were published via production alerts, including our AKS and Azure Container Apps service teams. Upon investigation, we identified the root cause as the bug in Ubuntu mentioned above, and began engaging other teams to explore appropriate mitigations. During this time, incoming customer support cases describing the issue validated that the issues were limited to the Ubuntu versions described above.

There were multiple mitigation and remediation steps, several of which were completed in partnership with Canonical / Ubuntu:

  • This bug and a potential fix have been highlighted on the Canonical / Ubuntu website, which we encouraged impacted customers to read (linked above).
  • For impacted Azure VM instances, we recommended that customers reboot the VM(s) or, if reboot was not an option, run a script to fix it. Azure provided a template script but encouraged customers to test and modify the script as needed before applying: .
  • For the impact to AKS nodes, our AKS team developed an automatic detection and remediation solution then rolled this out across all regions. This resolved the vast majority of customers, others required manual mitigations through support.
  • Additional downstream impact to other services was addressed through similar remediation steps to address the bug in their specific Ubuntu version, so some of these services recovered prior to the final mitigation time above.

How are we making incidents like this less likely or less impactful?

Already completed:

  • Improved monitoring of AKS data plane for alerts on the uptick of errors.
  • Reviewed AKS monitoring algorithms to help ensure detection and alerting upon similar VM errors that were experienced in this scenario.
  • Established an improved escalation path to Canonical during outages and include that in our internal Technical Service Guides (TSGs).

Short term:

  • AKS will take full control over the security patch mechanism, versus shared control with Canonical today. This includes additional testing and controlled release of these patches done directly by AKS. (Estimated completion: December 2022).

Medium term:

  • For IaaS VMs, we are working to engage with Canonical to run dedicated tests on proposed packages before they are published for Azure users.

Longer-term:

  • AKS will establish a process with Canonical to close the testing gap for the upgrade scenario. (Estimated completion: March 2023).
  • AKS will provide customers with maintenance window control and environment staging for these patches when they are deemed safe to release. (Estimated completion: December 2023).

How can our customers and partners make incidents like this less impactful?

How can we make our incident communications more useful?

Microsoft is piloting this “PIR” template as a potential replacement for our “RCA” (Root Cause Analysis) template.

You can rate this PIR and provide any feedback using our quick 3-question survey:

< View full status history