Summary of Impact: As early as 22:00 UTC on 08 Jan 2025, we noticed a partial impact to some of the Azure Services in East US2 due to a configuration change in a regional networking service. The configuration change caused inconsistent service state. This could have resulted in intermittent Virtual machine connectivity issues or failures in allocating resources or communicating with resources in the region. The services impacted include Azure Databricks, Azure Container Apps, Azure Function Apps, Azure App Service, SQL Managed Instances, Azure Data Factory, Azure Container Instances, PowerBI, VMSS, PostgreSQL flexible servers etc. Customers using resources with Private Endpoint NSG communicating with other services would also be impacted.

The impact is limited to a single zone in East US2 region. No other regions are impacted by this issue.

Current Status:

As early as 22:00 UTC on 08 Jan 2025, service monitoring alerted us to a networking issue in East US2 impacting multiple services. As part of the investigation, it was identified that a network configuration issue in one of the zones resulted in three of the Storage partitions going unhealthy. As an immediate remediation measure, traffic was re-routed away from the impacted zone, which brought some relief to the non-zonal services, and helped with newer allocations. However, services that sent zonal requests to the impacted zone continued to be unhealthy. Some of the impacted services initiated their own Disaster Recovery options to mitigate some of them.

Additional workstreams to rehydrate the impacted zone by bringing back the impacted partitions to a healthy state have been ongoing as per the plan. To avoid any further impact, we are validating the fix on one of the partitions, and once that is confirmed, the mitigation will be applied to the other unhealthy partitions as well. We have completed the validation process successfully for one of the partitions and are working on applying the mitigation to all the partitions. Once the mitigation is applied, we intend to complete additional validations before bringing the partitions online.

We do not have an ETA available at this time, but we expect to be able to share more details on our progress in the next update. We continue to advise customers to execute Disaster Recovery to expedite recovery of their impacted services. Customers that have already failed out of the region should not fail back until this incident is fully mitigated. The next update will be provided in 1 hour or as events warrant.

For customers impacted due to Private Link, a patch was applied, and we confirm dependent services should be available.

We have been able to confirm that customers impacted by Azure Databricks, App Services multi-tenant, Azure Function Apps, Logic Apps, and Azure Synapse should start seeing some recovery.

Ethan Robish // As promised in my previous post, part 1, this post shows how to place a tracking bug in a native .xlsx file.  Full credit for this method […]

The post Bugging Microsoft Files: Part 2 – Xlsx Files using Microsoft Excel appeared first on Black Hills Information Security, Inc..

Ethan Robish // If you’re familiar with ADHD and Web Word Bugs, you likely already know the method to create web tracking software using .html files renamed as .doc files. […]

The post Bugging Microsoft Files: Part 1 – Docx Files using Microsoft Word appeared first on Black Hills Information Security, Inc..

Ethan Robish // Here’s a short intro for anyone not familiar with ssh config files, which are usually located at ~/.ssh/config As an example, you have ssh running on port […]

The post SSH Config Files appeared first on Black Hills Information Security, Inc..

Kelsey Bellew // Maybe you don’t know what Direct Object References mean, if you Google it, you’d get this: This description uses the words “direct”, “object” and “reference” to describe a […]

The post Let’s Talk About Direct Object References appeared first on Black Hills Information Security, Inc..