Investigating failures to multiple deployment hosts in many Azure regions
Incident Report for ESS (Public)
Resolved
All issues with the Azure hosts are now resolved and deployments should be back to normal. In case you see any issues on your end, please reach out to our support team.
Posted Jun 27, 2022 - 03:30 UTC
Update
Small number of hosts in Australia East and East US 2 continue to be affected impacting a small number of deployments. The team is continuing to communicate with Azure and provide support to impacted deployments. Azure planned maintenance is over. We'll provide another update in 2 hours.
Posted Jun 26, 2022 - 23:27 UTC
Update
Hosts in Australia East, Southeast Asia, East US 2, South Central US and West US 2 continue to be affected, still impacting a small number of deployments. The team is continuing to communicate with Azure and provide support to impacted deployments. We'll provide another update in 2 hours.
Posted Jun 26, 2022 - 21:15 UTC
Update
At this time, hosts in Australia East, Southeast Asia, East US 2, South Central US and West US 2 are still being affected, impacting a small number of deployments. The team is continuing to communicate with Azure and provide support to impacted deployments. Next update in 1 hour.
Posted Jun 26, 2022 - 20:12 UTC
Update
Several additional hosts in Australia East, Southeast Asia, East US 2, South Central US and West US 2 have been affected, impacting a small number of deployments. We are still in communication with Azure, and our team is working on providing support to the affected deployments. Next update will be in 1 hour.
Posted Jun 26, 2022 - 19:09 UTC
Update
The situation is stable and there are no known problems.
We'll be in communication with Azure for as long as their maintenance continues, and our team is monitoring the situation for any further remediation needed.

Next update in 4 hour.
Posted Jun 26, 2022 - 15:05 UTC
Monitoring
At this time, we have resolved all issues with affected deployments hosts.

We are in communication with Azure, and our team is monitoring for any further remediation needed.

Next update in 4 hour.
Posted Jun 26, 2022 - 10:10 UTC
Identified
At this time, the impact is limited to a small number of deployments in Australia East, Southeast Asia, East US 2, South Central US and West US 2.
We are in communication with Azure, and our team is working on providing support to the affected deployments, and proceed with our remediation plan.

We'll provide more details in 1 hour.
Posted Jun 26, 2022 - 09:07 UTC
Investigating
We're investigating a situation in many Azure regions (including but not limited to Australia East, Southeast Asia, East US 2, South Central US) where multiple deployment hosts were terminated in parallel due to Azure maintenance operations. We're currently trying to determine customer impact. We're in contact with Azure and we'll provide more details in 1 hour.
Posted Jun 26, 2022 - 08:08 UTC
This incident affected: Azure Singapore (azure-southeastasia) (Deployment hosts: Azure azure-southeastasia), Azure New South Wales (azure-australiaeast) (Deployment hosts: Azure azure-australiaeast), Azure Virginia (azure-eastus2) (Deployment hosts: Azure azure-eastus2), Azure Washington (azure-westus2) (Deployment hosts: Azure azure-westus2), and Azure Texas (azure-southcentralus) (Deployment hosts: Azure azure-southcentralus).