Deployment creation and update failures related to capacity

Incident Report for ESS (Public)

Resolved

This incident has been resolved.

Posted Jul 22, 2023 - 21:26 UTC

Monitoring

Our engineers have fixed an issue with a backend system that affected our ability to allocate new virtual machine capacity. We are currently retrying any deployment plans that failed due to this issue.

Posted Jul 22, 2023 - 17:49 UTC

Update

Our engineers are continuing to work on implementing fixes for this issue.
Next update within one hour.

Posted Jul 22, 2023 - 17:10 UTC

Identified

The issue has been identified and a set of fixes are being implemented.
Next update within one hour.

Posted Jul 22, 2023 - 16:06 UTC

Investigating

We are investigating failures to allocate new virtual machine capacity. Currently impact is limited to a subset of customers experiencing errors creating new deployments, or changing cluster topology and configuration. Our next update will be in one hour.

Posted Jul 22, 2023 - 15:09 UTC

This incident affected: GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2), GCP Los Angeles (us-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west2), Azure Singapore (azure-southeastasia) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-southeastasia), GCP São Paulo (southamerica-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP southamerica-east1), Azure Texas (azure-southcentralus) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-southcentralus), AWS Tokyo (ap-northeast-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-northeast-1), AWS Ireland (eu-west-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-1), Azure London (azure-uksouth) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-uksouth), Azure New South Wales (azure-australiaeast) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-australiaeast), AWS Central (ca-central-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ca-central-1), GCP Oregon (us-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1), Azure Virginia (azure-eastus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus2), GCP Montreal (northamerica-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP northamerica-northeast1), GCP Taiwan (asia-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-east1), GCP Netherlands (europe-west4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west4), Global services (Deployment orchestration (Create/Edit/Restart/Delete)), AWS Ohio (us-east-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-2), AWS Bahrain (me-south-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS me-south-1), GCP Tokyo (asia-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast1), Azure Netherlands (azure-westeurope) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westeurope), AWS Paris (eu-west-3) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-3), AWS Sydney (ap-southeast-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-southeast-2), AWS Oregon (us-west-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-2), AWS N. California (us-west-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-west-1), AWS London (eu-west-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-west-2), GCP N. Virginia (us-east4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east4), AWS Seoul (ap-northeast-2) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-northeast-2), Azure São Paulo (azure-brazilsouth) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-brazilsouth), GCP Iowa (us-central1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1), AWS Frankfurt (eu-central-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-central-1), GCP Sydney (australia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP australia-southeast1), Azure Washington (azure-westus2) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-westus2), GCP Singapore (asia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-southeast1), GCP Seoul (asia-northeast3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast3), Azure johannesburg (azure-southafricanorth) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-southafricanorth), AWS São Paulo (sa-east-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS sa-east-1), GCP Mumbai (asia-south1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-south1), Azure Tokyo (azure-japaneast) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-japaneast), GCP Finland (europe-north1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-north1), Azure Iowa (azure-centralus) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-centralus), Azure France (azure-francecentral) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-francecentral), Azure Toronto (azure-canadacentral) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-canadacentral), AWS Mumbai (ap-south-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-south-1), AWS Hong Kong (ap-east-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-east-1), Azure Ireland (azure-northeurope) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-northeurope), AWS Singapore (ap-southeast-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS ap-southeast-1), AWS N. Virginia (us-east-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS us-east-1), GCP South Carolina (us-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east1), Azure Virginia (azure-eastus) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-eastus), Azure Pune (azure-centralindia) (Deployment orchestration (Create/Edit/Restart/Delete): Azure azure-centralindia), AWS Cape Town (af-south-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS af-south-1), and AWS Milan (eu-south-1) (Deployment orchestration (Create/Edit/Restart/Delete): AWS eu-south-1).