Capacity provisioning issues in GCP regions

Incident Report for ESS (Public)

Resolved

The issue has been resolved. Operations are back to normal.

Posted Mar 04, 2024 - 08:16 UTC

Monitoring

The fix has been implemented and capacity is correctly provisioning again. We'll continue to monitor for a few hours before closing the incident.

Posted Mar 04, 2024 - 03:10 UTC

Update

Updating status to reflect that all GCP regions are impacted by this problem.

Posted Mar 04, 2024 - 02:44 UTC

Identified

Cause has been identified and a fix is being tested.

Posted Mar 04, 2024 - 02:40 UTC

Investigating

We are aware of the issue with capacity provisioning in GCP asia-southeast1. Our teams are investigating.

Posted Mar 04, 2024 - 01:28 UTC

This incident affected: GCP Frankfurt (europe-west3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west3), GCP Belgium (europe-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west1), GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2), GCP Los Angeles (us-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west2), GCP São Paulo (southamerica-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP southamerica-east1), GCP Oregon (us-west1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-west1), GCP Montreal (northamerica-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP northamerica-northeast1), GCP Taiwan (asia-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-east1), GCP Netherlands (europe-west4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west4), GCP Tokyo (asia-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast1), GCP N. Virginia (us-east4) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east4), GCP Iowa (us-central1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-central1), GCP Sydney (australia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP australia-southeast1), GCP Singapore (asia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-southeast1), GCP Seoul (asia-northeast3) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast3), GCP Mumbai (asia-south1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-south1), GCP Finland (europe-north1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-north1), and GCP South Carolina (us-east1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP us-east1).