Degraded Deployment Orchestration in GCP Singapore
Incident Report for ESS (Public)
Resolved
The incident has been resolved. We have not observed any network connectivity problems in the last 6+ hours.
Posted Aug 23, 2024 - 12:06 UTC
Identified
We have identified an underlying network connectivity issue with elevated level of packet loss and significantly increased latency.
We are working with our cloud service providers towards resolving these underlying network issues.

We will provide an update within the next 2 hours or earlier if the situation has changed.
Posted Aug 22, 2024 - 16:51 UTC
Update
We are currently investigating an issue with the deployment orchestration in GCP Singapore (asia-southeast1), GCP Mumbai (asia-south1) and GCP Tokyo (asia-northeast1). Running deployments are still working fine, but customers can experience slowness or failed interactions for the creation of new deployments, updates to deployments or terminations of deployments.

We will provide an update in the next hour.
Posted Aug 22, 2024 - 14:40 UTC
Investigating
We are currently investigating an issue with the deployment orchestration in GCP Singapore (asia-southeast1). Running deployments are still working fine, but customers can experience slowness or failed creation of new deployments, updates to deployments or terminations.
We will provide an update in the next 30 minutes.
Posted Aug 22, 2024 - 14:01 UTC
This incident affected: GCP Tokyo (asia-northeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-northeast1), GCP Singapore (asia-southeast1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-southeast1), and GCP Mumbai (asia-south1) (Deployment orchestration (Create/Edit/Restart/Delete): GCP asia-south1).