Deployment creation/management impacted in GCP London
Incident Report for ESS (Public)
Resolved
Affected customer deployments have been restored and all cluster administration tasks are operational. Closing incident.
Posted Jul 20, 2022 - 23:59 UTC
Update
Remediation has completed for the majority of deployments in this region affected by the loss of the europe-west-2a availability zone. Engineers are continuing to work on the few remaining deployments that are taking longer to recover.
Posted Jul 20, 2022 - 17:36 UTC
Update
Remediation continues for the affected instances from this incident, including replacing instances that remain unhealthy in the europe-west-2a availability zone. We will post another update within 1 hour.
Posted Jul 20, 2022 - 16:28 UTC
Update
We are continuing to work on remediating any affected instances from this incident, including replacing instances that remain unhealthy in the europe-west-2a availability zone. We will post another update within 1 hour.
Posted Jul 20, 2022 - 15:22 UTC
Update
We are currently working on remediating any affected instances from this incident, including replacing any instances that remain unhealthy in the europe-west-2a availability zone. We will another update within 1 hour.
Posted Jul 20, 2022 - 14:29 UTC
Update
The gcp-europe-west2-a availability zone is still encountering issues. Customers running in non-HA configurations in this availability zone could be impacted. Create and update operations is this region could also experience issues while this issue persists. We are continuing to monitor the situation and will update as new information becomes available. For more detail from Google, please see https://status.cloud.google.com/regional/europe.
Posted Jul 20, 2022 - 01:07 UTC
Update
Google has posted that the cooling system in the europe-west2 availability zone should be restored by 23:30 UTC, after which our engineers can begin work on remediation of affected instances. We will post another update at 23:30 UTC
Posted Jul 19, 2022 - 21:48 UTC
Update
Google is continuing to work on the loss of the europe-west2-a availability zone in GCP London. At the moment we do not have an ETA for when this will be resolved but we are continuing to monitor the situation. We will post another update within 1 hour.
Posted Jul 19, 2022 - 21:12 UTC
Update
We are continuing to monitor status on Google's remediation of the issue with the europe-west2-a availability zone. We will post another update within 1 hour
Posted Jul 19, 2022 - 20:11 UTC
Identified
Google is continuing to investigate an issue with datacenter cooling that is affecting instances in the europe-west2-a availability zone.
Posted Jul 19, 2022 - 18:54 UTC
Update
Google has posted a status message, https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2#guoaCiLrsny92g8eoeyR, confirming there is an ongoing outage with one availability zone, europe-west2-a in this region. As such, we are upgrading this incident to major and are following up with Google.
Posted Jul 19, 2022 - 17:51 UTC
Investigating
We are currently investigating an issue with the backend coordination layer in GCP London (gcp-europe-west2) that may impact the ability to create, update, restart, or delete deployments in this region.
Posted Jul 19, 2022 - 17:44 UTC
This incident affected: GCP London (europe-west2) (Deployment orchestration (Create/Edit/Restart/Delete): GCP europe-west2).