GCP us-west1 connectivity outage

Incident Report for ESS (Public)

Resolved

While troubleshooting an issue in our proxy layer in this region, an operator removed two proxy instances from the load balancer via the GCP console, triggering an acknowledged UI bug which resulted in all proxies being removed from the load balancer. We apologise for any impact this has caused.

The outage lasted 18 mins (2021-08-29 00:21 UTC until 00:39).

Posted Aug 29, 2021 - 01:48 UTC

Monitoring

We have identified and remediated the root cause. Connectivity should be back to normal.

Posted Aug 29, 2021 - 00:40 UTC

Investigating

We are currently investigating an issue that will be causing connectivity problems to GCP us-west1 deployments.

Posted Aug 29, 2021 - 00:38 UTC

This incident affected: GCP Oregon (us-west1) (Deployment logging: GCP us-west1, Deployment metrics: GCP us-west1, Deployment hosts: GCP us-west1, Elasticsearch connectivity: GCP us-west1, Kibana connectivity: GCP us-west1, APM connectivity: GCP us-west1, App Search connectivity: GCP us-west1, Enterprise Search connectivity: GCP us-west1).