Connectivity issues in GCP regions, AWS Sao Paulo, AWS Ohio, and Azure Washington
Incident Report for ESS (Public)
Resolved
This incident has been resolved.
Posted Mar 15, 2022 - 22:46 UTC
Monitoring
We have deployed the fix to all affected regions. If you are still experiencing connectivity issues in one of these regions, please contact Support.
Posted Mar 15, 2022 - 22:11 UTC
Update
The fix has been deployed to all GCP regions. We are continuing to roll out the fix to the remaining affected regions.
Posted Mar 15, 2022 - 21:58 UTC
Update
The fix has been implemented in GCP Iowa (us-central1) and GCP Oregon (us-west1). We are continuing to roll out the fix to the remaining regions.
Posted Mar 15, 2022 - 21:11 UTC
Update
We are continuing with rolling out the component fix to all other affected regions.
Posted Mar 15, 2022 - 20:10 UTC
Update
We are working on implementing the fix in one of the affected regions. We will post another update in 30 minutes.
Posted Mar 15, 2022 - 19:48 UTC
Identified
We've identified the problem in one of the components that impacts connectivity for a subset of deployments. We are currently working on a fix. In the meantime if you are affected please contact support.
Posted Mar 15, 2022 - 18:21 UTC
Update
We continue to investigate the root cause of the incident. We will post the next update in 30 minutes.
Posted Mar 15, 2022 - 17:47 UTC
Update
We have identified further regions experiencing connectivity issues: all GCP regions, AWS Sao Paulo, AWS Ohio, and Azure Washington. We are currently rolling back a recent release in one of the affected regions
Posted Mar 15, 2022 - 16:55 UTC
Update
We are continuing to investigate this issue.
Posted Mar 15, 2022 - 15:47 UTC
Investigating
We are currently investigating reports of connectivity issues with deployments in our GCP Iowa (us-central1) region
Posted Mar 15, 2022 - 15:05 UTC
This incident affected: GCP South Carolina (us-east1) (APM connectivity: GCP us-east1, Elasticsearch connectivity: GCP us-east1, Kibana connectivity: GCP us-east1, Enterprise Search connectivity: GCP us-east1, App Search connectivity: GCP us-east1), GCP Montreal (northamerica-northeast1) (Elasticsearch connectivity: GCP northamerica-northeast1, APM connectivity: GCP northamerica-northeast1, Kibana connectivity: GCP northamerica-northeast1, App Search connectivity: GCP northamerica-northeast1, Enterprise Search connectivity: GCP northamerica-northeast1), Azure Washington (azure-westus2) (Elasticsearch connectivity: Azure azure-westus2, Kibana connectivity: Azure azure-westus2, APM connectivity: Azure azure-westus2, App Search connectivity: Azure azure-westus2, Enterprise Search connectivity: Azure azure-westus2), GCP N. Virginia (us-east4) (Elasticsearch connectivity: GCP us-east4, APM connectivity: GCP us-east4, Kibana connectivity: GCP us-east4, Enterprise Search connectivity: GCP us-east4, App Search connectivity: GCP us-east4), GCP Sao Paulo (southamerica-east1) (APM connectivity: GCP southamerica-east1, Elasticsearch connectivity: GCP southamerica-east1, Kibana connectivity: GCP southamerica-east1, Enterprise Search connectivity: GCP southamerica-east1, App Search connectivity: GCP southamerica-east1), GCP Sydney (australia-southeast1) (Kibana connectivity: GCP australia-southeast1, Elasticsearch connectivity: GCP australia-southeast1, APM connectivity: GCP australia-southeast1, Enterprise Search connectivity: GCP australia-southeast1, App Search connectivity: GCP australia-southeast1), GCP Tokyo (asia-northeast1) (Kibana connectivity: GCP asia-northeast1, APM connectivity: GCP asia-northeast1, Elasticsearch connectivity: GCP asia-northeast1, Enterprise Search connectivity: GCP asia-northeast1, App Search connectivity: GCP asia-northeast1), GCP Finland (europe-north1) (Elasticsearch connectivity: GCP europe-north1, Kibana connectivity: GCP europe-north1, APM connectivity: GCP europe-north1, App Search connectivity: GCP europe-north1, Enterprise Search connectivity: GCP europe-north1), GCP Netherlands (europe-west4) (APM connectivity: GCP europe-west4, Elasticsearch connectivity: GCP europe-west4, Kibana connectivity: GCP europe-west4, Enterprise Search connectivity: GCP europe-west4, App Search connectivity: GCP europe-west4), GCP Singapore (asia-southeast1) (Kibana connectivity: GCP asia-southeast1, APM connectivity: GCP asia-southeast1, Elasticsearch connectivity: GCP asia-southeast1, App Search connectivity: GCP asia-southeast1, Enterprise Search connectivity: GCP asia-southeast1), GCP Mumbai (asia-south1) (Elasticsearch connectivity: GCP asia-south1, Kibana connectivity: GCP asia-south1, APM connectivity: GCP asia-south1, Enterprise Search connectivity: GCP asia-south1, App Search connectivity: GCP asia-south1), GCP London (europe-west2) (APM connectivity: GCP europe-west2, Kibana connectivity: GCP europe-west2, Elasticsearch connectivity: GCP europe-west2, App Search connectivity: GCP europe-west2, Enterprise Search connectivity: GCP europe-west2), GCP Taiwan (asia-east1) (APM connectivity: GCP asia-east1, Kibana connectivity: GCP asia-east1, Elasticsearch connectivity: GCP asia-east1, App Search connectivity: GCP asia-east1, Enterprise Search connectivity: GCP asia-east1), GCP Los Angeles (us-west2) (Kibana connectivity: GCP us-west2, APM connectivity: GCP us-west2, Elasticsearch connectivity: GCP us-west2, App Search connectivity: GCP us-west2, Enterprise Search connectivity: GCP us-west2), AWS Ohio (us-east-2) (APM connectivity: AWS us-east-2, Kibana connectivity: AWS us-east-2, Elasticsearch connectivity: AWS us-east-2, App Search connectivity: AWS us-east-2, Enterprise Search connectivity: AWS us-east-2), GCP Frankfurt (europe-west3) (Elasticsearch connectivity: GCP europe-west3, Kibana connectivity: GCP europe-west3, APM connectivity: GCP europe-west3, App Search connectivity: GCP europe-west3, Enterprise Search connectivity: GCP europe-west3), GCP Iowa (us-central1) (Elasticsearch connectivity: GCP us-central1, Kibana connectivity: GCP us-central1, APM connectivity: GCP us-central1, App Search connectivity: GCP us-central1, Enterprise Search connectivity: GCP us-central1), GCP Belgium (europe-west1) (Kibana connectivity: GCP europe-west1, APM connectivity: GCP europe-west1, Elasticsearch connectivity: GCP europe-west1, App Search connectivity: GCP europe-west1, Enterprise Search connectivity: GCP europe-west1), GCP Oregon (us-west1) (Elasticsearch connectivity: GCP us-west1, Kibana connectivity: GCP us-west1, APM connectivity: GCP us-west1, App Search connectivity: GCP us-west1, Enterprise Search connectivity: GCP us-west1), and AWS São Paulo (sa-east-1) (Elasticsearch connectivity: AWS sa-east-1, Kibana connectivity: AWS sa-east-1, APM connectivity: AWS sa-east-1, App Search connectivity: AWS sa-east-1, Enterprise Search connectivity: AWS sa-east-1).