Cluster connectivity issues in us-east-1

Incident Report for Elastic Cloud (Public)

Resolved

This incident has been resolved.
Posted Feb 04, 2019 - 20:45 UTC

Update

We are currently monitoring for any odd or unusual behaviors.
Posted Feb 04, 2019 - 19:20 UTC

Monitoring

We've mitigated the issue for the remaining Kibana instances. We are currently monitoring for any odd or unusual behaviors.
Posted Feb 04, 2019 - 18:44 UTC

Update

We are still applying a fix for the affected Kibanas. Approximately 10 Kibana instances remain unhealthy.
Posted Feb 04, 2019 - 18:34 UTC

Update

We are still applying a fix for the affected Kibanas. Approximately 40 Kibana instances remain unhealthy.
Posted Feb 04, 2019 - 18:22 UTC

Update

We've identified around 100 Kibana clusters which are currently unhealthy. We are applying a fix.
Posted Feb 04, 2019 - 18:12 UTC

Update

Currently, we are fixing access to Kibana instances that may be experiencing cluster connectivity issues. Less than 1% of Kibana instances are currently affected by the accessibility issue.

We wanted a point of clarification: cluster connectivity was restored at 09:35 UTC and only Kibana is affected at this time.
Posted Feb 04, 2019 - 17:09 UTC

Update

We are continuing to work on a fix for this issue.
Posted Feb 04, 2019 - 16:47 UTC

Update

We are continuing to work on a fix for this issue.
Posted Feb 04, 2019 - 15:41 UTC

Identified

We have identified a remediation for the Kibana issue and we are currently applying it across the region.
Posted Feb 04, 2019 - 14:41 UTC

Monitoring

We are continuing to monitor for any further issues.
Posted Feb 04, 2019 - 12:10 UTC

Identified

Connectivity to Elasticsearch clusters has been restored, and we are focusing on some problems with the connectivity to kibana instances.
Posted Feb 04, 2019 - 11:03 UTC

Update

We are continuing to monitor the situation, and have observed increased stability in the region.
Posted Feb 04, 2019 - 10:34 UTC

Monitoring

We are experiencing much-improved stability in our us-east-1 region. The situation is being monitored closely.
Posted Feb 04, 2019 - 09:40 UTC

Update

We've resolved the issues accessing the cluster console at cloud.elastic.co. We're continuing to bring remaining services within us-east-1 back online.
Posted Feb 04, 2019 - 07:45 UTC

Update

The customer console at cloud.elastic.co is currently unavailable. We're continuing to work on bringing services within us-east-1 back online.
Posted Feb 04, 2019 - 06:58 UTC

Update

We've stabilised the co-ordination layer and are attempting to bring services within us-east-1 back online.
Posted Feb 04, 2019 - 06:22 UTC

Update

We're attempting to remediate the situation within our coordination layer within us-east-1. Issues within us-east-1 are ongoing.
Posted Feb 04, 2019 - 05:34 UTC

Investigating

We've had a reoccurrence of the original issue and we are actively looking into it.
Posted Feb 04, 2019 - 04:50 UTC

Identified

The issue has been identified and we're implementing a fix. Cluster connectivity should be restoring.
Posted Feb 04, 2019 - 03:56 UTC

Update

We are continuing to investigate this issue.
Posted Feb 04, 2019 - 03:28 UTC

Investigating

We are currently investigating connectivity issues on our proxy layer, which is affecting us-east-1.
Posted Feb 04, 2019 - 03:16 UTC
This incident affected: AWS N. Virginia (us-east-1) (Elasticsearch connectivity: AWS us-east-1).