Resolved
This incident has been resolved.
Posted Aug 10, 2021 - 06:22 UTC
Update
We’ve tested out our remediation process and have begun running it against the full fleet, which will take some time to complete successfully. We will update this incident once we have completed the remediation across all regions.
In the meanwhile, if you have a stateless application (Kibana, APM, Fleets, and Enterprise Search), force restarting any unhealthy application will work around this issue.
Posted Aug 09, 2021 - 15:53 UTC
Update
We're testing our remediation process and are running against canary instances; once validated, we'll run against the full fleet. We'll update again when remediation is complete.
Posted Aug 09, 2021 - 13:58 UTC
Identified
We are currently experiencing network connectivity issues in a small percentage of customers' stack components. This does not impact Elasticsearch (including ML) connectivity but may impact Kibana, APM, Fleets, and Enterprise Search. We've determined the issue and are working on a remediation process. We'll update again in an hour.
Posted Aug 09, 2021 - 13:22 UTC
This incident affected: GCP Sydney (australia-southeast1) (Kibana connectivity: GCP australia-southeast1, APM connectivity: GCP australia-southeast1, Enterprise Search connectivity: GCP australia-southeast1, App Search connectivity: GCP australia-southeast1), GCP São Paulo (southamerica-east1) (APM connectivity: GCP southamerica-east1, Kibana connectivity: GCP southamerica-east1, Enterprise Search connectivity: GCP southamerica-east1, App Search connectivity: GCP southamerica-east1), GCP South Carolina (us-east1) (APM connectivity: GCP us-east1, Kibana connectivity: GCP us-east1, Enterprise Search connectivity: GCP us-east1, App Search connectivity: GCP us-east1), Azure Ireland (azure-northeurope) (APM connectivity: Azure azure-northeurope, App Search connectivity: Azure azure-northeurope, Enterprise Search connectivity: Azure azure-northeurope, Kibana connectivity: Azure azure-northeurope), GCP Tokyo (asia-northeast1) (Kibana connectivity: GCP asia-northeast1, APM connectivity: GCP asia-northeast1, Enterprise Search connectivity: GCP asia-northeast1, App Search connectivity: GCP asia-northeast1), GCP N. Virginia (us-east4) (APM connectivity: GCP us-east4, Kibana connectivity: GCP us-east4, Enterprise Search connectivity: GCP us-east4, App Search connectivity: GCP us-east4), Azure Washington (azure-westus2) (Kibana connectivity: Azure azure-westus2, APM connectivity: Azure azure-westus2, App Search connectivity: Azure azure-westus2, Enterprise Search connectivity: Azure azure-westus2), Azure New South Wales (azure-australiaeast) (APM connectivity: Azure azure-australiaeast, Kibana connectivity: Azure azure-australiaeast, App Search connectivity: Azure azure-australiaeast, Enterprise Search connectivity: Azure azure-australiaeast), GCP Seoul (asia-northeast3) (Kibana connectivity: GCP asia-northeast3, App Search connectivity: GCP asia-northeast3, APM connectivity: GCP asia-northeast3, Enterprise Search connectivity: GCP asia-northeast3), AWS Frankfurt (eu-central-1) (APM connectivity: AWS eu-central-1, Kibana connectivity: AWS eu-central-1, App Search connectivity: AWS eu-central-1, Enterprise Search connectivity: AWS eu-central-1), GCP London (europe-west2) (APM connectivity: GCP europe-west2, Kibana connectivity: GCP europe-west2, App Search connectivity: GCP europe-west2, Enterprise Search connectivity: GCP europe-west2), GCP Montreal (northamerica-northeast1) (APM connectivity: GCP northamerica-northeast1, Kibana connectivity: GCP northamerica-northeast1, App Search connectivity: GCP northamerica-northeast1, Enterprise Search connectivity: GCP northamerica-northeast1), Azure Tokyo (azure-japaneast) (Kibana connectivity: Azure azure-japaneast, APM connectivity: Azure azure-japaneast, Enterprise Search connectivity: Azure azure-japaneast, App Search connectivity: Azure azure-japaneast), GCP Singapore (asia-southeast1) (Kibana connectivity: GCP asia-southeast1, APM connectivity: GCP asia-southeast1, App Search connectivity: GCP asia-southeast1, Enterprise Search connectivity: GCP asia-southeast1), GCP Taiwan (asia-east1) (APM connectivity: GCP asia-east1, Kibana connectivity: GCP asia-east1, App Search connectivity: GCP asia-east1, Enterprise Search connectivity: GCP asia-east1), GCP Finland (europe-north1) (Kibana connectivity: GCP europe-north1, APM connectivity: GCP europe-north1, App Search connectivity: GCP europe-north1, Enterprise Search connectivity: GCP europe-north1), GCP Netherlands (europe-west4) (APM connectivity: GCP europe-west4, Kibana connectivity: GCP europe-west4, Enterprise Search connectivity: GCP europe-west4, App Search connectivity: GCP europe-west4), Azure Iowa (azure-centralus) (Kibana connectivity: Azure azure-centralus, APM connectivity: Azure azure-centralus, App Search connectivity: Azure azure-centralus, Enterprise Search connectivity: Azure azure-centralus), Azure Virginia (azure-eastus) (App Search connectivity: Azure azure-eastus, Kibana connectivity: Azure azure-eastus, Enterprise Search connectivity: Azure azure-eastus, APM connectivity: Azure azure-eastus), Azure Texas (azure-southcentralus) (App Search connectivity: Azure azure-southcentralus, Kibana connectivity: Azure azure-southcentralus, Enterprise Search connectivity: Azure azure-southcentralus, APM connectivity: Azure azure-southcentralus), GCP Mumbai (asia-south1) (Kibana connectivity: GCP asia-south1, APM connectivity: GCP asia-south1, Enterprise Search connectivity: GCP asia-south1, App Search connectivity: GCP asia-south1), Azure London (azure-uksouth) (APM connectivity: Azure azure-uksouth, Kibana connectivity: Azure azure-uksouth, Enterprise Search connectivity: Azure azure-uksouth, App Search connectivity: Azure azure-uksouth), AWS Seoul (ap-northeast-2) (APM connectivity: AWS ap-northeast-2, Kibana connectivity: AWS ap-northeast-2, App Search connectivity: AWS ap-northeast-2, Enterprise Search connectivity: AWS ap-northeast-2), AWS Paris (eu-west-3) (Kibana connectivity: AWS eu-west-3, APM connectivity: AWS eu-west-3, Enterprise Search connectivity: AWS eu-west-3, App Search connectivity: AWS eu-west-3), AWS Mumbai (ap-south-1) (APM connectivity: AWS ap-south-1, Kibana connectivity: AWS ap-south-1, Enterprise Search connectivity: AWS ap-south-1, App Search connectivity: AWS ap-south-1), Azure France (azure-francecentral) (App Search connectivity: Azure azure-francecentral, Enterprise Search connectivity: Azure azure-francecentral, Kibana connectivity: Azure azure-francecentral, APM connectivity: Azure azure-francecentral), Azure Virginia (azure-eastus2) (APM connectivity: Azure azure-eastus2, Kibana connectivity: Azure azure-eastus2, App Search connectivity: Azure azure-eastus2, Enterprise Search connectivity: Azure azure-eastus2), Azure Netherlands (azure-westeurope) (APM connectivity: Azure azure-westeurope, Kibana connectivity: Azure azure-westeurope, App Search connectivity: Azure azure-westeurope, Enterprise Search connectivity: Azure azure-westeurope), AWS Ohio (us-east-2) (APM connectivity: AWS us-east-2, Kibana connectivity: AWS us-east-2, App Search connectivity: AWS us-east-2, Enterprise Search connectivity: AWS us-east-2), GCP Los Angeles (us-west2) (Kibana connectivity: GCP us-west2, APM connectivity: GCP us-west2, App Search connectivity: GCP us-west2, Enterprise Search connectivity: GCP us-west2), Azure Singapore (azure-southeastasia) (APM connectivity: Azure azure-southeastasia, Kibana connectivity: Azure azure-southeastasia, App Search connectivity: Azure azure-southeastasia, Enterprise Search connectivity: Azure azure-southeastasia), AWS Hong Kong (ap-east-1) (APM connectivity: AWS ap-east-1, Kibana connectivity: AWS ap-east-1, Enterprise Search connectivity: AWS ap-east-1, App Search connectivity: AWS ap-east-1), GCP Belgium (europe-west1) (Kibana connectivity: GCP europe-west1, APM connectivity: GCP europe-west1, App Search connectivity: GCP europe-west1, Enterprise Search connectivity: GCP europe-west1), GCP Frankfurt (europe-west3) (Kibana connectivity: GCP europe-west3, APM connectivity: GCP europe-west3, App Search connectivity: GCP europe-west3, Enterprise Search connectivity: GCP europe-west3), AWS Tokyo (ap-northeast-1) (Kibana connectivity: AWS ap-northeast-1, APM connectivity: AWS ap-northeast-1, Enterprise Search connectivity: AWS ap-northeast-1, App Search connectivity: AWS ap-northeast-1), AWS Central (ca-central-1) (Kibana connectivity: AWS ca-central-1, APM connectivity: AWS ca-central-1, Enterprise Search connectivity: AWS ca-central-1, App Search connectivity: AWS ca-central-1), GCP Iowa (us-central1) (Kibana connectivity: GCP us-central1, APM connectivity: GCP us-central1, App Search connectivity: GCP us-central1, Enterprise Search connectivity: GCP us-central1), GCP Oregon (us-west1) (Kibana connectivity: GCP us-west1, APM connectivity: GCP us-west1, App Search connectivity: GCP us-west1, Enterprise Search connectivity: GCP us-west1), AWS São Paulo (sa-east-1) (Kibana connectivity: AWS sa-east-1, APM connectivity: AWS sa-east-1, App Search connectivity: AWS sa-east-1, Enterprise Search connectivity: AWS sa-east-1), AWS N. California (us-west-1) (Kibana connectivity: AWS us-west-1, APM connectivity: AWS us-west-1, App Search connectivity: AWS us-west-1, Enterprise Search connectivity: AWS us-west-1), AWS Oregon (us-west-2) (Kibana connectivity: AWS us-west-2, APM connectivity: AWS us-west-2, App Search connectivity: AWS us-west-2, Enterprise Search connectivity: AWS us-west-2), AWS N. Virginia (us-east-1) (Kibana connectivity: AWS us-east-1, APM connectivity: AWS us-east-1, App Search connectivity: AWS us-east-1, Enterprise Search connectivity: AWS us-east-1), AWS Ireland (eu-west-1) (Kibana connectivity: AWS eu-west-1, APM connectivity: AWS eu-west-1, Enterprise Search connectivity: AWS eu-west-1, App Search connectivity: AWS eu-west-1), AWS Singapore (ap-southeast-1) (Kibana connectivity: AWS ap-southeast-1, APM connectivity: AWS ap-southeast-1, Enterprise Search connectivity: AWS ap-southeast-1, App Search connectivity: AWS ap-southeast-1), AWS London (eu-west-2) (Kibana connectivity: AWS eu-west-2, APM connectivity: AWS eu-west-2, App Search connectivity: AWS eu-west-2, Enterprise Search connectivity: AWS eu-west-2), and AWS Sydney (ap-southeast-2) (APM connectivity: AWS ap-southeast-2, Enterprise Search connectivity: AWS ap-southeast-2, App Search connectivity: AWS ap-southeast-2, Kibana connectivity: AWS ap-southeast-2).