Metrics Ingestion Delays in All Regions
Incident Report for ESS (Public)
Resolved
This incident has been resolved.
Posted Aug 08, 2019 - 22:38 UTC
Monitoring
The rollback for the user console is complete and page load times have returned to normal. At this point there should be no more customer impact. We will continue to monitor the affected systems to ensure they remain stable.
Posted Aug 08, 2019 - 21:30 UTC
Identified
We are rolling back additional changes contributing to the increased latency on the user console. We will post an update in approximately 2 hours.
Posted Aug 08, 2019 - 19:21 UTC
Update
We found an issue causing high load on the metrics store. The change is being rolled back that caused the extra load and we are monitoring for improvement. We're continuing to investigate and we'll post another update in the next 30 minutes.

This issue also have added a latency increase on the user console.
Posted Aug 08, 2019 - 18:41 UTC
Investigating
Starting at 16:26 UTC we started seeing metrics ingestion delays for all regions which affects visibility into "Native memory pressure" and "Disk usage" for customer clusters. Engineers are actively investigating the cause of the delay and we will provide an an update in the next 30 minutes.
Posted Aug 08, 2019 - 18:05 UTC