We are confident we've isolated the problems that have contributed to the conditions triggering OOM errors. Please reach out via https://discuss.elastic.co/c/cloud if you have any problems.
After thoroughly investigating everything from our own Elastic Cloud plugins, to Elasticsearch, Lucene, Netty, the JVM, and glibc’s memory allocator, here’s what we found:
* We identified and addressed small memory leaks in Elasticsearch and Lucene
* We optimised some JVM and glibc settings appropriate for small processes running on huge VMs
* We identified and are currently addressing a bug in the kernel’s memory accounter.
We’re optimistic that the bug in the kernel is the last big memory-related issue still to be fixed. This bug can cause the kernel to consider a container to be using more kernel memory than it really is, and therefore wrongfully terminate processes requesting memory that’s rightfully reserved to them. Smaller clusters are more prone to restarts, as users overload what Elasticsearch can do with a relatively small heap. In this case, however, even clusters with negligible load were restarted and could get stuck restarting.
We’re seeing OOM-restart levels return to normal as we move affected nodes to servers with patched kernels.
We apologise for the long time this issue has been going on. It has been a particularly grueling problem to hunt down, as it involved lots of small things, each contributing to the leaks. While we’re usually able to push out bug fixes and deal with performance issues relatively quickly, memory leaks can be notoriously hard to find–and we’ve had to weigh the risk of pushing out changes such as kernel upgrades against the fact that more than 99% of our clusters were running without any problems.
Posted Feb 07, 2017 - 21:12 UTC
Monitoring
We've been monitoring configuration changes that appear to improve overall stability for our smallest clusters which are primarily affected by the issue. In the next 48 hours, we will be migrating clusters to hosts that provide these new improvements and will update the status here as needed.
Posted Feb 04, 2017 - 23:34 UTC
Identified
We are tracking an ongoing memory issue in all regions. This can lead to an out of memory restart, especially on smaller clusters.