Three days earlier, new proxies were rolled out in the us-west-1 zone. The new proxies were running as Docker containers and after 24 hours in service, everything was looking good.

Around 48 hours after proxies were rolled out, the memory usage of the Docker daemon itself started growing, and after 8 hours it was using close to 80% of all available memory on the instance. The OOM-reaper killed the proxy container and the Docker daemon. This should normally have caused the Docker daemon to restart and the proxy container to be restarted as well, but due to another bug in Docker (https://github.com/docker/docker/issues/10685) it was unable to restart.

We were alerted to the situation and quickly added the original proxies back into the load balancer and started the investigation.

During the post-mortem investigation, we found that Docker has a memory leak when the containers log to their stdout (https://github.com/docker/docker/issues/9139). The workaround is currently to avoid logging to stdout, which is what we've done now.

In order to avoid this happening again, we've added an alert that will notify an on-duty engineer whenever a proxy instance has less free memory (excluding page caches) than expected.