A client I work with had a requirement for a 60 second plus HTTP connection timeout when running Docker on ElasticBeanstalk. Specifically, one of the Engineers was noticing that any HTTP requests taking 60 seconds or more to complete were not being returned by the ElasticBeanstalk application.
Identifying the Cause of the 60 Second Dropped Connections:
The 60 second timeout is actually set in two locations, described below:
- The Amazon Elastic Load Balancer, which uses a default “Idle Timeout” value of 60 seconds. The “Idle Timeout” of the given Elastic Load Balancer can be changed easily. (http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/config-idle-timeout.html).
- The nginx Application that acts as a proxy server in front of the Docker container also has a default timeout. The nginx default timeout is not exposed for configuration – you’ll need to modify the nginx configuration through the use of an .ebextensions file or another method. This will also be described within this blog post.
Setting the nginx Timeouts:
The method I used for setting the nginx timeouts can be described, at a high level as:
- creating an “ebextension” file that modifies the default nginx configuration used by ElasticBeanstalk. ebextension files are used by Amazon to modify the configuration of ElasticBeanstalk instances.
- creating a ZIP format “package” containing a Dockerrun.aws.json file as well as the .ebextension file used to modify the ElasticBeanstalk configuration.
The details are below:
- Create an “ebextension” file within the root of your project – the file should be at the path .ebextensions/nginx-timeout.config.
- The content of the file is described below:
files: "/etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy-timeout.conf": mode: "000644" owner: root group: root content: | proxy_connect_timeout 600; proxy_send_timeout 600; proxy_read_timeout 600; send_timeout 600; commands: "00nginx-create-proxy-timeout": command: "if [[ ! -h /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy-timeout.conf ]] ; then ln -s /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy-timeout.conf /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy-timeout.conf ; fi"
- Create an application package by running the following from the root of your application’s directory:
- zip -r ../application-name-$version.zip .ebextensions Dockerrun.aws.json
- the command above will package the “Dockerrun.aws.json” file as well as the contents of the .ebextensions directory
- Upload the resulting application-name-$version.zip to AWS ElasticBeanstalk to deploy your application with the nginx timeouts.
- Note that I’ll be continuing to do testing around the ideal values for the proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout and send_timeout values.