Allowing Long Idle Timeouts when using AWS ElasticBeanstalk and Docker

A client I work with had a requirement for a 60 second plus HTTP connection timeout when running Docker on ElasticBeanstalk. Specifically, one of the Engineers was noticing that any HTTP requests taking 60 seconds or more to complete were not being returned by the ElasticBeanstalk application.

Identifying the Cause of the 60 Second Dropped Connections:

The 60 second timeout is actually set in two locations, described below:

  1. The Amazon Elastic Load Balancer, which uses a default “Idle Timeout” value of 60 seconds. The “Idle Timeout” of the given Elastic Load Balancer can be changed easily. (http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/config-idle-timeout.html).

    ElasticBeanstalk - ELB - 600 Second Timeout
    ElasticBeanstalk ELB configured with 600 second Idle Timeout.
  2. The nginx Application that acts as a proxy server in front of the Docker container also has a default timeout. The nginx default timeout is not exposed for configuration – you’ll need to modify the nginx configuration through the use of an .ebextensions file or another method. This will also be described within this blog post.
ElasticBeanstalk - HTTP Request Flow
ElasticBeanstalk – HTTP Request Flow

Setting the nginx Timeouts:

The method I used for setting the nginx timeouts can be described, at a high level as:

  1. creating an “ebextension” file that modifies the default nginx configuration used by ElasticBeanstalk. ebextension files are used by Amazon to modify the configuration of ElasticBeanstalk instances.
  2. creating a ZIP format “package” containing a Dockerrun.aws.json file as well as the .ebextension file used to modify the ElasticBeanstalk configuration.

The details are below:

  • Create an “ebextension” file within the root of your project – the file should be at the path .ebextensions/nginx-timeout.config.
  • The content of the file is described below:
files:
  "/etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy-timeout.conf":     mode: "000644"
    owner: root
    group: root
    content: |
      proxy_connect_timeout       600;
      proxy_send_timeout          600;
      proxy_read_timeout          600;
      send_timeout                600;
commands:
  "00nginx-create-proxy-timeout":
    command: "if [[ ! -h /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy-timeout.conf ]] ; then ln -s /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy-timeout.conf /etc/nginx/sites-enabled/elasticbeanstalk-nginx-docker-proxy-timeout.conf ; fi"
  • Create an application package by running the following from the root of your application’s directory:
    • zip -r ../application-name-$version.zip .ebextensions Dockerrun.aws.json
    • the command above will package the “Dockerrun.aws.json” file as well as the contents of the .ebextensions directory
  • Upload the resulting application-name-$version.zip to AWS ElasticBeanstalk to deploy your application with the nginx timeouts.
  • Note that I’ll be continuing to do testing around the ideal values for the proxy_connect_timeout, proxy_send_timeout, proxy_read_timeout and send_timeout values.

Understanding the “EB Config” sub-command

Understanding the “EB Config” sub-command

The goal of the blog post below is aid a user in better understanding the “EB Config” sub-command – to be perfectly blunt, the eb config documentation does not actually provide enough information to allow a user to utilize the command effectively. A majority of my clients that do use the eb command line interface don’t actually use eb config to save configuration or to commit configuration to version control. Below I’ll describe each config option (delete, list, get, put, save) and what that command actually does.

eb config save $environment_name

Saves a running environment’s configuration in two locations:

  1. An S3 Bucket.
  2. As a YAML file on your own computer.

For instance, if you have an application named “www” and an environment named “www-qa01”, you can save the configuration for this environment by running “eb config save www-qa01.” The following will occur:

  1. User will be prompted to enter a name for the saved configuration – in this example, I’ll use www-qa01
  2. The configuration will be saved in S3 as s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 (note that the bucket name depends on your region and account ID and the key name depends on the application name and saved configuration name you provided previously)
  3. The configuration will be copied from S3 to saved to local disk as .elasticbeanstalk/saved_configs/www-qa01.cfg.yml (note that the filename depends on the saved configuration name you provided previously)

A before and after screenshot of running the eb config savecommand: Before command – notice that the AWS Console displays no “Saved Configurations”: eb config save - www-qa01 - Before After command – notice that the eb config save command has written a file to disk and that the AWS Console displays a saved configuration named “www-qa01.” eb config save - www-qa01 - After Lastly – a diagram of how the eb config save command works: eb config save - Diagram

eb config delete $environment_name

As an example, if I ran eb config delete www-qa01, then the following would occur:

  1. The www-qa01 saved configuration object will be removed from S3. As an example, if I ran eb config delete www-qa01, then the object s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 would be deleted.
  2. As a result of deleting the www-qa01 saved configuration object from S3, the www-qa01 saved configuration will be removed from AWS Console
  3. The YAML configuration file will be removed from .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config get $environment_name

If I ran eb config get www-qa01, then the following would occur:

  1. If exists, a saved configuration file will be copied from S3 (location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01) to local disk at .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config list

If I ran the eb config list, I will execute a “list” against the objects located at s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/*.

eb config put $environment_name

Uploads the named saved configuration to an Amazon S3 bucket. As an example, if I ran eb config put www-qa01, then the following would happen:

  1. A file named .elasticbeanstalk/saved_configs/www-qa01.cfg.yml would be uploaded to S3 at the following location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/.
  2. As a result of a new object being uploaded to S3, the saved configuration will appear in the AWS Console.

You can also upload a specific file (if not in .elasticbeanstalk/saved_configs/www-qa01.cfg.yml) by running the command as follows: eb config put www-qa01 --cfg ~/path/to/saved_config/www-qa02.config.yml where ~/path/to/saved_config/www-qa02.config.yml is a path to a valid Elastic Beanstalk Configuration file.