An Introduction to the AWS EC2 Container Service

An Introduction to the AWS EC2 Container Service:

Below is a quick introduction to the AWS EC2 Container Service. The most valuable part of this post may be the link to the CloudFormation stack that will build an EC2 Container Service Cluster and all associated resources. Creating this CloudFormation stack was the moment when I really understood ECS at a nuts and bolts level.

How I came to use ECS at all:

On November 13, 2014, Amazon released the “Amazon EC2 Container Service” – the news didn’t excite me much. Containerization, specifically Docker, was a hot technology but my own frustration with the industry’s lack of understanding of Docker’s benefits and costs coupled with the quality and thought behind Docker implementations I’d seen meant I had little interest in EC2 Container Service. My own clients were happy with Docker and didn’t require container packing or docker links – so there was no push to move to ECS or another container management software. I put ECS on the shelf to be revisited. The day when I would revisit ECS came about mid-June, when one of our developers needed to expose a port through an ELB “fronting” and Elastic Beanstalk Environment and also needed to publish multiple ports on each EC2 instance making up the Elastic Beanstalk Environment. Stated simply, the EC2 instances needed to expose port 8080 to a Load Balancer and the EC2 instances also needed to communicate with each other across an array of ports (specifically, we were running a tomcat7 hosted Java application that utilized Apache Ignite with a Replicated Cache).

Setting Up ECS:

My initial work with ECS was a challenge because of the ECS lexicon – “task definitions, tasks, services, container definitions” and the lack of maturity – AWS actually added CloudFormation support during my ECS prototyping. In any case, I’ll describe each of the resources that make up ECS, how they are used and how they relate to each other:

Clusters and Container Instances:

Clusters and Container Instances will make up the basis of your ECS Infrastructure. A quick diagram demonstrates the relationship between Clusters and Container Instances.

EC2 Container Service - Cluster and Container Instances

  • Clusters are a group of Container Instances.
Container Instances:
  • Container instances run one or more containers
  • Container instances must run the “ECS Container Agent” which will register with an ECS Cluster
    • the “Container Agent” is really well thought out – for instance, you can set or pass values to the ECS Container Agent during install so that EC2 instances within an Auto Scaling Group are automatically registered with an ECS cluster

Task Definitions, Tasks and Services:

You’ll run your application by creating a Task Definition and then running the Task Definition across one or more Container Instances. I’ll explain the Task Definitions, Tasks and Services resources below.

Task Definition:

A task definition contains three parts:

  • Container Definitions – this is a list of one or more containers that make up a task. An example would be an nginx container running front-end code and an nginx container running back-end code. An example of two different “Task Definitions” is given below – one of these task definitions utilizing only one container, while a second, different task definition requires two different containers.
    EC2 Container Service - Task Definition
  • Family – an arbitrary name for the task definition – if you iteratively modify your Task Definitions you can utilize the “Family” to keep these different versions.
  • Volumes (optional) – a list of volumes that will be made available to the containers running within a given Container Instance.
    • I have not needed to utilize Volumes as I’ve been able to clearly communicate or utilize other services to avoid the requirement for Block Storage.

A task is created when a “Task Definition” is run on a container instance. You can use one “Task Definition” to instantiate containers on multiple hosts – for instance, when you click on “Run new task” and select a task definition you’ll be asked how many “tasks” you want to run. If you have a task definition that contains two “Container Definitions” and you want to run 2 tasks, Amazon will place two containers on one of the Container Instances and two containers on the other Container Instance.


A service is a “superset” of a Task Definition. An example of a “Service” is shown below – note the addition of an Elastic Load Balancer.

EC2 Container Service - Services

When you create a service you define the following:

  • the “Task Definition” used to determine which Tasks/Containers will be run
  • a desired number of tasks
  • a Load Balancer

In return for the bit of extra work, the service will:

  • “run and maintain a specified number of tasks.” If you ask for 4 tasks when you instantiate a service, ECS you’ll always have 4 tasks running.
  • utilize a Load Balancer for routing traffic to Container Instances instances that are running a given Container.
Services versus Tasks:


  • if a task fails, ECS does not return the task to service
  • when defining a “Task Definition” you are not allowed to define a Load Balancer


  • if a task fails, ECS will return the task to service
    • an example would include the loss of a Container Instance that drops the number of running tasks below the number of – when a new Container Instance is brought into the cluster a new task will be started
  • when defining a “Service” you are optionally allowed to define a Load Balancer

Auto Scaling with ECS:

  • In regards to the “Container Instances” as part of an Auto Scaling Group, here is a real world example of this benefit:
    • terminate a Cluster Instance that is a member of an ECS Cluster and an Auto Scaling Group
    • the Auto Scaling Group will bring a new EC2 Instance in service
    • the new EC2 Instance’s user data contains instructions for configuring and installing the ECS Container Agent
    • the ECS Container Agent registers the new EC2 Instance with the cluster

Running ECS:

For those who prefer to “learn by doing” – if you navigate to my “Snippets” repository on GitHub you’ll notice the following file – this will create a working ECS Cluster running nginx and all other required resources. The link is here:

Future Post:

I’ll be covering some of the following in a future post:

  • How to best utilize Security Groups when working with ECS?
    • Note: it would seem obvious that Security Groups aren’t sufficient for securing Docker containers within ECS – maybe AWS is working on something?
  • Best practices for building clusters – do you build one large cluster for all environments, do you build one cluster per environment?
  • Does AWS / ECS distribute services/tasks across Availability Zones correctly?
  • Can I utilize ECS to have an unequal number of containers – for instance, can I have a 2 to 1 relationship of front-end to back-end instances?
  • Tuning resource usage of Containers.
  • How are “tasks” within a service considered healthy?
  • How to delete task definitions from AWS?

Reducing AWS Cost using Scheduled Scaling

Reducing AWS Cost using Scheduled Scaling

One of the ways to reduce AWS cost is to utilize Auto Scaling Groups and the “Scheduled Scaling” feature to scaled down EC2 resources during non-business hours. In the example below, I’ll walk you through the use of scheduled scaling to “Scale Down” an api-qa01 Auto Scaling Group to 0 instances after the close of business (assumed to be 7:00 pm pacific) and “Scale Up” an api-qa01 Auto Scaling Group to two instances at start of business (assumed to be 8:00 am pacific).

Setup Scheduled Scaling:

To setup scheduled scaling, you’ll need:

  1. Infrastructure that uses Auto Scaling Groups (you could do something similar with EC2 start/stop instances, but I don’t plan to cover this case in the blog). It would be ideal if you’ve configured your Auto Scaling Group to publish Auto Scaling Group Metrics – this will let you track the capacity of an Auto Scaling Group. To do so:
    1. Check “Enable CloudWatch detailed monitoring” when creating the Launch Configuration in the AWS Console, or use –instance-monitoring when creating a Launch Configuration using the AWS Command Line Interface.
    2. Use “aws autoscaling enable-metrics-collection” or check “Enable CloudWatch detailed monitoring” when creating the Auto Scaling Group from the AWS Console.
  2. A computer with the “AWS Command Line Interface” tools installed (scheduled scaling is not yet available in the AWS Console)

With the above prerequisites set, configuring scheduled scaling is as simple as doing the following:

  1. Identifying the name of the group for which you wish to enable scheduled scaling.
  2. Creating a “Scale Up” scheduled event, example below:
    1. aws autoscaling put-scheduled-update-group-action --scheduled-action-name scaleup-api-qa01 --auto-scaling-group-name api-qa01 --recurrence "0 15 * * *" --desired-capacity 2
  3. Creating a “Scale Down” scheduled event:
    1. aws autoscaling put-scheduled-update-group-action --scheduled-action-name scaledown-api-qa01 --auto-scaling-group-name api-qa01 --recurrence "0 2 * * *" --desired-capacity 0
  4. Once you done creating “Scale Up” and “Scale Down” events, you’ll want to ensure that you’ve setup the scheduled scaling actions correctly – the command to do this is below:
    1. aws autoscaling describe-scheduled-actions --auto-scaling-group-name api-qa01 --query 'ScheduledUpdateGroupActions[*].{Name:ScheduledActionName,DesiredCapacity:DesiredCapacity,Recurrence:Recurrence}' --output table
    2. The result should look something akin to the followingScheduled Scaling - Described Scheduled Actions
Confirming Scaling:

If you’ve enabled Auto Scaling Group Metrics you should be able to identify the changes in Desired Capacity and in InService Capacity. An example of the Auto Scaling Group api-qa01 is shown below:

Scheduled Scaling Example

Noteice that the “GroupDesiredCapacity” and “GroupTotalInstances” increases to 2 Instances daily at 15:00 UTC and then returns to 0 Instances daily at 2:00 UTC.

If You Don’t Use Auto Scaling:

Those not using Scheduled Scaling might be able to make be able to utilize a cron job that Starts and Stops EC2 instances based on a Tag. For instance, you could create Tags with the Key “StartAfter” and “StopAfter” and then utilize the Values of these tags to inform a cron job when to Start/Stop EC2 instances.

In Summary:
  1. You should have everything you need in order to schedule Start/Stop of instances based on time of day.
  2. Use of scheduled Scale Up / Scale Down is a great way to ensure that your staff is familiar with Auto Scaling (QA or Developers might need to Scale Up infrastructure outside of business horus) and that your AWS Infrastructure is capable of withstanding outages or periods of time when no EC2 instances are running.
  3. If you’ve got questions, please feel free to comment below or send me an email.

Understanding the “EB Config” sub-command

Understanding the “EB Config” sub-command

The goal of the blog post below is aid a user in better understanding the “EB Config” sub-command – to be perfectly blunt, the eb config documentation does not actually provide enough information to allow a user to utilize the command effectively. A majority of my clients that do use the eb command line interface don’t actually use eb config to save configuration or to commit configuration to version control. Below I’ll describe each config option (delete, list, get, put, save) and what that command actually does.

eb config save $environment_name

Saves a running environment’s configuration in two locations:

  1. An S3 Bucket.
  2. As a YAML file on your own computer.

For instance, if you have an application named “www” and an environment named “www-qa01”, you can save the configuration for this environment by running “eb config save www-qa01.” The following will occur:

  1. User will be prompted to enter a name for the saved configuration – in this example, I’ll use www-qa01
  2. The configuration will be saved in S3 as s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 (note that the bucket name depends on your region and account ID and the key name depends on the application name and saved configuration name you provided previously)
  3. The configuration will be copied from S3 to saved to local disk as .elasticbeanstalk/saved_configs/www-qa01.cfg.yml (note that the filename depends on the saved configuration name you provided previously)

A before and after screenshot of running the eb config savecommand: Before command – notice that the AWS Console displays no “Saved Configurations”: eb config save - www-qa01 - Before After command – notice that the eb config save command has written a file to disk and that the AWS Console displays a saved configuration named “www-qa01.” eb config save - www-qa01 - After Lastly – a diagram of how the eb config save command works: eb config save - Diagram

eb config delete $environment_name

As an example, if I ran eb config delete www-qa01, then the following would occur:

  1. The www-qa01 saved configuration object will be removed from S3. As an example, if I ran eb config delete www-qa01, then the object s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 would be deleted.
  2. As a result of deleting the www-qa01 saved configuration object from S3, the www-qa01 saved configuration will be removed from AWS Console
  3. The YAML configuration file will be removed from .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config get $environment_name

If I ran eb config get www-qa01, then the following would occur:

  1. If exists, a saved configuration file will be copied from S3 (location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01) to local disk at .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config list

If I ran the eb config list, I will execute a “list” against the objects located at s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/*.

eb config put $environment_name

Uploads the named saved configuration to an Amazon S3 bucket. As an example, if I ran eb config put www-qa01, then the following would happen:

  1. A file named .elasticbeanstalk/saved_configs/www-qa01.cfg.yml would be uploaded to S3 at the following location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/.
  2. As a result of a new object being uploaded to S3, the saved configuration will appear in the AWS Console.

You can also upload a specific file (if not in .elasticbeanstalk/saved_configs/www-qa01.cfg.yml) by running the command as follows: eb config put www-qa01 --cfg ~/path/to/saved_config/www-qa02.config.yml where ~/path/to/saved_config/www-qa02.config.yml is a path to a valid Elastic Beanstalk Configuration file.

AWS Security Group Visualization Tools

AWS Security Group Visualization Tools

I recently had a customer ask for a method of visualizing their collection of AWS Security Groups. I did some research and identified a few different tools – there are two particular tools I wanted to introduce.

Dome9 Clarity Visualization Tool:

I’ve attached a screenshot of the result of the use of the Dome9 Clarity visualization tool. For basic ingress mapping the tool is easy to understand. A screenshot is below:

Dome9 Clarity - Security Group Ingress

The tool can get a bit muddled with multiple security groups and I was also a bit frustrated by the fact that the tool seemed to operate on a schedule rather than reflecting real-time Security Group changes. The “yellow”/”green” lights highlighting potential security flaws are a frustration – allowing port 80 in from the entire world to a load-balancer facing the public Internet isn’t a warning – this is desired behavior. A yellow light would be port 22 into an RDS instance, which simply won’t work, or a Security Group shared by an EC2 instance and an RDS instance – which shouldn’t have the same set of inbound rules, ever.

Anay Nayak’s “aws-security-viz” Tool:

For those folks who wish to provide a visualization of Security Groups without subscribing to a service, who prefer open-source tooling or wish to customize output – Anay Nayak’s aws-security-viz tool (on GitHub) will fit the bill. I’m impressed by the tool for two reasons: it supports both inbound and outbound visualization and the fact that the tool provides a dot format or svg format output – meaning you can get your hands on the files themselves. I did find the tool a bit of a challenge to run under OS X – I ran in Vagrant. A screenshot of the aws-security-viz tool’s output is below:

aws-security-viz - dot Format Output


Hopefully the screenshot and description of each tool has provided enough of a description that you’ll be able to identify which tool that will best meet your needs.

Reducing Costs on AWS

Reducing Costs on AWS

One of the requests I get from growing or larger AWS customers is “how do I reduce my Amazon Web Services bill”? There are a lot of analysis tools available – this post is not going to focus on those. Instead I’ll be focusing a lesser known option for reducing cost – working with an Amazon Web Services Reseller or buying direct from Amazon.

How it works?

Typically AWS sells directly to distributors (avnet, datapipe, Ingram) which then sell to resellers – this resale is typically done at a percentage reduction off of a monthly Amazon bill. The distributors then sell to resellers, who then sell directly to AWS customers. The resellers typically have a small percentage markup as they have to handle billing and issues that might arise. The resellers also know that the AWS service itself is a commodity – so they strive to provide some additional benefit:

  • A managed service contract, where they offer monitoring or incident response.
  • Cost analysis or other tooling.
  • Suggestions to help you further lower your costs – they know you’ll leave if they aren’t acting in your best interests.

Which customers benefit?

Depends on the customer’s needs – if the customer benefits from a resellers value add(s), then working through a reseller is a good option. However, if cost savings is the entirety of a customer’s interest, larger or quickly growing customers tend to benefit the most. When reselling AWS, the reseller has to take on the additional overhead of billing and waiting for payment – this is a fixed cost (the cost of collection) and some risk (the risk of non-payment). As the reseller typically adds some percentage when reselling to AWS customers, larger accounts generate more revenue for a reseller (compare a 2% markup on a $100,000/mo account and a $10,000/mo account) – thus, resellers may be more willing to offer large discounts to larger accounts, whereas offering a discount to a smaller AWS customer may not be in anyone’s interest.

What if I am a small customer?

Smaller customers might be able to take advantage of an AWS Activate package, which are geared toward smaller startups with a particular degree of pedigree.

What are the discount amounts?

Short answer: I don’t have the freedom to be specific and I can not find discount percentages publicly available online.

Long answer: With the lack of transparency (apparently this is called “information asymmetry”) the entire process feels like purchasing a car in the United States. I’d be happy to make myself available If you have questions about the ecosystem or how to negotiate the best deal – ask in person or send me an email at

OS X Native MFA Code Generator

Although I am happy with using Google Authenticator for multi-factor authentication I was interested in having an OS X native application – after some searching I found “OTP Manager” which I have started using as well. OTP Manager is available at and also available for free through the App Store at

OTP Manager