Understanding the “EB Config” sub-command

Understanding the “EB Config” sub-command

The goal of the blog post below is aid a user in better understanding the “EB Config” sub-command – to be perfectly blunt, the eb config documentation does not actually provide enough information to allow a user to utilize the command effectively. A majority of my clients that do use the eb command line interface don’t actually use eb config to save configuration or to commit configuration to version control. Below I’ll describe each config option (delete, list, get, put, save) and what that command actually does.

eb config save $environment_name

Saves a running environment’s configuration in two locations:

  1. An S3 Bucket.
  2. As a YAML file on your own computer.

For instance, if you have an application named “www” and an environment named “www-qa01”, you can save the configuration for this environment by running “eb config save www-qa01.” The following will occur:

  1. User will be prompted to enter a name for the saved configuration – in this example, I’ll use www-qa01
  2. The configuration will be saved in S3 as s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 (note that the bucket name depends on your region and account ID and the key name depends on the application name and saved configuration name you provided previously)
  3. The configuration will be copied from S3 to saved to local disk as .elasticbeanstalk/saved_configs/www-qa01.cfg.yml (note that the filename depends on the saved configuration name you provided previously)

A before and after screenshot of running the eb config savecommand: Before command – notice that the AWS Console displays no “Saved Configurations”: eb config save - www-qa01 - Before After command – notice that the eb config save command has written a file to disk and that the AWS Console displays a saved configuration named “www-qa01.” eb config save - www-qa01 - After Lastly – a diagram of how the eb config save command works: eb config save - Diagram

eb config delete $environment_name

As an example, if I ran eb config delete www-qa01, then the following would occur:

  1. The www-qa01 saved configuration object will be removed from S3. As an example, if I ran eb config delete www-qa01, then the object s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01 would be deleted.
  2. As a result of deleting the www-qa01 saved configuration object from S3, the www-qa01 saved configuration will be removed from AWS Console
  3. The YAML configuration file will be removed from .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config get $environment_name

If I ran eb config get www-qa01, then the following would occur:

  1. If exists, a saved configuration file will be copied from S3 (location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/www-qa01) to local disk at .elasticbeanstalk/saved_configs/www-qa01.cfg.yml
eb config list

If I ran the eb config list, I will execute a “list” against the objects located at s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/*.

eb config put $environment_name

Uploads the named saved configuration to an Amazon S3 bucket. As an example, if I ran eb config put www-qa01, then the following would happen:

  1. A file named .elasticbeanstalk/saved_configs/www-qa01.cfg.yml would be uploaded to S3 at the following location: s3://elasticbeanstalk-us-west-2-187376578462/resources/templates/www/.
  2. As a result of a new object being uploaded to S3, the saved configuration will appear in the AWS Console.

You can also upload a specific file (if not in .elasticbeanstalk/saved_configs/www-qa01.cfg.yml) by running the command as follows: eb config put www-qa01 --cfg ~/path/to/saved_config/www-qa02.config.yml where ~/path/to/saved_config/www-qa02.config.yml is a path to a valid Elastic Beanstalk Configuration file.

AWS Security Group Visualization Tools

AWS Security Group Visualization Tools

I recently had a customer ask for a method of visualizing their collection of AWS Security Groups. I did some research and identified a few different tools – there are two particular tools I wanted to introduce.

Dome9 Clarity Visualization Tool:

I’ve attached a screenshot of the result of the use of the Dome9 Clarity visualization tool. For basic ingress mapping the tool is easy to understand. A screenshot is below:

Dome9 Clarity - Security Group Ingress

The tool can get a bit muddled with multiple security groups and I was also a bit frustrated by the fact that the tool seemed to operate on a schedule rather than reflecting real-time Security Group changes. The “yellow”/”green” lights highlighting potential security flaws are a frustration – allowing port 80 in from the entire world to a load-balancer facing the public Internet isn’t a warning – this is desired behavior. A yellow light would be port 22 into an RDS instance, which simply won’t work, or a Security Group shared by an EC2 instance and an RDS instance – which shouldn’t have the same set of inbound rules, ever.

Anay Nayak’s “aws-security-viz” Tool:

For those folks who wish to provide a visualization of Security Groups without subscribing to a service, who prefer open-source tooling or wish to customize output – Anay Nayak’s aws-security-viz tool (on GitHub) will fit the bill. I’m impressed by the tool for two reasons: it supports both inbound and outbound visualization and the fact that the tool provides a dot format or svg format output – meaning you can get your hands on the files themselves. I did find the tool a bit of a challenge to run under OS X – I ran in Vagrant. A screenshot of the aws-security-viz tool’s output is below:

aws-security-viz - dot Format Output

Summary:

Hopefully the screenshot and description of each tool has provided enough of a description that you’ll be able to identify which tool that will best meet your needs.

Reducing Costs on AWS

Reducing Costs on AWS

One of the requests I get from growing or larger AWS customers is “how do I reduce my Amazon Web Services bill”? There are a lot of analysis tools available – this post is not going to focus on those. Instead I’ll be focusing a lesser known option for reducing cost – working with an Amazon Web Services Reseller or buying direct from Amazon.

How it works?

Typically AWS sells directly to distributors (avnet, datapipe, Ingram) which then sell to resellers – this resale is typically done at a percentage reduction off of a monthly Amazon bill. The distributors then sell to resellers, who then sell directly to AWS customers. The resellers typically have a small percentage markup as they have to handle billing and issues that might arise. The resellers also know that the AWS service itself is a commodity – so they strive to provide some additional benefit:

  • A managed service contract, where they offer monitoring or incident response.
  • Cost analysis or other tooling.
  • Suggestions to help you further lower your costs – they know you’ll leave if they aren’t acting in your best interests.

Which customers benefit?

Depends on the customer’s needs – if the customer benefits from a resellers value add(s), then working through a reseller is a good option. However, if cost savings is the entirety of a customer’s interest, larger or quickly growing customers tend to benefit the most. When reselling AWS, the reseller has to take on the additional overhead of billing and waiting for payment – this is a fixed cost (the cost of collection) and some risk (the risk of non-payment). As the reseller typically adds some percentage when reselling to AWS customers, larger accounts generate more revenue for a reseller (compare a 2% markup on a $100,000/mo account and a $10,000/mo account) – thus, resellers may be more willing to offer large discounts to larger accounts, whereas offering a discount to a smaller AWS customer may not be in anyone’s interest.

What if I am a small customer?

Smaller customers might be able to take advantage of an AWS Activate package, which are geared toward smaller startups with a particular degree of pedigree.

What are the discount amounts?

Short answer: I don’t have the freedom to be specific and I can not find discount percentages publicly available online.

Long answer: With the lack of transparency (apparently this is called “information asymmetry”) the entire process feels like purchasing a car in the United States. I’d be happy to make myself available If you have questions about the ecosystem or how to negotiate the best deal – ask in person or send me an email at colin@cloudavail.com.

Using Elastic Beanstalk with a Docker Registry

Using Elastic Beanstalk with DockerHub or a Docker Registry

Combining DockerHub (using either public or private repositories) or your own Docker registry and Amazon’s Elastic Beanstalk creates a very simple method of delivering a versioned application into a production environment.

The Benefit of Elastic Beanstalk with DockerHub:

For smaller engineering organizations, for organizations running lean on Operations or just staying within the walled-garden of AWS the Elastic Beanstalk <-> DockerHub/Registry combination combines low operational overhead, versioned application rollout/rollback and reliable PaaS service. Assuming you are already building Docker images, the entire setup can be completed within a half of a day.

Overview of Elastic Beanstalk and Docker Deployment Flow:

  1. Build source code and dependencies into a Docker image.
  2. Push an image to DockerHub (use “docker push”)
  3. Push an “Application Version” (use a Dockerfile).
  4. Deploy the application version to a “QA”/”Staging”/”Pre-production” environment, if required.
  5. Deploy the application version to Production

The diagram below is a visual representation of the deployment flow, from start (creating an image) to finish (deployment into Production).

ElasticBeanstalk Deployment from DockerHub

Initial Setup:

If you wish to utilize a private DockerHub repository or your own registry/repository combination and you requires authentication, you’ll need to do a bit of preparation. This preparation is described below.

1. You will need to create and upload a “.dockercfg” file to S3. This file provides authentication information to Amazon’s Elastic Beanstalk. The dockercfg file contains the following text:

{"https://index.docker.io/v1/":{"auth":"3srCWya6OO3y9KUkKLrNZNunIMGiB2Fc","email":"colin@cloudavail.com"}}

You can create a .dockercfg file by running docker login and entering your username and password and the login prompt. This will create a .dockercfg file in the format required by Elastic Beanstalk.

2. If you do use a dockercfg file, the Elastic Beanstalk hosts will need access to it through an AWS IAM profile.

Performing a Deployment:

Upload an Image to DockerHub

Upload Docker image(s) to DockerHub. This makes images available for Elastic Beanstalk or any other host capable of running Docker. A typical Docker push can be done through the command line:

docker push my_organization/my_repository:version (example: cloudavail/test_tomcat_app:0.1.0)

or as a target when using a build tool. For example, I’ve pushed Docker code using Transmode’s gradle-docker plugin.

Create a Dockerrun file for the given “Application Version”

The Dockerrun file is used to describe an application to the Elastic Beanstalk platform. Practically speaking a Dockerun file creates an “Application Version” and instructs Elastic Beanstalk how to do both of the following:

  • “get” an image from DockerHub
  • how to configure a Docker container.

A dockerrun file looks something akin to the following:

{
  "AWSEBDockerrunVersion": "1",
  "Authentication": {
    "Bucket": "my_bucket",
    "Key": "my_dockercfg_file"
  },
  "Image": {
    "Name": "cloudavail/my_tomcat_app:0.1.0",
    "Update": "true"
  },
 "Ports": [
    {
      "ContainerPort": "8080"
    }
  ]
}

The “Authentication” section is required if authentication is needed to access images that are stored in a private repository: “my_bucket” is the name of the bucket where the authentication file is stored and “Key” is the path to the given authentication file. In my example, I’d use something akin to the following

"Bucket": "cloudavail_releases",
 "Key": "docker/dockercfg"

The “Image” section contains the path to a given image and tag.
The “Ports” section contains the port that should be open – which, by default, is mapped to port 80.

Upload the “Application Version” to Elastic Beanstalk

You’ll need to submit the previously created “Dockerrun” file to Amazon. You can do this in one of two ways:

  1. Upload through the UI.
  2. Upload the Dockerrun file to S3 and then issue an API call using Amazon’s CLI tools or custom tooling of your choice.

One cool thing to note – Amazon retains old application versions to allow versioned rollout and rollback. For instance, you can select an application version “0.1.2”, deploy this version to QA, test and then deploy the exact same version to Production. If you needed to rollback, you can select the application version 0.1.1 and deploy this version to Production. The screenshot below demonstrates this:

ElasticBeanstalk - Application Versions

Deploy the newly created Application Version
  1. Deploy through the UI.
  2. Deploy through an API call using Amazon’s CLI tools or custom tooling of your choice.

Exploring Eventual Consistency of the EC2 API

Exploring Eventual Consistency of the EC2 API

I’ve recently been assisting on a project that relies heavily on the boto python interface to manage AWS. In particular the project creates a security group and then subsequently queries boto to get and set attributes. Due to the eventually consistent nature of Amazon’s EC2 API the results were frustrating – for instance the program would to create a group and assign rules, but the authorize_security_group API call would occasionally fail because the group didn’t exist.

I did notice during testing that particular calls were more or less likely to fail and wanted to explore this further. In particular, I thought I was seeing that:

  1. API calls using VPC resources failed more frequently
  2. Security Group “Gets” using filters failed more frequently than those using using names or security group ids.
  3. Security Group “Gets” using names failed more frequently than those using security group ids.

I decided that I would run some tests to determine the consistency of the Amazon EC2 API and to test the success of the different commands.

The Test:

Note that the test is available in my “Snippets” github repository by following this link.

  1. Create an EC2 security group.
  2. Get the security group using either a filter, groupname or security group id. I used the “get_all_security_groups” boto method for this.
  3. Delete the EC2 security group.

Test Results:

A screenshot of the ec2 api test results for a run creating, getting (using a filter) and deleting 100 security groups in EC2-VPC.

ec2_api_eventual_consistency

Group in EC2-Classic, using name based filter for gets:

note that you should *never* do this – if groups with the same name exists in both EC2-Classic and EC2-VPC this will return *both* groups.

group_filter = {‘group-name’: group_name}
groups = conn.get_all_security_groups(filters=group_filter)

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

Group in EC2-VPC, using name based filters for gets:

group_filter = {‘group-name’: group_name, ‘vpc-id’: vpc_id}
groups = conn.get_all_security_groups(filters=group_filter)

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 32
  • errors_delete: 38

Group in EC2-Classic, using group_names for gets:

groups = conn.get_all_security_groups(groupnames=[group_name])

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

# note that errors in “gets” and “deletes” did occur in occasional runs of this test

Group in EC2-VPC, using group_names for gets:

groups = conn.get_all_security_groups(groupnames=[group_name])
Invalid value ‘test-1’ for groupName. You may not reference Amazon VPC security groups by name. Please use the corresponding id for this operation.

Group in EC2-Classic, using group_ids for gets:

groups = conn.get_all_security_groups(group_ids=[created.id])

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

Group in EC2-VPC, using group_ids for gets:

groups = conn.get_all_security_groups(group_ids=[created.id])

Results:

  • errors_create: 0
  • errors_get: 42
  • errors_count: 42
  • errors_delete: 31

Conclusion:

More to come in a future post, but I’m going to be testing the following:

  1. caching result data to reduce the number of EC2 API calls. For example, I am experimenting with a cache that stores the the returned security group object and can be re-used for subsequent queries.
  2. eliminating the need for subsequent calls – for instance, create a resource and set a resource’s attributes in a single call, if possible.
  3. a back-off algorithm, such as the backoff algorithm suggested in the AWS General Reference: Error Retries and Exponential Backoff document

 

VPC Introduction – Part 3

VPC Introduction – Part 3

Part 3 of my blog post on VPC will cover creating a CloudFormation Stack containing a VPC. A sample CloudFormation stack that can be used to create the VPC described in this blog post can be downloaded from my “Snippets” Repository on GitHub.

  • each resource required by a VPC configuration in a CloudFormation template is listed
  • the resource is then defined a list of required properties for that resource is provided (example: a subnet resource must contain an IP range)
  • if helpful, I elaborate on the properties of that resource (example: the VPCZoneIdentifier of an Auto Scaling Group is required to place Auto Scaling Group instances within a VPC)
  • lastly, a snippet creating the resource is provided if it would be helpful

Resources:

VPC: resource type “AWS::EC2::VPC”. This defines the VPC resource itself. A description of the VPC resource is available in VPC Introduction – Part 1.

Subnet: resource type “AWS::EC2::Subnet”. This resource defines a subnet within the VPC. The Subnet resource definition must contain all of the following:

  • CidrBlock – example
  • VpcId – the VPC to which the subnet will be associated

The Subnet resource may also contain an “AvailabilityZone” property. Amazon recommends that you allow Amazon to place these resources automatically with an Availability Zone – until I have evidence that Amazon will place subnet resources in alternating Availabilty Zones (for instance, they will always be placed in “us-east-1a” and “us-east-1b”) I recommend that the Availability Zone be defined. Further description of the Subnet resource is available in VPC Introduction – Part 1.

A snippet defining a subnet with a VPC is below:

"SubnetC" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : "us-east-1c",
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
}

SubnetRouteTableAssociation: resource type “AWS::EC2::SubnetRouteTableAssociation”. The Subnet resource definition must contain all of the following:

  • a RouteTableId – this should reference the Route Table that the Subnet will be associated with
  • a SubnetId – this should reference the Subnet that the RouteTable will be associated with

Two notes:

  • The SubnetRouteTableAssociation is not required by a Subnet resource – if you don’t include a SubnetRouteTableAssociation then the Subnet will be associated with the “Default” route table. I’d recommend against using this configuration, however – if you use the “default” configuration for a subnet and then desire to make a change, this change will impact *all* subnets that use the “Default” route table. For instance, if you have a four subnets using the “Default” Route Table and decide two of these subnets need Internet access and two do not, you’ll need to create a new Route Table, Route Table association and then associate the two subnets that require Internet access with a new Route Table. Easier to plan ahead.
  • I was surprised to find that the SubnetRouteTableAssociation does not need a “DependsOn” attribute for both of the Subnet and RouteTable it is to be associated with. When viewing the “Events” from the VPC CloudFormation table these resources were created in the correct order: meaning Subnet and RouteTable both created before the SubnetRouteTableAssociation.

Internet Gateway: resource type “AWS::EC2::InternetGateway”. This resource provides a gateway to the Internet. A description of the Internet Gateway resource is in VPC Introduction – Part 2.

VPC Gateway Attachment: resource type “AWS::EC2::VPCGatewayAttachment”. This resource attaches a gateway to a VPC. The VPCGatewayAttachment resource definition must contain both of the following:

  • the VPC to which the gateway will be attached
  • the gateway ID (which is either InternetGatewayId or VpnGatewayId)

An example VPC Gateway Attachment is Below:

"VPCGatewayAttachment" : {
  "Type" : "AWS::EC2::VPCGatewayAttachment",
  "Properties" : {
    "InternetGatewayId" : { "Ref" : "InternetGateway" },
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route Table: resource type “AWS::EC2::RouteTable”. This resource defines a route table that informs Amazon where to route traffic. If you wish to route traffic beyond the VPC itself you will need to define “Route” resources. The Route Table Resource definition must link the Route Table to the VPC itself. A further description of the Route Table resource is in VPC Introduction – Part 1. A Route Table snippet is below:

"PublicInternetRouteTable" : {
  "Type" : "AWS::EC2::RouteTable",
  "Properties" : {
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route: resource type “AWS::EC2::Route”. This resource creates a route in “Route Table.” The Route resource definition must contain all of the following:

  • a DestinationCidrBlock
  • a target (which is one of GatewayID, InstanceID, NetworkInterfaceId or VpcPeeringConnectionId)
  • a RouteTableID

An example providing a route to the Internet is below:

"PublicInternetRoute" : {
  "Type" : "AWS::EC2::Route",
  "DependsOn" : [ "InternetGateway", "PublicInternetRouteTable" ] ,
  "Properties" : {
    "DestinationCidrBlock" : "0.0.0.0/0",
    "GatewayId" : { "Ref" : "InternetGateway" },
    "RouteTableId" : { "Ref" : "PublicInternetRouteTable" }
  }
}

AutoScalingGroup: resource type “AWS::AutoScaling::AutoScalingGroup”. This resource creates an Auto Scaling Group. The AutoScalingGroup resource definition must contain all of the following properties:

  • AvailabilityZones – a list of Availability Zones where instances can be launched.
  • an AMI (which must be provided by either LaunchConfigurationName or InstanceId)
  • MinSize – the minimum size of the group
  • MaxSize – the maximum size of the group

Lastly, if you want the Auto Scaling Group to be able to placed instances within a VPC, you’ll need to provide a VPCZoneIdentifier which is described below:

  • VPCZoneIdentifier – the VPCZoneIdentifier is reponsible for associating an Auto Scaling Group with a VPC. If you don’t specify a VPCZoneIdentifier your Auto Scaling Group instances will be placed within EC2-Classic.

A sample AutoScalingGroup resource is provided below:

"AutoScalingGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "AvailabilityZones" : [ "us-east-1c" , "us-east-1d" ],
    "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
    "MinSize" : "1",
    "MaxSize" : "2",
    "DesiredCapacity" : "2"
  }
}

LaunchConfiguration: resource type “AWS::AutoScaling::LaunchConfiguration”. This resource creates a LaunchConfiguration. The LaunchConfiguration resource definition must contain the following properties:

  • ImageId: the AMI ID to be used when creating Instances.
  • InstanceType: the instance type to be used when creatng instances.

There are a few other properties of note when creating a Launch Configuration that will be used by an Auto Scaling Group within a VPC:

  • AssociatePublicIpAddress: if set to “true” the AssociatePublicIpAddress property will automatically assign IP addresses to all instances launched within the defined Auto Scaling Group. Note that if you set AssociatePublicIpAddress to true, you must also specify a VPCZoneIdentifier within the associated Auto Scaling Group.
  • KeyName: you may be tempted to create instances and not specify a KeyName, instead relying on the UserData specified within a Launch Configuration to configure the instance so I can login. This turns out to be impractical when troubleshooting – if the bootstrap or configuration of an instance fails you won’t be able to login to resolve issues.
  • SecurityGroups: specifies one or more security groups that a Launch Configuration should be associated with. If no Security Group is specified, instances will be launched without any Security Groups (as opposed to be assigned to a “default” security group)
  • Spot Price (http://aws.amazon.com/ec2/purchasing-options/spot-instances/): the use of the spot price property deserves a blog post all its own, but suffice to say that if you bid the “OnDemand” price across all available AZs you will generally have an instance available to you, but will pay only the current spot price of an instance.
  • UserData: user-data allows you to run a script during instance startup. A typical use case is to inject a “bootstrap” script that brings the instance into service – for instance, by running configuration management or by starting required services.

A sample Launch Configuration is below:

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Properties" : {
    "KeyName" : { "Ref" : "KeyName" },
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "SecurityGroups" : [ { "Ref" : "SecurityGroup" } ],
    "InstanceType" : { "Ref" : "InstanceType" }
  }
}

SecurityGroup: resource type “AWS::EC2::SecurityGroup. This resource creates an Security Group. The Security Group does not require any properties, but a number of them should be defined when used with VPC.

  • VpcId: required if you wish to use the Security Group within a VPC.
  • SecurityGroupEgress: provides rules that allow egress of traffic to the resource that uses the Security Group.
  • SecurityGroupIngress: provides rules that allow ingress of traffic to the resource that uses the Security Group.

A sample Security Group resource is below:

"SampleServerSecurityGroup" : {
  "Type" : "AWS::EC2::SecurityGroup",
  "Properties" : {
    "GroupDescription" : "Enable SSH",
    "VpcId" : { "Ref" : "VPC" },
    "SecurityGroupIngress" : [ {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    } ],
    "SecurityGroupEgress" : [ { 
      "IpProtocol" : "tcp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "udp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, { 
      "IpProtocol" : "icmp",
      "FromPort" : "-1",
      "ToPort" : "-1",
      "CidrIp" : "0.0.0.0/0"
      } ]
    }
  }
}