Exploring Eventual Consistency of the EC2 API

Exploring Eventual Consistency of the EC2 API

I’ve recently been assisting on a project that relies heavily on the boto python interface to manage AWS. In particular the project creates a security group and then subsequently queries boto to get and set attributes. Due to the eventually consistent nature of Amazon’s EC2 API the results were frustrating – for instance the program would to create a group and assign rules, but the authorize_security_group API call would occasionally fail because the group didn’t exist.

I did notice during testing that particular calls were more or less likely to fail and wanted to explore this further. In particular, I thought I was seeing that:

  1. API calls using VPC resources failed more frequently
  2. Security Group “Gets” using filters failed more frequently than those using using names or security group ids.
  3. Security Group “Gets” using names failed more frequently than those using security group ids.

I decided that I would run some tests to determine the consistency of the Amazon EC2 API and to test the success of the different commands.

The Test:

Note that the test is available in my “Snippets” github repository by following this link.

  1. Create an EC2 security group.
  2. Get the security group using either a filter, groupname or security group id. I used the “get_all_security_groups” boto method for this.
  3. Delete the EC2 security group.

Test Results:

A screenshot of the ec2 api test results for a run creating, getting (using a filter) and deleting 100 security groups in EC2-VPC.

ec2_api_eventual_consistency

Group in EC2-Classic, using name based filter for gets:

note that you should *never* do this – if groups with the same name exists in both EC2-Classic and EC2-VPC this will return *both* groups.

group_filter = {‘group-name’: group_name}
groups = conn.get_all_security_groups(filters=group_filter)

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

Group in EC2-VPC, using name based filters for gets:

group_filter = {‘group-name’: group_name, ‘vpc-id’: vpc_id}
groups = conn.get_all_security_groups(filters=group_filter)

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 32
  • errors_delete: 38

Group in EC2-Classic, using group_names for gets:

groups = conn.get_all_security_groups(groupnames=[group_name])

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

# note that errors in “gets” and “deletes” did occur in occasional runs of this test

Group in EC2-VPC, using group_names for gets:

groups = conn.get_all_security_groups(groupnames=[group_name])
Invalid value ‘test-1’ for groupName. You may not reference Amazon VPC security groups by name. Please use the corresponding id for this operation.

Group in EC2-Classic, using group_ids for gets:

groups = conn.get_all_security_groups(group_ids=[created.id])

Results:

  • errors_create: 0
  • errors_get: 0
  • errors_count: 0
  • errors_delete: 0

Group in EC2-VPC, using group_ids for gets:

groups = conn.get_all_security_groups(group_ids=[created.id])

Results:

  • errors_create: 0
  • errors_get: 42
  • errors_count: 42
  • errors_delete: 31

Conclusion:

More to come in a future post, but I’m going to be testing the following:

  1. caching result data to reduce the number of EC2 API calls. For example, I am experimenting with a cache that stores the the returned security group object and can be re-used for subsequent queries.
  2. eliminating the need for subsequent calls – for instance, create a resource and set a resource’s attributes in a single call, if possible.
  3. a back-off algorithm, such as the backoff algorithm suggested in the AWS General Reference: Error Retries and Exponential Backoff document

 

Mapping of Vagrant Networks to VirtualBox Adapter Types

Mapping of Vagrant Networks to VirtualBox Adapter Types

Users of Vagrant may have noticed that the the configuration options for networks (config.vm.network) in Vagrant don’t match up cleanly to the network Adapter types in VirtualBox – as an example, when defining a “private_network” in a Vagrantfile Vagrant will actually provision your virtual appliance with a “Host-only Adapter.” The remainder of this blog post will describe the mapping of network configuration in a Vagrantfile to the adapters in VirtualBox in more detail.

forwarded_port Network Configuration

“forwarded_port” maps to a “NAT” type adapter
note that Vagrant always installs a NAT type adapter as the first network interface (typically en0 or eth0) when using VirtualBox. This is not well documented on the Internet, but can be found in Mitchell Hashimoto’s Vagrant: Up and Running.

private_network Network

“private_network” maps to a “Host-only Adapter”
if you provide an IP address (example: config.vm.network “private_network”, ip: “192.168.2.2”) which does not exist on a VirtualBox “Host-only Network” (such as vboxnet0) Vagrant will instruct VirtualBox to create a network for your virtual appliance.

public_network Network:

“public_network” maps to a “Bridged Adapter”
note that if you are using a public_network without specifying an adapter to bridge to you will be asked “what interface should the network bridge to” and be given a list of physical interfaces provided buy the Host OS.

Screenshot resulting from Testing

I attached a screenshot that resulted from the Vagrantfile that I used to create this mapping. Essentially the screenshot shows an instance that contains one of each type of adapter:

Vagrant-VirtualBox-Create_all_network_types

Vagrantfile used in Testing

To perform the mapping test I created a Vagrantfile that attaches each type of network interface. This Vagrantfile is available in my Snippets GitHub repository at https://github.com/colinbjohnson/snippets/blob/master/vagrant/create_all_network_types/Vagrantfile.

Questions or Comments?

If you have any questions please feel free to comment.

VPC Introduction – Part 3

VPC Introduction – Part 3

Part 3 of my blog post on VPC will cover creating a CloudFormation Stack containing a VPC. A sample CloudFormation stack that can be used to create the VPC described in this blog post can be downloaded from my “Snippets” Repository on GitHub.

  • each resource required by a VPC configuration in a CloudFormation template is listed
  • the resource is then defined a list of required properties for that resource is provided (example: a subnet resource must contain an IP range)
  • if helpful, I elaborate on the properties of that resource (example: the VPCZoneIdentifier of an Auto Scaling Group is required to place Auto Scaling Group instances within a VPC)
  • lastly, a snippet creating the resource is provided if it would be helpful

Resources:

VPC: resource type “AWS::EC2::VPC”. This defines the VPC resource itself. A description of the VPC resource is available in VPC Introduction – Part 1.

Subnet: resource type “AWS::EC2::Subnet”. This resource defines a subnet within the VPC. The Subnet resource definition must contain all of the following:

  • CidrBlock – example
  • VpcId – the VPC to which the subnet will be associated

The Subnet resource may also contain an “AvailabilityZone” property. Amazon recommends that you allow Amazon to place these resources automatically with an Availability Zone – until I have evidence that Amazon will place subnet resources in alternating Availabilty Zones (for instance, they will always be placed in “us-east-1a” and “us-east-1b”) I recommend that the Availability Zone be defined. Further description of the Subnet resource is available in VPC Introduction – Part 1.

A snippet defining a subnet with a VPC is below:

"SubnetC" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : "us-east-1c",
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
}

SubnetRouteTableAssociation: resource type “AWS::EC2::SubnetRouteTableAssociation”. The Subnet resource definition must contain all of the following:

  • a RouteTableId – this should reference the Route Table that the Subnet will be associated with
  • a SubnetId – this should reference the Subnet that the RouteTable will be associated with

Two notes:

  • The SubnetRouteTableAssociation is not required by a Subnet resource – if you don’t include a SubnetRouteTableAssociation then the Subnet will be associated with the “Default” route table. I’d recommend against using this configuration, however – if you use the “default” configuration for a subnet and then desire to make a change, this change will impact *all* subnets that use the “Default” route table. For instance, if you have a four subnets using the “Default” Route Table and decide two of these subnets need Internet access and two do not, you’ll need to create a new Route Table, Route Table association and then associate the two subnets that require Internet access with a new Route Table. Easier to plan ahead.
  • I was surprised to find that the SubnetRouteTableAssociation does not need a “DependsOn” attribute for both of the Subnet and RouteTable it is to be associated with. When viewing the “Events” from the VPC CloudFormation table these resources were created in the correct order: meaning Subnet and RouteTable both created before the SubnetRouteTableAssociation.

Internet Gateway: resource type “AWS::EC2::InternetGateway”. This resource provides a gateway to the Internet. A description of the Internet Gateway resource is in VPC Introduction – Part 2.

VPC Gateway Attachment: resource type “AWS::EC2::VPCGatewayAttachment”. This resource attaches a gateway to a VPC. The VPCGatewayAttachment resource definition must contain both of the following:

  • the VPC to which the gateway will be attached
  • the gateway ID (which is either InternetGatewayId or VpnGatewayId)

An example VPC Gateway Attachment is Below:

"VPCGatewayAttachment" : {
  "Type" : "AWS::EC2::VPCGatewayAttachment",
  "Properties" : {
    "InternetGatewayId" : { "Ref" : "InternetGateway" },
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route Table: resource type “AWS::EC2::RouteTable”. This resource defines a route table that informs Amazon where to route traffic. If you wish to route traffic beyond the VPC itself you will need to define “Route” resources. The Route Table Resource definition must link the Route Table to the VPC itself. A further description of the Route Table resource is in VPC Introduction – Part 1. A Route Table snippet is below:

"PublicInternetRouteTable" : {
  "Type" : "AWS::EC2::RouteTable",
  "Properties" : {
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route: resource type “AWS::EC2::Route”. This resource creates a route in “Route Table.” The Route resource definition must contain all of the following:

  • a DestinationCidrBlock
  • a target (which is one of GatewayID, InstanceID, NetworkInterfaceId or VpcPeeringConnectionId)
  • a RouteTableID

An example providing a route to the Internet is below:

"PublicInternetRoute" : {
  "Type" : "AWS::EC2::Route",
  "DependsOn" : [ "InternetGateway", "PublicInternetRouteTable" ] ,
  "Properties" : {
    "DestinationCidrBlock" : "0.0.0.0/0",
    "GatewayId" : { "Ref" : "InternetGateway" },
    "RouteTableId" : { "Ref" : "PublicInternetRouteTable" }
  }
}

AutoScalingGroup: resource type “AWS::AutoScaling::AutoScalingGroup”. This resource creates an Auto Scaling Group. The AutoScalingGroup resource definition must contain all of the following properties:

  • AvailabilityZones – a list of Availability Zones where instances can be launched.
  • an AMI (which must be provided by either LaunchConfigurationName or InstanceId)
  • MinSize – the minimum size of the group
  • MaxSize – the maximum size of the group

Lastly, if you want the Auto Scaling Group to be able to placed instances within a VPC, you’ll need to provide a VPCZoneIdentifier which is described below:

  • VPCZoneIdentifier – the VPCZoneIdentifier is reponsible for associating an Auto Scaling Group with a VPC. If you don’t specify a VPCZoneIdentifier your Auto Scaling Group instances will be placed within EC2-Classic.

A sample AutoScalingGroup resource is provided below:

"AutoScalingGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "AvailabilityZones" : [ "us-east-1c" , "us-east-1d" ],
    "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
    "MinSize" : "1",
    "MaxSize" : "2",
    "DesiredCapacity" : "2"
  }
}

LaunchConfiguration: resource type “AWS::AutoScaling::LaunchConfiguration”. This resource creates a LaunchConfiguration. The LaunchConfiguration resource definition must contain the following properties:

  • ImageId: the AMI ID to be used when creating Instances.
  • InstanceType: the instance type to be used when creatng instances.

There are a few other properties of note when creating a Launch Configuration that will be used by an Auto Scaling Group within a VPC:

  • AssociatePublicIpAddress: if set to “true” the AssociatePublicIpAddress property will automatically assign IP addresses to all instances launched within the defined Auto Scaling Group. Note that if you set AssociatePublicIpAddress to true, you must also specify a VPCZoneIdentifier within the associated Auto Scaling Group.
  • KeyName: you may be tempted to create instances and not specify a KeyName, instead relying on the UserData specified within a Launch Configuration to configure the instance so I can login. This turns out to be impractical when troubleshooting – if the bootstrap or configuration of an instance fails you won’t be able to login to resolve issues.
  • SecurityGroups: specifies one or more security groups that a Launch Configuration should be associated with. If no Security Group is specified, instances will be launched without any Security Groups (as opposed to be assigned to a “default” security group)
  • Spot Price (http://aws.amazon.com/ec2/purchasing-options/spot-instances/): the use of the spot price property deserves a blog post all its own, but suffice to say that if you bid the “OnDemand” price across all available AZs you will generally have an instance available to you, but will pay only the current spot price of an instance.
  • UserData: user-data allows you to run a script during instance startup. A typical use case is to inject a “bootstrap” script that brings the instance into service – for instance, by running configuration management or by starting required services.

A sample Launch Configuration is below:

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Properties" : {
    "KeyName" : { "Ref" : "KeyName" },
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "SecurityGroups" : [ { "Ref" : "SecurityGroup" } ],
    "InstanceType" : { "Ref" : "InstanceType" }
  }
}

SecurityGroup: resource type “AWS::EC2::SecurityGroup. This resource creates an Security Group. The Security Group does not require any properties, but a number of them should be defined when used with VPC.

  • VpcId: required if you wish to use the Security Group within a VPC.
  • SecurityGroupEgress: provides rules that allow egress of traffic to the resource that uses the Security Group.
  • SecurityGroupIngress: provides rules that allow ingress of traffic to the resource that uses the Security Group.

A sample Security Group resource is below:

"SampleServerSecurityGroup" : {
  "Type" : "AWS::EC2::SecurityGroup",
  "Properties" : {
    "GroupDescription" : "Enable SSH",
    "VpcId" : { "Ref" : "VPC" },
    "SecurityGroupIngress" : [ {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    } ],
    "SecurityGroupEgress" : [ { 
      "IpProtocol" : "tcp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "udp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, { 
      "IpProtocol" : "icmp",
      "FromPort" : "-1",
      "ToPort" : "-1",
      "CidrIp" : "0.0.0.0/0"
      } ]
    }
  }
}

VPC Introduction – Part 2

VPC Introduction – Part 2

This is the second part of a 4 part introduction to Amazon’s VPC. Part 1 examined the VPC resource itself, as well as the subnet, Route Table and Network ACL resources. Part 2 examines the Internet Gateway resource, the EC2-VPC Security Group resource and Auto Scaling Groups when used in VPC.

Internet Gateway Resource

An Internet Gateway provides connectivity to the Internet. Simply creating an Internet Gateway resource is not enough to provide access to the Internet, however, you’ll also need to do the following:

  1. Create or modify a Route Table to include a route to the Internet. An Internet route is typically defined as follows: Destination: 0.0.0.0/0, Target: <Internet Gateway Resource Number>
  2. Provide a Network ACL that allows outbound and inbound traffic from the Internet.
  3. Associate any subnet that requires Internet access to the previously created/modified Route Table and Network ACL.
  4. Provide each instance that requires Internet access with a Public IP address – the Internet Gateway does not providing Internet access while using Public IP addresses because the Internet Gateway does not function as a NAT router.

Note that using an Internet Gateway and Public IP addresses for instances is only one way to provide Internet connectivity to EC2 instances – part 3 will cover this in greater depth.

EC2-VPC Security Group Resource

EC2-VPC security groups are comprised of inbound and outbound rules and are associated with EC2 instances and other resources such as RDS Security Groups or ElastiCache. Inbound and Outbound rules filter based on IP addressing or security groups and port and both default to “Deny” traffic if not explicitly allowed by a rule. I’ve described the inbound and outbound rules below:

1. Inbound Rules. Inbound rules filter based on a packet’s source IP address or security group and source port. Amazon provides a number of rule templates for you (for ssh and HTTP, for example). Custom rules can also be created – a rule allowing port 81 in from the Internet would look like:

  • Type: “Custom TCP Rule”
  • Protocol: TCP
  • Port Range: 81
  • Source: 0.0.0.0/0

2. Outbound Rules. Outbound rules filter traffic based on a destination packet’s IP address or security group and destination port. An example outbound rule that allows unfettered tcp access to the Internet is below:

  • Type: All TCP Rule
  • Protocol: TCP
  • Port Range: 0 – 65535
  • Destination: Anywhere: 0.0.0.0/0

An example outbound rule that allows only access to HTTP resources on the Internet is below:

  • Type: HTTP
  • Protocol: TCP
  • Port Range: 80
  • Destination: Anywhere: 0.0.0.0/0

notice that we allow port 80 as the destination port but no other ports.

If you are familiar with EC2-Classic, the differences from EC2-Classic Security Groups are in the VPC Security Groups User Guide under “VPC Security Group Differences.”

Auto Scaling Groups and Launch Configurations

Auto Scaling Groups and Launch Configurations in VPC differ only slightly from Auto Scaling Groups and Launch Configurations in EC2-Classic. The two important differences are described below:

  • An Auto Scaling Group must have one or more associated subnets in order to launch instances.
  • A Launch Configuration includes an “IP Address Type” – this allows instances to be automatically given a public IP address.

The image below describes a VPC that provides Internet access to instances in two subnets. The VPC is comprised of a VPC, an Internet Gateway, a Route Table, a Network ACL, two subnets, an EC2-VPC Security Group, an Auto Scaling Group, a Launch Configuration and the instances that make up the Auto Scaling Group.

VPC - Internet Gateway and SG and ASG

VPC Introduction – Part 1

VPC Introduction – Part 1

My 3 part VPC introduction will provide an understanding of each resource that makes up a VPC, how the resources relate and how to stand a VPC up using AWS’s CloudFormation. The content of each part is as follows:

  • part 1: covers the VPC resource itself, subnets, Network ACLs and Route Tables
  • part 2: covers Internet Gateway, EC2-VPC Security Groups, Auto Scaling Groups and Launch Configurations
  • part 3: the CloudFormation template itself and each resource that makes up a CloudFormation Stack that creates a VPC

VPC Resources and Subnets:

VPC Resource

The “VPC” resource itself. At its most simple, a VPC is:

  1. A VPC Resource defines a CIDR block of addresses. For example, vpc-c057aaaa5 uses the IP range 10.0.0.0/24.
  2. A VPC Resource must be associated with three additional resources that Amazon will automatically create when you build a VPC. The three required resources are:
    1. a DHCP options set
    2. a route table
    3. a network ACL.
  3. A VPC resource should contain one or more subnet resources – the subnet resources are required for hosting Amazon resources require an IP address.
Subnets

Subnets are logical divisions of the VPC resource into smaller networks. You can build a VPC that contains only one subnet, but best practice dictates that you have at least two subnets per VPC, with each of these to subnets in a different availability zone. Subnet resources are described below:

  1. A subnet must contain an CIDR IP range that is within the VPC and does not overlap with another subnet. For instance, in the VPC with IP address range 10.0.0.0/24, I can create two subnets comprised of IP ranges 10.0.0.0/25 and 10.0.0.128/25.
  2. A subnet may only be in one Availability Zone but may not span Availability Zones. For instance, subnet-09b5e321 may be in us-east-1e or us-east-1b both not both us-east-1e and us-east-1b.
  3. A subnet must be associated with:
    1. a Route Table (providing one or more “target” for outbound packets – a “target” is Amazon’s terminology for “next hop.”)
    2. a Network ACL (providing control of inbound and outbound traffic)
  4. A subnet can automatically assign public IP addresses to instances when they are brought into service. As of July 4, 2014 the public assignment of IP addresses can not be done within an “EC2::Subnet” type of resource from within CloudFormation, but rather must be applied on either the Auto Scaling Group or EC2 Instance types.

The image below describes a VPC that contains the CIDR IP Range 10.0.0.0/24 and two subnets: 10.0.0.0/25 and 10.0.0.128/25.

VPC - VPC and Subnet

A quick notes regarding VPC IP addressing:

VPCs can use the same IP address range. For instance, vpc-6e558a0b and vpc-00548b65 can both use IP ranges 10.0.0.0/16 – as they are both separate private clouds the addresses will not conflict. I would recommend against this practice – routing traffic to two different VPCs that use the same IP address scheme is technically possible (NAT) but would be difficult.

Route Tables and Network ACLS:

As mentioned earlier, a VPC requires three additional resources that Amazon will create for you. These three resources are a Route Table, a Network ACL and a DHCP Options Set. The Route Table and Network ACL resources are described in further detail below.

Route Tables

Route tables determine where network traffic is directed.

  • A Route Table contains one or more routes that are responsible for determining the “target” of packets sent to a particular destination
  • A Route Table always contains a “local” route that directs traffic within the subnet. Assuming a VPC with CIDR IP Range 10.0.0.0/24, Amazon creates the following route automatically:
    • Destination: 10.0.0.0/24
    • Target: local
  • A Route Table will may contain more than the local routes that directs traffic within a subnet – these routes will allow instances to send traffic beyond the VPC where they are located. The rout described below will route traffic to the Public Internet:
    • Destination: 0.0.0.0/0
    • Target: igw-3547bf50 (igw-3547bf50 is an Internet Gateway)
  • Each subnet can be associated with only one route table. However, a route table may be used by more than one subnet.
    • Allowing only one route table per subnet makes sense: merging two route tables would be complex and might even result in conflict in where to send traffic.
    • Allowing a route table to be used by multiple subnets makes sense: a common case might be two subnets that are allowed to send data within the VPC and also to the Internet – given that they share identical routes within the VPC (route 10.0.0.0/24 is “local”) and to the Internet (route 0.0.0.0/0 is igw-3547bf50), sharing a route table with multiple subnets makes sense.
Network ACLs

Network ACLs determine if traffic can be passed.

  • Network ACLs are required for inbound and outbound traffic.
    • inbound rules match on source IP of packet
    • outbound rules match on destination IP of packet
  • Network ACLs are stateless (as opposed to stateful)
  • Network ACLs only allow filtering based on IP address and port – for instance, I can allow traffic in from IP addresses 10.0.0.128/25 on port 80. Note that a Network ACL does not understand the concept of a security group, so I can not allow traffic in from sg-6482490e on port 80.

The image below describes a VPC that allows traffic flow between within subnets as and also allows traffic to flow to and from the Public Internet. Notice the Network ACL has an “Inbound” rule allowing traffic from source 0.0.0.0/0 and an Outbound rule allowing traffic to destination 0.0.0.0/0. The Route Table contains a rule directory all traffic to 0.0.0.0/0 to be delivered to an Internet Gateway.

VPC - Route Table and Network ACL

More Information