Using AWS CloudFormation’s Transform Function

Why use CloudFormation’s Transform Function?

There are two good reasons for using CloudFormation’s “Transform” function to include files. These two reasons are described below:

  1. Consistency.
    1. By including a snippet in each and every CloudFormation template – you’ll ensure that the included code is the same, stack to stack.
  2. Code reuse.
    1. You won’t need to update code across multiple stacks when you need to make changes. You will need to update stacks to get changes made to the included files – but you won’t have to update the actual code in each stack.

How to do this?

Creating a CloudFormation File that uses an Include.

You need to include a Fn::Transform statement where the given file is to be included. An example included is below:

Fn::Transform:
  Name: AWS::Include
  Parameters:
    Location : s3://187376578462-fn-transform-include/ubuntu_ami.yaml

An example of an include in the “Mappings” section of a CloudFormation template would look like:

Mappings:
  Fn::Transform:
    Name: AWS::Include
    Parameters:
      Location : s3://187376578462-fn-transform-include/ubuntu_ami.yaml

Lastly, here is a screenshot of a CloudFormation file that uses an include – see line 29.

CloudFormation - Fn Transform
CloudFormation template that utilizes a Transform function to include a file.

Creating the Included File

You will need to create a file that will be included in a given CloudFormation stack. This file is going to be inserted where the Fn::Transform statement is – this is akin to “import” or “include” in a programming language or “catting” two files together in a *nix Operating System.

The included file should look akin to the following:

AWSRegionArch2AMI:
  us-east-1:
    '64': ami-ddf13fb0
  us-west-1:
    '64': ami-b20542d2
  us-west-2:
    '64': ami-b9ff39d9
CloudFormation - File to be Included
File to be included in a CloudFormation template.

Uploading the Included File

The file that _will be_ included needs to be uploaded to S3. You can do this using the aws s3 command – see below:

aws s3 cp ubuntu_ami.yaml s3://$ubuntu_ami_file_s3_path --region us-west-2
CloudFormation - Included File Upload
AWS S3 command uploading a file to be included in a CloudFormation template.

Creating the CloudFormation Stack with an Include

You’ll need to use the “aws cloudformation deploy” command to deploy or update the given template. An example is below:

aws cloudformation deploy --stack-name FunctionTransformInclude --template-file autoscaling_with_yaml_userdata.yaml --parameter-overrides ubuntuAMISMapping3Location=s3://$ubuntu_ami_file_s3_path --region us-west-2
CloudFormation - Fn Transform Launch Stack
AWS CloudFormation “Deploy” command creating a CloudFormation stack

Summary

I’m planning on using for AMI mappings in particular, as well as for including sections of CloudFormation that might be better generated using code (for instance, user-data might be a consideration). I’ve yet to consider the use of “Fn::Transform / Include” to improve the security of stacks by removing passwords.

If you have questions or comments – reach me at colin@cloudavail.com.

VPC Introduction – Part 3

VPC Introduction – Part 3

Part 3 of my blog post on VPC will cover creating a CloudFormation Stack containing a VPC. A sample CloudFormation stack that can be used to create the VPC described in this blog post can be downloaded from my “Snippets” Repository on GitHub.

  • each resource required by a VPC configuration in a CloudFormation template is listed
  • the resource is then defined a list of required properties for that resource is provided (example: a subnet resource must contain an IP range)
  • if helpful, I elaborate on the properties of that resource (example: the VPCZoneIdentifier of an Auto Scaling Group is required to place Auto Scaling Group instances within a VPC)
  • lastly, a snippet creating the resource is provided if it would be helpful

Resources:

VPC: resource type “AWS::EC2::VPC”. This defines the VPC resource itself. A description of the VPC resource is available in VPC Introduction – Part 1.

Subnet: resource type “AWS::EC2::Subnet”. This resource defines a subnet within the VPC. The Subnet resource definition must contain all of the following:

  • CidrBlock – example
  • VpcId – the VPC to which the subnet will be associated

The Subnet resource may also contain an “AvailabilityZone” property. Amazon recommends that you allow Amazon to place these resources automatically with an Availability Zone – until I have evidence that Amazon will place subnet resources in alternating Availabilty Zones (for instance, they will always be placed in “us-east-1a” and “us-east-1b”) I recommend that the Availability Zone be defined. Further description of the Subnet resource is available in VPC Introduction – Part 1.

A snippet defining a subnet with a VPC is below:

"SubnetC" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : "us-east-1c",
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
}

SubnetRouteTableAssociation: resource type “AWS::EC2::SubnetRouteTableAssociation”. The Subnet resource definition must contain all of the following:

  • a RouteTableId – this should reference the Route Table that the Subnet will be associated with
  • a SubnetId – this should reference the Subnet that the RouteTable will be associated with

Two notes:

  • The SubnetRouteTableAssociation is not required by a Subnet resource – if you don’t include a SubnetRouteTableAssociation then the Subnet will be associated with the “Default” route table. I’d recommend against using this configuration, however – if you use the “default” configuration for a subnet and then desire to make a change, this change will impact *all* subnets that use the “Default” route table. For instance, if you have a four subnets using the “Default” Route Table and decide two of these subnets need Internet access and two do not, you’ll need to create a new Route Table, Route Table association and then associate the two subnets that require Internet access with a new Route Table. Easier to plan ahead.
  • I was surprised to find that the SubnetRouteTableAssociation does not need a “DependsOn” attribute for both of the Subnet and RouteTable it is to be associated with. When viewing the “Events” from the VPC CloudFormation table these resources were created in the correct order: meaning Subnet and RouteTable both created before the SubnetRouteTableAssociation.

Internet Gateway: resource type “AWS::EC2::InternetGateway”. This resource provides a gateway to the Internet. A description of the Internet Gateway resource is in VPC Introduction – Part 2.

VPC Gateway Attachment: resource type “AWS::EC2::VPCGatewayAttachment”. This resource attaches a gateway to a VPC. The VPCGatewayAttachment resource definition must contain both of the following:

  • the VPC to which the gateway will be attached
  • the gateway ID (which is either InternetGatewayId or VpnGatewayId)

An example VPC Gateway Attachment is Below:

"VPCGatewayAttachment" : {
  "Type" : "AWS::EC2::VPCGatewayAttachment",
  "Properties" : {
    "InternetGatewayId" : { "Ref" : "InternetGateway" },
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route Table: resource type “AWS::EC2::RouteTable”. This resource defines a route table that informs Amazon where to route traffic. If you wish to route traffic beyond the VPC itself you will need to define “Route” resources. The Route Table Resource definition must link the Route Table to the VPC itself. A further description of the Route Table resource is in VPC Introduction – Part 1. A Route Table snippet is below:

"PublicInternetRouteTable" : {
  "Type" : "AWS::EC2::RouteTable",
  "Properties" : {
    "VpcId" : { "Ref" : "VPC" }
  }
}

Route: resource type “AWS::EC2::Route”. This resource creates a route in “Route Table.” The Route resource definition must contain all of the following:

  • a DestinationCidrBlock
  • a target (which is one of GatewayID, InstanceID, NetworkInterfaceId or VpcPeeringConnectionId)
  • a RouteTableID

An example providing a route to the Internet is below:

"PublicInternetRoute" : {
  "Type" : "AWS::EC2::Route",
  "DependsOn" : [ "InternetGateway", "PublicInternetRouteTable" ] ,
  "Properties" : {
    "DestinationCidrBlock" : "0.0.0.0/0",
    "GatewayId" : { "Ref" : "InternetGateway" },
    "RouteTableId" : { "Ref" : "PublicInternetRouteTable" }
  }
}

AutoScalingGroup: resource type “AWS::AutoScaling::AutoScalingGroup”. This resource creates an Auto Scaling Group. The AutoScalingGroup resource definition must contain all of the following properties:

  • AvailabilityZones – a list of Availability Zones where instances can be launched.
  • an AMI (which must be provided by either LaunchConfigurationName or InstanceId)
  • MinSize – the minimum size of the group
  • MaxSize – the maximum size of the group

Lastly, if you want the Auto Scaling Group to be able to placed instances within a VPC, you’ll need to provide a VPCZoneIdentifier which is described below:

  • VPCZoneIdentifier – the VPCZoneIdentifier is reponsible for associating an Auto Scaling Group with a VPC. If you don’t specify a VPCZoneIdentifier your Auto Scaling Group instances will be placed within EC2-Classic.

A sample AutoScalingGroup resource is provided below:

"AutoScalingGroup" : {
  "Type" : "AWS::AutoScaling::AutoScalingGroup",
  "Properties" : {
    "AvailabilityZones" : [ "us-east-1c" , "us-east-1d" ],
    "LaunchConfigurationName" : { "Ref" : "LaunchConfig" },
    "MinSize" : "1",
    "MaxSize" : "2",
    "DesiredCapacity" : "2"
  }
}

LaunchConfiguration: resource type “AWS::AutoScaling::LaunchConfiguration”. This resource creates a LaunchConfiguration. The LaunchConfiguration resource definition must contain the following properties:

  • ImageId: the AMI ID to be used when creating Instances.
  • InstanceType: the instance type to be used when creatng instances.

There are a few other properties of note when creating a Launch Configuration that will be used by an Auto Scaling Group within a VPC:

  • AssociatePublicIpAddress: if set to “true” the AssociatePublicIpAddress property will automatically assign IP addresses to all instances launched within the defined Auto Scaling Group. Note that if you set AssociatePublicIpAddress to true, you must also specify a VPCZoneIdentifier within the associated Auto Scaling Group.
  • KeyName: you may be tempted to create instances and not specify a KeyName, instead relying on the UserData specified within a Launch Configuration to configure the instance so I can login. This turns out to be impractical when troubleshooting – if the bootstrap or configuration of an instance fails you won’t be able to login to resolve issues.
  • SecurityGroups: specifies one or more security groups that a Launch Configuration should be associated with. If no Security Group is specified, instances will be launched without any Security Groups (as opposed to be assigned to a “default” security group)
  • Spot Price (http://aws.amazon.com/ec2/purchasing-options/spot-instances/): the use of the spot price property deserves a blog post all its own, but suffice to say that if you bid the “OnDemand” price across all available AZs you will generally have an instance available to you, but will pay only the current spot price of an instance.
  • UserData: user-data allows you to run a script during instance startup. A typical use case is to inject a “bootstrap” script that brings the instance into service – for instance, by running configuration management or by starting required services.

A sample Launch Configuration is below:

"LaunchConfig" : {
  "Type" : "AWS::AutoScaling::LaunchConfiguration",
  "Properties" : {
    "KeyName" : { "Ref" : "KeyName" },
    "ImageId" : { "Fn::FindInMap" : [ "AWSRegionArch2AMI", { "Ref" : "AWS::Region" }, { "Fn::FindInMap" : [ "AWSInstanceType2Arch", { "Ref" : "InstanceType" }, "Arch" ] } ] },
    "SecurityGroups" : [ { "Ref" : "SecurityGroup" } ],
    "InstanceType" : { "Ref" : "InstanceType" }
  }
}

SecurityGroup: resource type “AWS::EC2::SecurityGroup. This resource creates an Security Group. The Security Group does not require any properties, but a number of them should be defined when used with VPC.

  • VpcId: required if you wish to use the Security Group within a VPC.
  • SecurityGroupEgress: provides rules that allow egress of traffic to the resource that uses the Security Group.
  • SecurityGroupIngress: provides rules that allow ingress of traffic to the resource that uses the Security Group.

A sample Security Group resource is below:

"SampleServerSecurityGroup" : {
  "Type" : "AWS::EC2::SecurityGroup",
  "Properties" : {
    "GroupDescription" : "Enable SSH",
    "VpcId" : { "Ref" : "VPC" },
    "SecurityGroupIngress" : [ {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "tcp",
      "FromPort" : "22",
      "ToPort" : "22",
      "CidrIp" : "0.0.0.0/0"
    } ],
    "SecurityGroupEgress" : [ { 
      "IpProtocol" : "tcp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, {
      "IpProtocol" : "udp",
      "FromPort" : "0",
      "ToPort" : "65535",
      "CidrIp" : "0.0.0.0/0"
    }, { 
      "IpProtocol" : "icmp",
      "FromPort" : "-1",
      "ToPort" : "-1",
      "CidrIp" : "0.0.0.0/0"
      } ]
    }
  }
}

One-click PuppetMaster

Project and Objective:

provide a client with the ability to build a PuppetMaster using only one click and able to be brought up in any Amazon region.

Needs:

The tool needed to be supportable, produce consistent results and work quickly. Customization and portability were not particularly important – they aren’t going to move off of Amazon anytime soon. Given the needs I utilized CloudFormation for the basis of the the tool. By leveraging the CloudFormation service anyone with familiarity with Amazon can easily build and modify the tooling.

Benefits:

Consistency, speed and availability.

Process:

If you peak under the hood of the tool you’d see the following happening in the order below:

  1. User instantiates CloudFormation with a region parameter.
  2. CloudFormation provisions resources such as an EC2 instance, a security group and an IAM Role (see “Resources Used”)
  3. EC2 resource is created, cloud-init runs user-data shell script.
  4. user-data shell script performs the following:
    1. lightweight machine configuration
    2. installs puppet from Puppetlabs
    3. gets puppetmaster configuration from GitHub
    4. installs the puppetmaster server through a puppet apply

Resources Used:

  1. CloudFormation, used for provisioning AWS resources:
    1. EC2 instance: used for running the PuppetMaster
    2. Security Group: used for allowing access to the PuppetMaster machine
    3. Route53 Resource Record: for locating the PuppterMaster
    4. IAM Role: used for getting configuration information required to access GitHub.
    5. cloud-init with user-data: used for lightweight configuration, connecting to GitHub and bootstrapping puppet.
  2. GitHub: used for storing the PuppetMaster configuration and Puppet client configuration.
  3. Puppet: once running puppet apply was used to build the initial PuppetMaster server
  4. Python and tool cloudinit inject

Result:

Running the following command created a fully-functioning PuppetMaster server in the AWS us-east-1 region.

cfn-create-stack puppetmaster --template-file puppetmaster_merged.json --parameters "puppetFQDN=puppet-prod-iad-001.test.com" --capabilities CAPABILITY_IAM