Creating VPCs and Subnets across Regions with a Single CloudFormation File

I’ve often encountered clients who want to utilize a single CloudFormation to build VPCs and Subnets across different AWS Regions and different AWS Accounts. In this blog post will describe exactly how to do this – as well as some of the pain points that are encountered when trying to utilize a single CloudFormation to build VPCs and subnets in different regions and accounts. The post is divided up into two parts – part one describes the solutions (and provides links to CloudFormation files which are stored in GitHub) and part two describes the solutions in more depth.

Part 1: a Single CloudFormation file for building VPC and Subnets in any Region or Account

The solution for building a any-region/any-account CloudFormation file containing a VPC and subnets is going to be different depending on if you need to provide a CloudFormation file that is multi-region or is both multi-region and multi-account. As a result of this, the blog post is divided into “Part 1-A” which covers multi-region only and “Part 1-B” which covers any-region/any-account.

Part 1-A: a Single CloudFormation file for building VPC and Subnets in any Region

If you don’t have a requirement that the you build VPCs and subnets across multiple accounts, you’ll have a relatively straightforward process:

First, you’ll create a mapping that maps each Region to Availability Zones in which subnets can be created. Be careful here: in my own personal AWS account I work can not create a subnet in “us-east-1a”. The end result looks something like below:

"AWSRegion2AZ" : {
  "us-east-1" : { "1" : "us-east-1b", "2" : "us-east-1c", "3" : "us-east-1d", "4" : "us-east-1e" },
  "us-west-1" : { "1" : "us-west-1b", "2" : "us-west-1c" },
  "us-west-2" : { "1" : "us-west-2a", "2" : "us-west-2b", "3" : "us-west-2c" }
}

Second, for each resource that requires a subnet, you’ll need to “Ref” the subnet. An example is below:

"PublicSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::FindInMap" : [ "AWSRegion2AZ", { "Ref" : "AWS::Region" }, "1" ] }
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},

A link to a CloudFormation template that will create a VPC and subnets in any AWS region: https://github.com/colinbjohnson/snippets/tree/master/aws/cloudformation/multi_region_vpc_cloudformation

Below I’ve used the CloudFormation file to create VPCs and subnets in 3 AWS regions: us-west-2, us-east-1 and us-west-1. Screenshots are below:

 

Part 1-B: a Single CloudFormation file for building VPC and Subnets in any Region or any Account

A CloudFormation file that builds a VPC and subnets in any Region or Account is going to be similar to the above (using a Map with defined Availability Zones) with one exception – each account will have different Availability Zones where subnets can be built. An example: my own account allows VPC subnets in the us-east-1b, us-east-1c, us-east-1d and us-east-1e Availability Zones (notice: no subnets can be built in us-east-1a) whereas a different account might allow subnets in us-east-1a, us-east-1b and us-east-1c Availability Zones. To account for this difference you’ll need a map that provides a VPC subnet to Availability Zone mapping for both Region and Account. The solution is shown below:

First, create a Map that accepts “Regions” and “Accounts” and returns a list of Availability Zones where VPC Subnets can be built.

"RegionAndAccount2AZ": {
  "us-east-1" : { 
    "Production" : [ "us-east-1b", "us-east-1c", "us-east-1d" ] ,
    "Development" : [ "us-east-1b", "us-east-1c", "us-east-1d" ]
  },
  "us-west-2" : { 
    "Production" : [ "us-west-2a", "us-west-2b", "us-west-2c" ] ,
    "Development" : [ "us-west-2a", "us-west-2b", "us-west-2c" ]
  }
},

Second, for each resource that needs to be build in a specific Availability Zone you’ll need select an item from the RegionAndAccount2AZ list:

"PublicSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "0", { "Fn::FindInMap" : [ "RegionAndAccount2AZ", { "Ref" : "AWS::Region"}, { "Ref" : "Account" } ] } ] },
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},
"PublicSubnet2" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "1", { "Fn::FindInMap" : [ "RegionAndAccount2AZ", { "Ref" : "AWS::Region"}, { "Ref" : "Account" } ] } ] },
    "CidrBlock" : "10.0.0.128/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},

And … here is a link to the CloudFormation template that will create a VPC and subnets in either different AWS regions and in different AWS accounts: https://github.com/colinbjohnson/snippets/tree/master/aws/cloudformation/multi_region_and_account_vpc_cloudformation

Part 2: Why is this all required?

Any time you have complexity it is important to keep focus on what actually needs to be done and why. I’ll describe the reasons why the additional complexity is required below:

  1. We need to ensure that when creating subnets using CloudFormation that the subnets are created in different Availability Zones. AWS doesn’t provide a facility for doing this. Result: we must define Availability Zones when creating subnet resources.
  2. AWS provides no mechanism for getting the Availability Zones in which subnets can be created. Result: we must manually provide a list of Availability Zones where subnets can be created. We do this using a map.
  3. If multiple accounts are used we run into a problem where the manually provided list of Availability Zones where subnets may be created are potentially different in each different account. Result: we need a map that allows CloudFormation to select Availability Zones where subnets can be built and that takes the “account” into account.

I’ve described the solutions to each problem above in more detail below.

Choosing Subnets Yourself

If you simply define subnets without specifying an “Availability Zone” property for each subnet there is a good chance that Amazon will create these subnets in the same Availability Zone. An example of defining subnets without an Availability Zone property is below:

"PublicSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},
"PublicSubnet2" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "CidrBlock" : "10.0.0.128/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},

There is a pretty good chance that this will cause two problems:

  1. If you are creating resources that use these subnets – such as an ELB – the ELB resource creation will fail due to the fact that an ELB can only have /one/ subnet per AZ. In the case above, if PublicSubnet1 and PublicSubnet2 are both in us-east-1b – ELB creation will fail.
  2. You may end up with an availability problem as a result of resources being created in the same Availability Zone. For example, if PublicSubnet1 and PublicSubnet2 are both in us-east-1b and you create an Auto Scaling Group that utilizes both PublicSubnet1 and PublicSubnet2 – your instances will still all be brought up in us-east-1b.

The solution would be to use “Fn::GetAZs” but…

“Fn::GetAZs” Returns AZs Where Subnets Can’t Be Placed

To solve the problem of placing subnets in the same Availability Zone, you’d think that you want to use Amazon’s “Fn::GetAZs”. For example, you’d call “{ “Fn::GetAZs” : { “Ref” : “AWS::Region” }” (this returns a list of Availability Zones) and then you’d build PublicSubnet1 in the first Availability Zone, PublicSubnet2 in the second Availability Zone and so on. An example is below:

"PublicSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "0", { "Fn::GetAZs" : { "Ref" : "AWS::Region" } } ] },
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},
"PublicSubnet2" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "1", { "Fn::GetAZs" : { "Ref" : "AWS::Region" } } ] },
    "CidrBlock" : "10.0.0.128/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},

However, if you use Amazon’s “Fn::GetAZs” – you’ll get a list of all Availability Zones – not just those Availability Zones in which a subnet can be created. As an example, if I call “Fn::GetAZs” using my own account in the us-east-1 region, the return values are [ “us-east-1a”, “us-east-1b”, “us-east-1c”, “us-east-1d”, “us-east-1e” ]. A problem arises because the “us-east-1a” Availability Zone isn’t available to me for subnet creation, so CloudFormation stack creation fails. Here’s a screenshot of that behavior:

FnGetAZs Returns AZs Where Subnets Cant Be Built.png

“Mapping Method” to the Rescue

Using a Map solves this mess. The solution isn’t ideal as it requires one time creation of a map containing a list of Availability Zones where subnets can be created. This map does allow you:

  1. ensure subnets are built in different AZs.
  2. provide support for multiple regions.

Availability Zones that Support VPC Subnets are Different Per Account

If you require VPCs built in different accounts you’ll be required to take one additional step – specifically, you’ll need to provide an Availability Zone to Subnet map per account because each account may have different Availability Zone properties. An example of this mapping is below:

"Mappings" : {
  "RegionAndAccount2AZ": {
    "us-east-1" : { 
      "Production" : [ "us-east-1a", "us-east-1b", "us-east-1c" ] ,
      "Development" : [ "us-east-1b", "us-east-1c", "us-east-1d" ]
    },
    "us-west-2" : { 
      "Production" : [ "us-west-2a", "us-west-2b", "us-west-2c" ] ,
      "Development" : [ "us-west-2a", "us-west-2b", "us-west-2c" ]
    }
  }
},

And an example of using this mapping to place a subnet in the correct Availability Zone:

"PublicSubnet1" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "0", { "Fn::FindInMap" : [ "RegionAndAccount2AZ", { "Ref" : "AWS::Region"}, { "Ref" : "Account" } ] } ] },
    "CidrBlock" : "10.0.0.0/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},
"PublicSubnet2" : {
  "Type" : "AWS::EC2::Subnet",
  "Properties" : {
    "AvailabilityZone" : { "Fn::Select" : [ "1", { "Fn::FindInMap" : [ "RegionAndAccount2AZ", { "Ref" : "AWS::Region"}, { "Ref" : "Account" } ] } ] },
    "CidrBlock" : "10.0.0.128/25",
    "VpcId" : { "Ref" : "VPC" }
  }
},

And… in Conclusion:

  1. The situation of building VPCs and subnets across regions and accounts using CloudFormation will likely improve. Examples of potential improvements might include a “Fn::GetAZs” pseudo parameter that returns only Availability Zones where subnets can be built or for loops that can build 1 to “x” subnets.
  2. The techniques described in this blog post can likely be improved by using conditionals or lambda. If anyone does this – let me know and I’ll update the post.
  3. Other tools that support “shelling out” or running arbitrary commands may provide better mechanisms that allow a single file to create VPCs and Subnets – although using a tool outside of CloudFormation may not be an option you are open to considering.

Hope that you have found this post useful – if you have questions or comments please feel free to send me an email: colin@cloudavail.com.

One thought on “Creating VPCs and Subnets across Regions with a Single CloudFormation File

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s