ELB Behavior when Utilizing Backend Instances with Self-Signed Certs

There are AWS questions that I don’t know the answer to – and sometimes these questions need answers. In a case about a week ago, the following question was posed:

How does an Amazon Web Services ELB function when the backend instances utilize a self-signed certificate?

I was lucky enough to have the time to investigate. If you are just looking for the answer, see “short answer” below. For instructions (and a CloudFormation file) to allow you to duplicate my work, see”long answer” further below.

Short answer:

Yes. The AWS ELB will work with backend instances that utilize a self-signed certificate.

Long answer:

if you’d like to duplicate the test I utilized a CloudFormation file that builds this very infrastructure (an ELB, an Auto Scaling Group and Ubuntu instances running Apache accepting HTTPS connections on port 443) you can get this file from my “Snippets” repository at https://github.com/colinbjohnson/snippets/tree/master/aws/elb/elb_backend_selfsigned_cert A diagram describing the configuration is below:

ELB to BE HTTPS.png

 

After performing tests to ensure that the backend instances where healthy and serving requests I wanted to dig a bit deeper to confirm that the data was, in fact, encrypted. I went ahead and ran some requests against the backend web servers and utilized tcpdump on a backend instance, capturing data on port 443. Please note that the Security Group utilized in testing only allows port 443 inbound, so I could have run this test without filtering on “port”. A screen capture of the data captured by tcpdump is shown below:

Backend Instance - tcpdump Capture.png

I ran another tcpdump capture and loaded the data into Wireshark for visualization – the result is show below:

Backend Instance - Encrypted Data.png Notice that the capture has no filters applied – and specifically – that payload is shown as “Encrypted Application Data.” If this were an HTTP connection I would be able to view the data that was being sent. After the packet capture my curiosity was satisfied – the body of HTTP requests was being encrypted in transit.

Conclusion:

If you require “end to end” encryption of data in transit you can utilize backend instances with self-signed certificates.

 

Inner Workings of the AWS ELB, Part 1

Inner Workings of the AWS ELB, Part 1

Ever wonder what an AWS ELB is made of? This blog post will:

  1. Provide more transparency into AWS ELB for those who rely on AWS technology.
  2. Let ELB users know what they can expect from an Amazon ELB.
  3. Let ELB users know what they can ask Amazon for.

What is an ELB?

Simply stated: an ELB is made up of one (or more) EC2 instances per availability zone, with these instances routing traffic to back-end EC2 instances. While AWS does not publicly document ELB architecture, the following should be convincing:

Behavior of ELB when Availability Zones are Added and Removed:
  1. Create an ELB with only one availability zone (example: us-east-1a)
  2. Run “host” with the ELB’s hostname – notice that only one IP address is returned.
    Image
  3. Add a second availability zone to your ELB.
  4. Run host with the ELB’s hostname – notice that two IP addresses are returned.
    Image
  5. Add an additional availability zone – you’ll see a third IP address when performing a DNS query of the ELB’s hostname.
Return of DNS Lookup of ELB IP addresses:

Run “dig -x” with the IP addresses of the given instances.

Example: dig -x 50.19.115.50 +short

The return of these requests are the same values as EC2 instance public DNS names.

Example return value: ec2-50-19-115-50.compute-1.amazonaws.com.

Image

Amazon’s own Documentation:

Amazon’s own documentation hints at the construction of the ELB – the ELB Concepts page notes “When you register your EC2 instances, Elastic Load Balancing provisions load balancer nodes in all the Availability Zones that has the registered instances.”

How does this help me?

If Amazon Web Service’s ELB’s are made up of EC2 instances in an Auto Scaling Group then you have a good idea of what you can ask the AWS ELB support group for. Examples below:

You can “pre-warm” an ELB to prepare for a flood of traffic:

Let’s say you had a client that was a tech-centered advertising network and you knew that an upcoming Apple announcement was going to generate a large amount of traffic on your servers – knowing this, you could do one of the following:

  1. Call Amazon and request that the ELB be scaled up before this occurred. Note that Amazon documents your ability to request “pre-warming” in their Best Practices in Evaluating Elastic Load Balancing document.
  2. Pre-warm the ELB by gradually scaling up the ELB by sending your own automatically generated traffic.
You could change the EC2 Instance Types that make up an ELB:

Let’s say you knew that your traffic was naturally bursty – enough to overwhelm a small EC2 instance type. You could Amazon support and ask them to change the instance type that comprises your ELB to an instance that offered greater network performance.

You can change the Auto Scaling Policy supporting the ELB:

Again, assuming a situation where you have bursty traffic or you wish to have multiple ELB EC2 instances in each availability zone – you could call Amazon and suggest that you wish for the ELB to bring additional EC2 nodes into service immediately when traffic starts or to always maintain two nodes per AZ. Regarding the “scale up” time period, Amazon’s documentation states that “the time required for Elastic Load Balancing to scale can range from 1 to 7 minutes, depending on the changes in the traffic profile” – particular customers might wish for this time period to be more predictable, or for their ELB to have over-provisioned capacity from the start.

Conclusion:

If you use or are implementing Amazon’s ELB, I’d suggest you:

  1. Implement the ELB and back-end instances in the most vanilla manner that meets your needs. The ELB doesn’t provide many tunables and remaining “vanilla” ensures that tunables aren’t needed.
  2. Engage your Amazon Account Manager or support group. In particular the Account Managers I have worked with have been valuable in ensuring predictable ELB implementation.
  3. If you need ELB modifications, ask. The ELB has a very limited API, but a number of parameters are able to be tuned by placed a call to Amazon support.

 

AWS ELB pre-open Connection Exposé, Part 2

Exposé of AWS’s ELB “pre-open” Optimization, part 2

or “what are all these ..reading.. requests” for?

Summary:

To get straight to the point: the pre-opened connections are an Amazon ELB provided optimization to eliminate the tcp handshake time between the ELB and EC2 instance when a client request is placed – I’ve confirmed this behavior with a member of the Amazon ELB engineering team. Note: if you do not wish to use the pre-open optimization AWS support can disable the optimization for you. If the details interest you the remainder of the post will describe how I went about discovering the purpose of the Amazon pre-open optimization prior to my discussion with AWS.

Investigation:

From the start I suspected that the “pre-open” behavior was a result of the AWS ELB “warming” connections to the EC2 backend. By identifying the source of a request by the source IP of the ELB, I came to understand that the tcp connection “pre-open” flow resulted in the following:

  1. A tcp connection being established: SYN (LB) -> SYN,ACK (EC2) -> SYN,ACK (LB)
  2. The EC2 instance transmitting a second ACK 1 second later.
  3. The EC2 instance closing the established tcp stream connecting 20 seconds later (note: the time waiting for the stream to be closed is set by the Apache configuration).

The image below is a screenshot of a captured “…reading…” connection.

ELB Reading Connection - TCP Stream

I wanted to confirm that this behavior was, in fact, a “pre-open” optimization and to confirm my suspicion that these connections would be used by the ELB if requests were placed during the window in which the connection was open. To test this, planned to prime the ELB with a number of requests, wait a number of seconds (for this example, 5 seconds) and then generate additional requests of a particular unique URL. When performing this test results showed that the pre-open connection was created by an ELB and then, after 5 seconds, an HTTP request would be placed over an already established connection. By changing the “wait” number in my tests, I was able to effectively control the period after a connection was opened that additional data would be sent. For example, if I primed the ELB with traffic and waited 5 seconds before sending a future http request, I’d see a tcp connection that waited 5 seconds before delivering an http request. If I primed an ELB with traffic and waited 18 seconds before sending a future http request, I’d see a tcp connection that waited 18 seconds before delivering an http request.

Final Thoughts:
  1. No documentation exists for the “pre-open” optimization. This isn’t good.
  2. In most cases the ELB “pre-open” optimization will have a positive impact on performance but and will not have a negative impact on service availability.
  3. Amazon support can be persuaded to turn off the “pre-open” optimization for ELBs that do not use sticky-sessions.

AWS ELB pre-open Connection Exposé, Part 1

Exposé of AWS’s ELB “pre-open” Optimization, part 1

or “what are all these ..reading.. connections in Apache when I use an AWS ELB?”

A colleague of mine reported a problem where their Apache servers were reporting a number of connections in “..reading..” status. They suspected that Amazon’s Elastic Load Balancer was causing the additional connections.

Screenshot server-status page of an EC2 instance behind an AWS ELB and the resulting ..reading.. requests.
Screenshot server-status page of an EC2 instance behind an AWS ELB and the resulting ..reading.. requests.

Determining the source of the ..reading.. Requests

I devised a simple method to identify the source of the ..reading.. requests: comparing the results shown by Apache’s “Server Status” page when viewed on an EC2 instance running Apache and the results shown by Apache’s “Server Status” page on an EC2 instance that was located behind an Amazon ELB.

Configuration was as follows:

  • AMI: ami-a73264ce
  • OS: Ubuntu 12.04.3 LTS
  • Apache Version: Apache/2.2.22, prefork
  • Apache Configuration: KeepAlive Off

To test, I visited http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status. The server status page showed no ..reading.. connections. Next, I added an ELB in front of this instance. Visiting http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status returned one ..reading.. request. When adding additional listening ports to the ELB (for instance, adding port 443 as an Apache listening port) I could determine that whenever a new listener was added an additional ..reading.. request was opened.

Predicting ..reading.. behavior

I was curious about the number of ..reading.. requests – at this point, I’d only seen one ..reading.. request per listener. This behavior did not match my colleagues assertion that he was seeing a number of ..reading.. requests. I suspected that the number of ..reading.. requests was related to the amount of traffic directed through an ELB so I created a test where I would send a number of sequential requests through the ELB to an EC2 instance – the results are below:

Example test:

for i in {1..5}
do
curl test-738445577.us-east-1.elb.amazonaws.com
done
# count reading requests
curl 127.0.0.1/server-status
# result
<dt>12 requests currently being processed, 0 idle workers</dt>
</dl><pre>RRRRRRRRWRRR....................................................

The “R” requests above demonstrate an active ..reading.. request. To reiterate the above: I placed 5 sequential requests through an ELB and, in return, 11 “Reading” requests were opened on the client. Further tests demonstrated that for anywhere between 1 to 10 sequential requests results in an additional 5-7 ..reading.. requests. Typical results from testing are below:

  • 0 http requests = 1 reading request
  • 1 http request = 7 reading requests
  • 2 http requests = 7 reading requests
  • 5 http requests = 11 reading requests
  • 8 http requests = 14 reading requests
  • 10 http requests = 16 reading requests
  • 15 http requests = 17 reading requests

Conclusion, part 1:

  1. Confirmation that Amazon’s ELB creates opened connections to an EC2 instance which are reported as ..reading.. requests by Apache.
  2. Confirmation that the number of additional opened connections to an EC2 instance is greater than one opened connection.

Next Steps?

Part 2 of the AWS ELB “pre-open” optimization blog post will detail the investigation into the purpose behind the Amazon ELB opening a number of ..reading.. requests, some discussion with Amazon regarding the undocumented behavior and, lastly, some discussion of the potential problems this behavior could cause.