AWS ELB pre-open Connection Exposé, Part 2

Exposé of AWS’s ELB “pre-open” Optimization, part 2

or “what are all these ..reading.. requests” for?

Summary:

To get straight to the point: the pre-opened connections are an Amazon ELB provided optimization to eliminate the tcp handshake time between the ELB and EC2 instance when a client request is placed – I’ve confirmed this behavior with a member of the Amazon ELB engineering team. Note: if you do not wish to use the pre-open optimization AWS support can disable the optimization for you. If the details interest you the remainder of the post will describe how I went about discovering the purpose of the Amazon pre-open optimization prior to my discussion with AWS.

Investigation:

From the start I suspected that the “pre-open” behavior was a result of the AWS ELB “warming” connections to the EC2 backend. By identifying the source of a request by the source IP of the ELB, I came to understand that the tcp connection “pre-open” flow resulted in the following:

  1. A tcp connection being established: SYN (LB) -> SYN,ACK (EC2) -> SYN,ACK (LB)
  2. The EC2 instance transmitting a second ACK 1 second later.
  3. The EC2 instance closing the established tcp stream connecting 20 seconds later (note: the time waiting for the stream to be closed is set by the Apache configuration).

The image below is a screenshot of a captured “…reading…” connection.

ELB Reading Connection - TCP Stream

I wanted to confirm that this behavior was, in fact, a “pre-open” optimization and to confirm my suspicion that these connections would be used by the ELB if requests were placed during the window in which the connection was open. To test this, planned to prime the ELB with a number of requests, wait a number of seconds (for this example, 5 seconds) and then generate additional requests of a particular unique URL. When performing this test results showed that the pre-open connection was created by an ELB and then, after 5 seconds, an HTTP request would be placed over an already established connection. By changing the “wait” number in my tests, I was able to effectively control the period after a connection was opened that additional data would be sent. For example, if I primed the ELB with traffic and waited 5 seconds before sending a future http request, I’d see a tcp connection that waited 5 seconds before delivering an http request. If I primed an ELB with traffic and waited 18 seconds before sending a future http request, I’d see a tcp connection that waited 18 seconds before delivering an http request.

Final Thoughts:
  1. No documentation exists for the “pre-open” optimization. This isn’t good.
  2. In most cases the ELB “pre-open” optimization will have a positive impact on performance but and will not have a negative impact on service availability.
  3. Amazon support can be persuaded to turn off the “pre-open” optimization for ELBs that do not use sticky-sessions.