AWS ELB pre-open Connection Exposé, Part 1

Exposé of AWS’s ELB “pre-open” Optimization, part 1

or “what are all these ..reading.. connections in Apache when I use an AWS ELB?”

A colleague of mine reported a problem where their Apache servers were reporting a number of connections in “..reading..” status. They suspected that Amazon’s Elastic Load Balancer was causing the additional connections.

Screenshot server-status page of an EC2 instance behind an AWS ELB and the resulting ..reading.. requests.
Screenshot server-status page of an EC2 instance behind an AWS ELB and the resulting ..reading.. requests.

Determining the source of the ..reading.. Requests

I devised a simple method to identify the source of the ..reading.. requests: comparing the results shown by Apache’s “Server Status” page when viewed on an EC2 instance running Apache and the results shown by Apache’s “Server Status” page on an EC2 instance that was located behind an Amazon ELB.

Configuration was as follows:

  • AMI: ami-a73264ce
  • OS: Ubuntu 12.04.3 LTS
  • Apache Version: Apache/2.2.22, prefork
  • Apache Configuration: KeepAlive Off

To test, I visited http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status. The server status page showed no ..reading.. connections. Next, I added an ELB in front of this instance. Visiting http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status returned one ..reading.. request. When adding additional listening ports to the ELB (for instance, adding port 443 as an Apache listening port) I could determine that whenever a new listener was added an additional ..reading.. request was opened.

Predicting ..reading.. behavior

I was curious about the number of ..reading.. requests – at this point, I’d only seen one ..reading.. request per listener. This behavior did not match my colleagues assertion that he was seeing a number of ..reading.. requests. I suspected that the number of ..reading.. requests was related to the amount of traffic directed through an ELB so I created a test where I would send a number of sequential requests through the ELB to an EC2 instance – the results are below:

Example test:

for i in {1..5}
do
curl test-738445577.us-east-1.elb.amazonaws.com
done
# count reading requests
curl 127.0.0.1/server-status
# result
<dt>12 requests currently being processed, 0 idle workers</dt>
</dl><pre>RRRRRRRRWRRR....................................................

The “R” requests above demonstrate an active ..reading.. request. To reiterate the above: I placed 5 sequential requests through an ELB and, in return, 11 “Reading” requests were opened on the client. Further tests demonstrated that for anywhere between 1 to 10 sequential requests results in an additional 5-7 ..reading.. requests. Typical results from testing are below:

  • 0 http requests = 1 reading request
  • 1 http request = 7 reading requests
  • 2 http requests = 7 reading requests
  • 5 http requests = 11 reading requests
  • 8 http requests = 14 reading requests
  • 10 http requests = 16 reading requests
  • 15 http requests = 17 reading requests

Conclusion, part 1:

  1. Confirmation that Amazon’s ELB creates opened connections to an EC2 instance which are reported as ..reading.. requests by Apache.
  2. Confirmation that the number of additional opened connections to an EC2 instance is greater than one opened connection.

Next Steps?

Part 2 of the AWS ELB “pre-open” optimization blog post will detail the investigation into the purpose behind the Amazon ELB opening a number of ..reading.. requests, some discussion with Amazon regarding the undocumented behavior and, lastly, some discussion of the potential problems this behavior could cause.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s