Exposé of AWS’s ELB “pre-open” Optimization, part 1
or “what are all these ..reading.. connections in Apache when I use an AWS ELB?”
A colleague of mine reported a problem where their Apache servers were reporting a number of connections in “..reading..” status. They suspected that Amazon’s Elastic Load Balancer was causing the additional connections.
Determining the source of the ..reading.. Requests
I devised a simple method to identify the source of the ..reading.. requests: comparing the results shown by Apache’s “Server Status” page when viewed on an EC2 instance running Apache and the results shown by Apache’s “Server Status” page on an EC2 instance that was located behind an Amazon ELB.
Configuration was as follows:
- AMI: ami-a73264ce
- OS: Ubuntu 12.04.3 LTS
- Apache Version: Apache/2.2.22, prefork
- Apache Configuration: KeepAlive Off
To test, I visited http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status. The server status page showed no ..reading.. connections. Next, I added an ELB in front of this instance. Visiting http://ec2-54-204-235-46.compute-1.amazonaws.com/server-status returned one ..reading.. request. When adding additional listening ports to the ELB (for instance, adding port 443 as an Apache listening port) I could determine that whenever a new listener was added an additional ..reading.. request was opened.
Predicting ..reading.. behavior
I was curious about the number of ..reading.. requests – at this point, I’d only seen one ..reading.. request per listener. This behavior did not match my colleagues assertion that he was seeing a number of ..reading.. requests. I suspected that the number of ..reading.. requests was related to the amount of traffic directed through an ELB so I created a test where I would send a number of sequential requests through the ELB to an EC2 instance – the results are below:
Example test:
for i in {1..5}
do
curl test-738445577.us-east-1.elb.amazonaws.com
done
# count reading requests
curl 127.0.0.1/server-status
# result
<dt>12 requests currently being processed, 0 idle workers</dt>
</dl><pre>RRRRRRRRWRRR....................................................
The “R” requests above demonstrate an active ..reading.. request. To reiterate the above: I placed 5 sequential requests through an ELB and, in return, 11 “Reading” requests were opened on the client. Further tests demonstrated that for anywhere between 1 to 10 sequential requests results in an additional 5-7 ..reading.. requests. Typical results from testing are below:
- 0 http requests = 1 reading request
- 1 http request = 7 reading requests
- 2 http requests = 7 reading requests
- 5 http requests = 11 reading requests
- 8 http requests = 14 reading requests
- 10 http requests = 16 reading requests
- 15 http requests = 17 reading requests
Conclusion, part 1:
- Confirmation that Amazon’s ELB creates opened connections to an EC2 instance which are reported as ..reading.. requests by Apache.
- Confirmation that the number of additional opened connections to an EC2 instance is greater than one opened connection.
Next Steps?
Part 2 of the AWS ELB “pre-open” optimization blog post will detail the investigation into the purpose behind the Amazon ELB opening a number of ..reading.. requests, some discussion with Amazon regarding the undocumented behavior and, lastly, some discussion of the potential problems this behavior could cause.
