Problems with Python performance testing

Question

A group of friends and I are starting an initiative to create a Reset on github in order to test the performance (benchmark) of http services implemented in various languages and frameworks. One of the languages we have already implemented is Python, without using any third-party framework.

The implementation was basically like this:

#!/usr/bin/python
from BaseHTTPServer import BaseHTTPRequestHandler,HTTPServer

PORT_NUMBER = 3000

class serverHandler(BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header('Content-type','text/html')
        self.end_headers()
        self.wfile.write('yo')
        return

try:
    #Create a web server and define the handler to manage the
    #incoming request
    server = HTTPServer(('', PORT_NUMBER), serverHandler)
    print 'Started httpserver on port ' , PORT_NUMBER

    #Wait forever for incoming htto requests
    server.serve_forever()

except KeyboardInterrupt:
    print '^C received, shutting down the web server'
    server.socket.close()

To perform the performance tests we use the boom (https://github.com/rakyll/boom) through the command:

boom -n 100000 -c 10000 http://localhost:3000

When we fire the boom against the service implemented in Python we have a disappointing performance, which at first should not occur.

Upshot:

Summary:
  Total:    25.5676 secs
  Slowest:  4.2101 secs
  Fastest:  0.9872 secs
  Average:  1.7560 secs
  Requests/sec: 380.0513

Status code distribution:
  [200] 9717 responses

Response time histogram:
  0.987 [1] |
  1.309 [1792]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.632 [2670]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.954 [1681]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.276 [2616]  |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.599 [446]   |∎∎∎∎∎∎
  2.921 [283]   |∎∎∎∎
  3.243 [171]   |∎∎
  3.565 [31]    |
  3.888 [1] |
  4.210 [25]    |

Latency distribution:
  10% in 1.0313 secs
  25% in 1.5239 secs
  50% in 1.7353 secs
  75% in 2.0630 secs
  90% in 2.2717 secs
  95% in 2.6155 secs
  99% in 3.1570 secs

I’d like to know what I’ve implemented wrong. The intention in this case is to be as simplistic as possible, without using allegories such as cache, or connection pool control (except if this is implicit in the language).

For those who want to check other results and implementations and/or want to contribute new implementations in languages and frameworks, feel free. The codes and results of all implementations are in: https://github.com/ericmdantas/bm

Answer 1

In the example code you can only process one request at a time (there is no parallelism). In other words, the Python interpreter will be idle while waiting for the data to be sent to the client.

In computational terms, this waiting time is precious and could be used to process other requests - which is why, when you analyze frameworks performance (such as Tornado, using a Event loop) you notice a big difference in the number of requests met.

Another factor is that as the boom runs on the actual machine on which the server is running, there is a competition for CPU resources between the implemented server and the benchmark software (in this case, the boom) - which distorts the results. Although the server runs on a different machine you would need to consider other issues such as bandwidth, CPU of the machine that is triggering the test and operating system settings.

Any benchmark reflects only a part of reality - keep this in mind when comparing results.