Asynchronous http client, or why multithreading is superfluous

Some time ago , a note about a client-parser of sites on Python jumped to Habré. The author on this example analyzed the problems of multithreaded network applications.

But it seemed to me that the same task (or rather, its main part - parallel connections with an http-server) can be effectively solved without flows. Having looked



, for a start, in my article about Twisted and Tornado, scratching my head and digging into the documentation for non-blocking sockets, I sketched an asynchronous http-client server .

Below is the source code of the key part of the application with explanations:

import socket
import select
import sys
import errno
import time

from config import *

def ioloop ( ip_source , request_source ) :
    "" "Asynchronous self-loop

    ip_source - infinite iterable, issuing IP addresses for connections;
    request_source - iterable, generating request bodies;
    »" "
    Starttime = time . time ( )

    # open the socket pool; dictionaries that store connections of the body of requests and responses
    epoll = select . epoll ( )
    connections ={ } ; responses = { } ; requests = { }
    bytessent = 0.0
    bytesread = 0.0
    timeout = 0.3

    # select the first request
    request = request_source . next ( )
    try :
        while True :
            # check the number of connections, if they are less than the minimum
            # possible and there are requests left - add one more.
            #
            connection_num = len (connections )
                
            if connection_num < CLIENT_NUM and request :
                ip = ip_source . next ( )
                print “Opening a connection to% s.” % ip
                clientsocket = socket . socket ( socket . AF_INET ,
                                             socket . SOCK_STREAM )
                # Somewhat nontrivial. A non-blocking socket throws an
                EINPROGRESS # exception exception if it cannot connect right away.
                # Ignore the error and start waiting for the event on the socket.
                #
                clientsocket . setblocking ( 0 )
                try :
                    res = clientsocket . connect ( ( ip , 80 ) )
                except socket . error , err :
                    #
                    if err . errno ! = errno . EINPROGRESS :
                        raise
                # Add the socket to the pool and
                epoll connection dictionary. register ( clientsocket . fileno ( ) , select . EPOLLOUT )
                connections [ clientsocket . fileno ( ) ] = clientsocket
                requests [ clientsocket . fileno ( ) ] = request
                responses [ clientsocket . fileno ( ) ] = ""
                
            # "Pooling" - that is, collection of events
            #
            events = epoll . poll ( timeout )
            for fileno , event in events :
                if event & select . EPOLLOUT :
                    # Sending part of the request ...
                    #
                    try :
                        byteswritten = connections [ fileno ] . send ( requests [ fileno ] )
                        requests [ fileno ] = requests[ fileno ] [ byteswritten : ]
                        print byteswritten , “bytes sent.”
                        bytessent + = byteswritten
                        if len ( requests [ fileno ] ) = = 0 :
                            epoll . modify ( fileno , select . EPOLLIN )
                            print “switched to reading.”
                    except socket . error , err :
                        print “Socket write error:„ , err
                    except Exception , err :
                        print “Unknown socket error:„ , err
                elif event & select . EPOLLIN :
                    # Read part of the answer ... “
                    #
                    try :
                        bytes = connections [ fileno ] . recv ( 1024 )
                    except socket . error , err :
                        # We catch the error "connection reset by peer" -
                        # happens with a large number of connections
                        #
                        if err . errno = = errno . ECONNRESET :
                            epoll . unregister ( fileno )
                            connections [ fileno ] . close ( )
                            del connections [ fileno ]
                            print »Connection reset by peer."
                            continue
                        else :
                            raise err

                    print len ( bytes ) ,“
                    Bytes read.” Bytesread + = len ( bytes )
                    responses [ fileno ] + = bytes
                    if not bytes :
                        epoll . unregister ( fileno )
                        connections [ fileno ] . close ( ) ;
                        del connections [ fileno ]
                        print "Done reading ... Closed."

    # select the following query
            ifrequest :
                request = request_source . next ( )

            print “Connections left:„ , len ( connections )
            if not len ( connections ) :
                break
    except KeyboardInterrupt :
        print “Looping interrupted by a signal.”
        for fd , sock in connections . items ( ) :
            sock . close ( )
    epoll.close()

    endtime = time.time()
    timespent = endtime - starttime
    return responses, timespent, bytesread, bytessent

Мораль тут простая — не всюду следует пихать потоки, более того, существуют ситуации, когда многопоточность только снизит надежность программы, создаст известные проблемы в тестировании и станет источником неуловимых багов. Если некритична производительность, но очень хочется что-то распараллелить, то часто вполне оправдывают себя даже обычные процессы и примитивный IPC.

Кроме того, в Питоне все равно не существует настоящих потоков уровня ядра, а здравствует и по сей день треклятый GIL. Соответственно, никаких преимуществ в производительности на многоядерных процессорах получить нельзя.

Данный скрипт, конечно, жутковато и на скорую руку исполнен, не обрабатывает обрывы соединения сервером и ошибки на операциях чтения/записи в сокет, не разбирает ответы сервера, но зато тащит многократно корень сайта cnn.com на пределе возможностей моего канала — 800-1000 Кб/с. :)

Целиком исходники скрипта можно найти где-то тут

PS Может, у кого есть мысли, для чего можно использовать производительные
асинхронные клиенты? :)