Create a lot of asynchronous requests using Grequests

Requests are good, but grequests are better. I don’t know a better, more efficient library that can quickly and elegantly execute HTTP requests than requests, this library is the undoubted leader in this regard.

But since it is limping with asynchrony, it is possible to execute asynchronous requests using threading or gevent.

Wrote grequests, the same author as wrote requests. Only using gevent + requests. I will not procrastinate the topic for a long time, I will give you detailed information about this library.

Grequests - is an asynchronous wrapper over regular requests.

Let's make a regular POST request for a lot of urls:

import grequests 
with open("C:\\path\\urls.txt") as werewolves: 
    array = [row.strip() for row in werewolves]
params = {'a':'b', 'c':'d'}
rs = [grequests.post(u, data=params) for u in array]
for r in grequests.imap(rs, size=16) 
    print(r[0].status_code, r[0].url)

Everything is quite simple, the library is imported, the file is opened for reading, a list is created, the params variable is assigned the values ​​a: b, c: d.

Next, we create the rs variable which will be responsible for the POST request itself, for the r variable we create grequests.map ([rs], size = is an asynchronous value, the larger the value, the faster the http requests will be executed, though it makes no sense to set more than 16) .

Now, since we passed all the arguments to the r variable, that is, in grequests.imap (), we can interact with this variable as in regular requests.

And the last step we need to display all the status code, url address, rs also acts as a list, we do this so that there are no indexing errors by type:

TypeError: 'Response' object does not support indexing

If everything works out exactly with traceback with this error, I suggest the option:

def exception_handlerr(request, exception):
    print("Request failed", request.url) 
import grequests 
with open("C:\\path\\urls.txt") as werewolves: 
    array = [row.strip() for row in werewolves]
params = {'a':'b', 'c':'d'}
rs = [grequests.post(u, data=params) for u in array]
for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) 
    print(r[0].status_code, r[0].url)

Now we will refer to the variable r as a list, in order to avoid indexing errors.
The main steps we have taken. You can “TANK” the server. Although, this library does not possess transcendental asynchrony. You can take a look at Aiohttp .

I would also like to talk about exceptions in grequests. Since grequests does not use the error classes from requests, it does as follows:

def send(self, **kwargs):
    """
    Prepares request based on parameter passed to constructor and optional      ``kwargs```.
    Then sends request and saves response to :attr:`response`
    :returns: ``Response``
    """
    merged_kwargs = {}
    merged_kwargs.update(self.kwargs)
    merged_kwargs.update(kwargs)
    try:
        self.response = self.session.request(self.method,
                                            self.url, **merged_kwargs)
    except Exception as e:
        self.exception = e
        self.traceback = traceback.format_exc()
    return self

We catch using exception_handler:

def exception_handlerr(request, exception):
    print("Request failed", request.url) 
    # print(str(exception)) 

Full source code:

def exception_handlerr(request, exception):
    print("Request failed", request.url) 
import grequests 
with open("C:\\path\\urls.txt") as werewolves: 
    array = [row.strip() for row in werewolves]
params = {'a':'b', 'c':'d'}
rs = [grequests.post(u, data=params) for u in array]
for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) 
    print(r.status_code, r.url)

So we can with full confidence catch errors.

With GET requests, everything is just as simple as with POST requests:

def exception_handlerr(request, exception):
    print("Request failed", request.url) 
import grequests 
with open("C:\\path\\urls.txt") as werewolves: 
    array = [row.strip() for row in werewolves]
rs = [grequests.get(u) for u in array]
for r in grequests.map([rs], size=16, exception_handler=exception_handlerr) 
    print(r.status_code, r.url)

→ Source code grequests
→ Documentation on requests Have a nice

day!