Error in requests with aiohttp in asyncio

Question

import time
import urllib.request
import asyncio
import aiohttp

URL = 'https://api.github.com/events'
MAX_CLIENTS = 10


def fetch_sync(pid):
    print('Fetch sync process {} started'.format(pid))
    start = time.time()
    response = urllib.request.urlopen(URL)
    datetime = response.getheader('Date')

    print('Process {}: {}, took: {:.2f} seconds'.format(
        pid, datetime, time.time() - start))

    return datetime


async def fetch_async(pid):
    print('Fetch async process {} started'.format(pid))
    start = time.time()
    response = await aiohttp.request('GET', URL)
    datetime = response.headers.get('Date')

    print('Process {}: {}, took: {:.2f} seconds'.format(
        pid, datetime, time.time() - start))

    response.close()
    return datetime


def synchronous():
    start = time.time()
    for i in range(1, MAX_CLIENTS + 1):
        fetch_sync(i)
    print("Process took: {:.2f} seconds".format(time.time() - start))


async def asynchronous():
    start = time.time()
    tasks = [asyncio.ensure_future(
        fetch_async(i)) for i in range(1, MAX_CLIENTS + 1)]
    await asyncio.wait(tasks)
    print("Process took: {:.2f} seconds".format(time.time() - start))


print('Synchronous:')
synchronous()

print('Asynchronous:')
ioloop = asyncio.get_event_loop()
ioloop.run_until_complete(asynchronous())
ioloop.close()

I’m studying the Python asyncIO module and came across this code on Github(https://github.com/yeraydiazdiaz/asyncio-ftwpd) but the same is generating this error:

Traceback (most recent call last):
  File "test_one.py", line 198, in fetch_async
    response = await aiohttp.request('GET', URL)
TypeError: object _SessionRequestContextManager can't be used in 'await' expression

I don’t know much about the module yet, someone could tell me why this error is occurring and how I can do it in the right way. Thank you in advance.

Answer 1

There’s a much simpler way to solve this. First, let’s define an asynchronous function that will be responsible for making the request, using the module aiohttp:

async def fetch(url):
  async with aiohttp.ClientSession() as session:
    async with session.get(url) as response:
      return await response.json()

Basically the function defines a session, performs a GET request for the specified URL, and returns the decoded response as JSON. Based on this function, we define the function that will deal with the answers obtained:

async def get(url, name):
  print(f'Iniciando a requisição {name}')
  response = await fetch(url)
  print(f'Quantidade de eventos da requisição {name}:', len(response))

The function basically performs the request, waits for the response and displays the number of records you got in it. The magic here happens in the await, because during the HTTP request, the Python interpreter does not need to be locked into this routine and can be directed to the others, making all requests in parallel. Thus, it is worth remembering that it is not guaranteed the order that the requisitions will be made.

To use the function get, we can do:

requests = [get(URL, i) for i in range(MAX_CLIENTS)]

And we must create the event loop to handle all requests:

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*requests))

Thus remaining the final code:

import aiohttp
import asyncio

URL = 'https://api.github.com/events'
MAX_CLIENTS = 10

async def fetch(url):
  async with aiohttp.ClientSession() as session:
    async with session.get(url) as response:
      return await response.json()

async def get(url, name):
  print(f'Iniciando a requisição {name}')
  response = await fetch(url)
  print(f'Quantidade de eventos da requisição {name}:', len(response))

requests = [get(URL, i) for i in range(MAX_CLIENTS)]

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(*requests))

^{See working on Repl.it}

A possible output of the program is:

Iniciando a requisição 1
Iniciando a requisição 7
Iniciando a requisição 2
Iniciando a requisição 8
Iniciando a requisição 3
Iniciando a requisição 9
Iniciando a requisição 4
Iniciando a requisição 5
Iniciando a requisição 0
Iniciando a requisição 6
Quantidade de eventos da requisição 8: 30
Quantidade de eventos da requisição 2: 30
Quantidade de eventos da requisição 0: 30
Quantidade de eventos da requisição 6: 30
Quantidade de eventos da requisição 5: 30
Quantidade de eventos da requisição 1: 30
Quantidade de eventos da requisição 9: 30
Quantidade de eventos da requisição 3: 30
Quantidade de eventos da requisição 4: 30
Quantidade de eventos da requisição 7: 30