Python asynchronous generators

Question

With the introduction of the library asyncio, a new syntax was introduced to define coroutines in Python >= 3.5 and with it it it is possible to define asynchronous methods, asynchronous iterators and even asynchronous generators.

How can I implement asynchronous generators and what would be their common use cases?

Answer 1

The best source of information about asynchronous generators is the PEP that describes their need, and their implementation - the PEP 525.

To briefly describe asynchronous generators, let us briefly review the use of asynchronous functions (but not their internal mechanism). Asynchronous code in Python and other languages is characterized by code capable of running in a single thread, but in which the programmer can place dots explicit where the code of a function (generator, method, etc...) may be paused pending a result. During this pause, a "controller" of the execution of the asynchronous code passes the execution to other asynchronous functions paused in the same way, checks whether results have arrived that are being expected, etc... .

In Python this pause is characterized by the keywords await and async - whenever one of them appears in the body of a function , the code ` must being executed in an asynchronous context.

It turns out that in Python 3.5 the command had already been defined async for - namely, a for where at each innervation the program control is passed to the event loop, until the next iterator element is available. The iterator to be used in a async for Python 3.5 would have to define the special methods __aiter__ and __anext__ as co-routines (asynchronous functions, declared with async def)- in contrast to the __iter__ and __next__ of normal iterators.

Only for a for normal, non-asynchronous, any function containing the expression yield in your body is converted autmatically into a generator: that is, an object that has the methods __iter__ and __next__ and can be used in a for. A async for, before Python 3.6 and PEP 525, could only iterate over a class defined with the asynchronous versions of these methods, and try to use, in the body of an asynchronous function, the expression yield, resulted in a syntax error:

Python 3.5.5 (default, Jun  8 2018, 09:55:12) 
...
>>> async def a():
...    yield 1
... 
  File "<stdin>", line 2
SyntaxError: 'yield' inside async function

Only in Python 3.6 is it possible to declare directly an async Generator:

Python 3.6.5 (default, Jun  8 2018, 09:56:48) 
...
>>> async def a():
...    yield 1
... 
>>> 

And it is usable, in another asynchronous function, always managed by the event loop, by the command async for:

import asyncio

loop = asyncio.get_event_loop()

import random
async def random_pause_generator(start, stop):
    for i in range(start, stop):
        await asyncio.sleep(random.random())
        yield i

async def counter(start, stop):
    async for j in random_pause_generator(start, stop):
        print(j)


async def main():
    await asyncio.gather(counter(0, 10), counter(110, 120))

loop.run_until_complete(main())

Run this code and note how the two "counter" routine calls run in parallel, with results of each call appearing out of order. It is interesting to note the call to asyncio.gather that unifies the waiting of the two calls in a "single wait". If it is used simply await counter(10, 20), the end result is the same as in a synchronous function: the program would only continue on the bottom line after the call to counter completely resolve.

Extra note on async: the controller code in Python is referred to simply by the "event loop", and is a special object that in case of use of the asyncio library can always be recovered by the call asyncio.get_event_loop . (Note that it is possible have other implementations of libraries to coordinate the execution of asynchronous code, and they may have distinct calls to retrieve the event loop - but since asyncio is flexible enough to allow the use of other classes as event loop, It’s just reinventing the wheel)