Task

It is necessary to start several sessions and that each of them would fulfill a certain number of requests. I want to note that it is necessary to have several sessions, and not one, as presented in almost all examples of aiohttp. Below is the code in which:

  1. through loop kurutin runs
  2. 3 sessions are created in run and as many tasks ( runfetchers ) to which created sessions and download lists are transferred.
  3. in each runfetchers , 20 fetch runs per session.
  4. after downloading all the results should be collected in one place and open sessions should be closed.

Problem

  1. I just started to understand asyncio \ aiohttp and maybe I don’t think correctly, please correct me.
  2. The code is working, but an error occurs at the closing of open sessions (as I have not tried)

Before that, I tried to do everything in one session and everything works fine, as I used the code from the examples that use the context manager, who closed the session himself, and the examples used the same level of "immersion".

Code

import asyncio import aiohttp import math # генератор который будет разрезать списки def listpartial(inlist, n): if n > 0: s = math.ceil(len(inlist) / n) for q in range(s): yield inlist[q * n:(q + 1) * n] # собсвенно загрузчик страниц async def fetch(num, session): try: url = f'https://en.wikipedia.org/wiki/{num}' async with session.get(url) as r: await asyncio.sleep(0.1, 0.3) return await r.text(encoding='utf-8') except Exception as e: print('fetch', e) return '' async def runfetchers(numlist, session): tasks2 = [] # создам количество задач на загрузку каждого элемента for num in numlist: task = asyncio.ensure_future(fetch(num, session)) tasks2.append(task) responses = await asyncio.gather(*tasks2) return responses async def run(allnums): tasks = [] sessions = [] # получии объект генератор который будет выдавать списки по 20 элементов numlist = listpartial(allnums, 20) # создам 3 сессии и столько же задач и запущу их for _ in range(3): session = aiohttp.ClientSession() sessions.append(session) task = asyncio.ensure_future(runfetchers(next(numlist), session)) tasks.append(task) responses = await asyncio.gather(*tasks) # закрываю открытые сессии # ?????? как закрыть вот эти сессии # for i in sessions: # sessions[i].close() return responses if __name__ == '__main__': nums = [x for x in range(1200)] loop = asyncio.get_event_loop() future = asyncio.ensure_future(run(nums)) res = loop.run_until_complete(future) print(res) 

Conclusion

The answer to my question is presented below (thanks again), but I simultaneously wrote code that solves my problem differently, it may be useful to someone:

 # собсвенно загрузчик страниц async def fetch(num, session): try: url = f'https://en.wikipedia.org/wiki/{num}' async with session.get(url) as r: await asyncio.sleep(0.1, 0.3) return await r.text(encoding='utf-8') except Exception as e: print('fetch', e) return '' # запустим сессию, которая запустит несколько запросов # и потом сама себя закроет async def start_sesssion(numlst): async with aiohttp.ClientSession() as session: futs = [asyncio.create_task(fetch(n, session)) for n in numlst] return await asyncio.gather(*futs) if __name__ == '__main__': nums = [x for x in range(1200)] numlist = listpartial(nums, 20) loop = asyncio.get_event_loop() futures = [loop.create_task(start_sesssion(next(numlist))) for i in range(3)] result = loop.run_until_complete(asyncio.gather(*futures)) print(result) 

    1 answer 1

    ClientSession.close () is a coroutine, therefore

     for session in sessions: await session.close() 

    or

     await asyncio.wait([s.close() for s in sessions]) 
    • I tried it like this, TypeError: list indices must be a garrythehotdog
    • and still error connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x00000171D54C8588>, 307372.078), connector: <aiohttp.connector.TCPConnector object at 0x00000171D52FC9E8> Unclosed connector - garrythehotdog 1313
    • @AlexNasov did not even notice that your cycle was incorrectly written. Corrected. - Sergey Gornostaev
    • Thank you very much! It all worked. Can you explain in a few words why the version presented by me is not correct? - garrythehotdog
    • The @AlexNasov in Python loop variable contains an element of the collection being iterated, not its index. - Sergey Gornostaev