Good afternoon. There is a window application written in Python 3.5 + PyQt5 . The application is a parallel client that makes many requests to a third-party API . Since the application is a windowed application, performing asynchronous operations using the asyncio library and not blocking the main thread just won't work, Quamash is also Quamash .

The question arose just in the implementation of parallel waiting for a response from the API . I wrote in two ways: asynchronous and multithreaded. I give simplified versions, I will try to reflect the essence. The first one looks like this:

 async def get_async(items): to_do = [] for item in items: # items содержит 35 объектов, соответственно будет выполнено 35 запросов к API url = "..." # подготовка url адреса params = {...} # подготовка GET параметорв to_do.append(asend_request(url, params)) to_do_iter = asyncio.as_completed(to_do) for future in to_do_iter: # получение результата и последующая обработка result = await future async def asend_request(url, params): response = await aiohttp.request("GET", url, params=params) json = await response.json() return json

Everything works fine, the thread is not blocked, but the problem is that it runs in one thread and it is overloaded. The window starts to slow down, abruptly the application starts to eat a lot of operatives. The second option:

 async def get_async(items): loop = asyncio.get_event_loop() futures = {} with QThreadExecutor(35) as executor: for item in items: url = "..." params = {...} future = loop.run_in_executor(executor, send_request, url, params) futures[future] = (item, url, params) for future in futures.keys(): item, url, params = futures[future] response = await future def send_request(url, params): response = requests.get(url, params=params) return response.json()

This option works much better, firstly it processes the answers 3 times faster. Secondly, the window does not drop a bit. Well, this is how I understood it because 35 real threads are created. But it is alarming. As I read, each separate thread of the operating system is a large memory overhead. In theory, there would be enough two. One is responsible for processing the window, and the second asynchronously makes requests, in which case the main thread will not reboot and the window will not lag.

Question: Is it possible to somehow reduce everything to the asynchronous variant and not create one thread for each request, but also so that the main thread is not overloaded? I tried to call an asynchronous function inside QThreadExecutor , it didn't work out, it all comes down to the number of threads, which I pass to the executor 's constructor.

  • Look here at stackoverflow.com/questions/6783194/… - vadim vaduxa
  • @vadimvaduxa, in those examples, the PyQt 'ovsky thread is created and the necessary calculations are performed in it. I tried to do this, allocate a separate stream and load data asynchronously. Nothing happened, I can not use asyncio at asyncio inside this thread. The error takes off that there is no event loop 'a, or something like that. - Klym
  • Here is a description of the error: RuntimeError: There is no current event loop in thread 'Dummy-1'. - Klym
  • It is not clear to what there is more than one thread - all the time on IO goes. In the first variant, the session, semaphore , is not enough to limit the number of simultaneous connections to 35. - jfs
  • @jfs, with a semaphore is even worse, and with him more often eksepshny fly - Klym

1 answer 1

I think the answer here is that each thread should have its own event cycle

asyncio.run_coroutine_threadsafe , concurrent.futures.Executor