from multiprocessing.dummy import Pool as ThreadPool import requests import random import time def func_main(thr): time1 = time.time() while True: proxy_line = proxy_lines[random.randint(1, len(proxy_lines)-1)].strip() session = requests.session() session.temeout = (5 , 5) Adapter = requests.adapters.HTTPAdapter(pool_connections = 1, pool_maxsize = 0, max_retries = 0) session.mount('http://', Adapter) session.mount('https://', Adapter) session.proxies = { 'http' : proxy_line, 'https' : proxy_line } try: request = session.post('http://www.yahex.ru/') request.close() break except Exception as exception: #print(exception) #print('Thread #' + str(thr + 1) + '|Reconnect|') pass time2 = time.time() - time1 global thr_quantity thr_quantity = thr_quantity - 1 print('Thread #' + str(thr + 1) + ' завершил работу|Прокси:' + proxy_line + '|Активных потоков:' + str(thr_quantity) + '|Время выполнения:{:.3f}'.format(time2)) return() #*******************************************************************# with open('C:\\proxy.txt' , 'r', encoding = 'utf-8') as proxy_file: proxy_lines = proxy_file.readlines() class Profiler(object): def __enter__(self): self._startTime = time.time() def __exit__(self, type, value, traceback): print("*************************\nВремя выполнения программы : {:.3f} sec".format(time.time() - self._startTime)) thr_quantity = 50 with Profiler() as p: with ThreadPool(thr_quantity) as pool: pool.map(func_main, range(thr_quantity)) pool.close()

The problem is this: I run the code above in 50 threads, for example, 49 threads are executed, and on the 50th stream, everything hangs and the stream is not executed. Either on the last one hangs, or on the last 10 streams. Like that. I can not understand the reason. The screenshot below shows the speed of execution of threads. Last 10 minutes as much as he did. But in most cases it does not end. The speed of the last thread

  • I want to note that without using a proxy, everything works correctly, even in 300 threads. Only using a proxy catch sticking of the last few threads. - Mr.Twister
  • what operating system? it looks like tcp connections limit - eri
  • What happens if session.timeout replaced by session.timeout ? - jfs

1 answer 1

By default, Windows has a limit of 20 tcp connections. From this, it turns out dead-lock.

Discussion https://social.technet.microsoft.com/Forums/windows/en-US/18667011-c034-43bc-ab2e-0e87bf811e5e/windows-7-increase-the-limit-of-concurrent-tcp-connections-not -related-to-eula-file-sharing? forum = w7itpronetworking

On Linux, the limitations of hooks are not directly present, but some parameters affect their number.

Discussion https://stackoverflow.com/questions/410616/increasing-the-maximum-number-of-tcp-ip-connections-in-linux

  • Did as you wrote. Alas, it did not help. - Mr.Twister
  • Ie if you limit the number of threads to 16 - does not help? - eri
  • Why exactly 16? I removed the restriction, as you suggested. In fact, the hang occurs regardless of the number of threads. Those. if I put 20 or 300, in 95% of cases, the last couple of threads do not complete their work. You can run the code for the test. With one proxy, or without a proxy, everything works. Perhaps there is no response from the proxy with which the connection has already occurred and this becomes the cause of sticking. I can not understand how to prevent this, because requests.timeout, it helps only for the initial connection. If the connection to the proxy took place, try except, as in my case, it does not help in the course ... - Mr.Twister
  • I hinted to reduce the number of connections - eri
  • And the number of connections may be limited on servers - eri