Tell me if there is a solution for this problem. There is a queue of tasks

  • url1 proxy1
  • url2 proxy2
  • url3 proxy1
  • url4 proxy2

How can I make a queue of tasks so that each request is no more than 5 seconds, for example, if Url2 is frozen, so that tasks 1 and 3 are not executed at the same time. Tried the gearman queue server. The solution to this problem there could not be implemented. What other options are there?

Php language

    2 answers 2

    Use cURL parallelism: see the curl_multi_exec example. Your script will need to monitor the health of each proxy.

    Upd. under "monitor" is meant to monitor how long the previous request ended through this proxy - compare timestamps

    • curl_multi does not fit. monitor the proxy is not necessary because proxy their. but the multi-case will not work because there it is impossible to control the queue of requests for a specific proxy. for example, we sent a queue of 10 URLs to 5 proxies, some URLs quickly returned the result, some delayed .. there will be a shift in the order and the probability that 3-4 requests in a row will go to one proxy. - coderxlsn
    • @artur, and you are sure that it is convenient to do a similar task on pkhp? - avp
    • need some kind of server queue and access it via php) - coderxlsn
    • one
      why curl_multi not suitable? The first comment to curl_multi_select gives an example of sorting handles during processing. And before sending a batch of tasks for execution, you remember which handle with which proxy is associated. - Sergiks

    solved the issue through gearman