There is a file that runs every N minutes. When this file is launched, something like this happens there:

  1. Displays a list of groups from MySQL.
  2. Starts updating information about groups by making a request to the VK API.
  3. Receives the necessary information and writes them to MySQL.

If done consistently, it turns out a critical timeout. It is important that 100 groups run for no more than 1 minute. If you do this task not sequentially but in parallel, in theory it will take approximately 10 seconds or even less. The question will already be in the capacity of the server, but this is not a problem.

You can of course create dozens of files and put them in the CROWN task. But it seems to me that there is a much more elegant solution and in the end, in the end, the right one.

  • one
    The solution depends on the exact known problem. According to your description, nothing concrete can be offered. Most often, when working with SQL, such problems are solved by creating a single query that processes all groups at once. Any work with SQL-DB by small queries processing small parts of data is usually very slow. But of course there are exceptions, when one request turns out to be worse or for some reason impossible at all, and then you have to look for other solutions. At the same time, not forgetting that if operations heavily load the disk, their parallel execution is slower than the sequential one. - Mike
  • @Mike Updated question information. - Dmitriy Movsesyan
  • Well, it turns out the case rests on the calculations and not in the database, but in network requests. Then you have to look towards asynchronous work with the network, so that the script does not stand idle while waiting for a response from the server. If of course VK will allow you to send so many requests at the same time. That's where the curl_multi is offered in the answer, it's just an asynchronous mechanism - Mike
  • @Mike Then such a question. For example, there is a file with 2 functions. How to make them start at the same time and start running parallel to each other? - Dmitri Movsesian
  • Since these are functions that both use the processor, it is only possible to run them in separate threads. habr.com/post/300952 or use a separate process using fork. If you have no complicated calculations that require a processor, I still recommend looking towards asynchrony, and not multitasking - Mike

2 answers 2

In the context of the execution of the PHP script is initially single-threaded. There are various tricks on parallelization, but they are close in their spirit to setting crutches.

Options are as follows:

  1. Use another language. Go example - it is very good in terms of multithreading. Sad but true.
  2. Optimize the script. 10–20 seconds to process 5 entities is, in general, very long. But here you need to know your specifics.
  3. Run multiple daemon processes that will spin in an infinite loop and read jobs from the queue (for example, RabbitMQ, Redis, etc.)
  4. Same as 3, but push assignments to the workers from the master process of the orchestrator.

UPD due to issue update

Most likely your bottleneck is a request to the VK API. In this case, you can use curl_multi_init to parallelize queries.

  • Updated the question information. - Dmitriy Movsesyan
  • Then such a question. For example, there is a file with 2 functions. How to make them start at the same time and start running parallel to each other? - Dmitri Movsesian
  • @DmitryMovsesyan in the framework of one script - no way. You can use the pthreads extension ( php.net/manual/ru/book.pthreads.php ) to start child processes that you can pass on the task, but this is fraught with and not just say - rjhdby

You can try the pthreads or multicurl extension (pay attention to the http://docs.guzzlephp.org/en/stable/ library). Maybe this option will suit you http://php.net/manual/ru/function.pcntl-fork.php . As I understand it, the list of groups can be requested once and then, based on the contents of the list, it is necessary to execute requests to the external API.