Stop the script and restore its work

Question

Hello. The task is the following, there is a large unloading of goods, the problem is that when downloading pictures, and there are about a few thousand ~ 7000-8000, the server does not have time to process the whole thing, which is logical.

How can you stop the script, for example, after 100 downloads of pictures, and then restore it at the moment of stopping.

My assumption is that you can write the entire array of pictures with a key to a temporary file, then run for `and when it reaches 100 to reload the page, but to transfer to GET where we stopped and run from that place. Tobish turns out, we ran the first 100 products, then passed the parameter script.php? Id = 100 and already from 100 we run to 200 and so on until the end until the array is finished.

The question is, how correct are my assumptions or is it easier to think of something?

UPD

Unloading from another server to your base

At img, hang up the event onready or onload, you can hang up the handler, so that you assign it from src2 to the src url in the next picture of the event in the current one.
сервер не успевает обработать все это дело - and what happens to the server, it gets tired and goes on vacation?

ilyaplot ilyaplot 3.198 one 9 29 · Accepted Answer · 2016-06-17T08:07:24

The most correct solution for downloading data from the supplier's server to your server is a console script that does not have a timeout, unlike a web server. Running such a script from the console will allow all entries to be processed.
The second option is to write to the session the number of the last processed record from the supplier server and to continue processing after this record each time the script is run.

The saddest thing is that the client wants the script to work as well as the crown, and so that he himself started up, I think it will not be easy to train his console
@Komar can raise Jenkins and teach the client to poke a button)
Make a page that will record the command to run the script in a file or database.
Run the script on the crown, which will read this file or the database table and run the download script, if the launch was assigned.
Run the script from the browser! = Watch how it is executed and dump time-out.

slava slava 366 one 13 · Answer 2 · 2016-06-17T08:07:46

To solve a similar problem, I used the following script:

 <?php // Здесь получаем список картинок, которые нужно загрузить - все 8000. // Дальше считаем, что список содержит урл картинки. $images = listImages(); // CACHE_IMAGES - файл с кешем уже загруженных картинок. // Если кеш уже существует, то загружаем его в память, если нет // то создаём пустой кеш. if(file_exists(CACHE_IMAGES)) { $newUrls = unserialize(file_get_contents(CACHE_IMAGES)); } else { $newUrls = array(); } // Счётчик загруженных картинок в текущем запуске. $i = 1; // Пробегаемся в цикле по всем картинкам, которые нужно загрузить. foreach($images as $image) { // Если в кеше уже есть картинка, то пропускаем её загрузку. if(array_key_exists($image, $newUrls)) { continue; } // Если уже загрузили 100 картинок, то завершаем обработку картинок. if($i++ > 100) { break; } // Отмечаем загруженную картинку в кеше. $newUrls[$image] = 1; // Собственно, загружаем картинку. downloadImage($image); } // Сохраняем кеш картинов обратно в файл для следующего запуска file_put_contents(CACHE_IMAGES, serialize($newUrls)); ?>

The script is convenient to run on the crown (I did), but you can also from the browser. An important condition is that you cannot allow multiple instances of the script to work simultaneously.

Based on the above code, you can make different variations - store the cache in the database; do not store the cache at all, but check already downloaded images (on disk or in the database), etc.

In any case, it is better to load in small portions (even when launched from the console), so as not to clog your channel and not put someone else's server.

Stop the script and restore its work

2 answers 2

More articles: