Actually, how to make a progress bar when downloading files is understandable, but the fact of the matter is that I have a function to import data from a file. We load the file instantly, it doesn’t weigh anything, then we parse it for each line and enter it into the database:

INSERT INTO `datas` ...

This, of course, is done via AJAX, but if there is a lot of data, I would like to show some progress bar: how much is already loaded, how much is left.

But I don’t want to contact the database in parallel every second and see how many new lines there are, this is an extra load, I think. Is it possible to do something differently?

  • It is necessary to store the process id and the counter somewhere. Ajax should periodically refer to the counter by process id. And in the database you will store it or in a file - as you write - Dmitry Kozlov
  • @DmitryKozlov, but where will the theory go faster? I think in the file. - Shevsky

1 answer 1

1. Time based on statistics.

If the time to parse a single line and add it to the database is essentially constant, then no AJAX is required at all. Sent the file, returned the number of lines, multiply the number of lines by parsing and show the user that the processing is in progress.

 Кол-во строк * время разбора одной строки = общее время выполнения.

Of course, there may be errors in time, but a couple of seconds of waiting (+ -) will not frighten the user.

2. Write the result to the cache

When you upload a file, you return the ID of this file to the user; now the user accesses a special method to get the number of processed lines every 3 seconds (for example). The processing script writes the number of rows in the fast storage, for example memcached or redis, that is, in the storage where data is stored in memory and accessed by O (1). In fact, thousands of requests per second for such a storage will not even drop the overall performance.

In general, fantasy is unlimited here :) You can send data to the user in general on web sockets and then the server will send the data to the user, and not the user will receive from somewhere.

  • The first option is clearly not suitable, the time will be different for everyone, I make a program that will be used on my server. - Shevsky
  • For the second I did not understand, I never used it before. What a fast repository, how to write to it (php) and how to get something from it (javascript)? - Shevsky
  • @Shevsky redis, memcached is the key-value of the repository that stores data in RAM, so that access to it is carried out for O (1). Connecting to them via PHP using the driver. I think you can find out more about them by making a request to the search engine. At first, memcached will be enough for you. - Firepro
  • and the client (javascript) can access the RAM? - Shevsky
  • @Shevsky javascript is a client language and in no way has direct access to the server’s RAM. Just write a controller method in PHP that will access memcached and retrieve data by ID, and JS will access it with some ID. - Firepro