There is a site about games. On each page there is a link to buy the game in the store, in which the lowest price.

Once a day I download 5 xml files from different stores, some of them are several megabytes. To create a page I read each file, I find in it a game and a price. I choose the lowest and insert it into the page.

And so for each page you need to read this 5 xml files of several megabytes. Pages a lot. Visitors around 2000 per day. This does not seem to be very efficient, and it consumes a lot of disk and processor resources. How to properly organize this process?

Resolved! I will store in the database.

What do you think is better to store in separate tables or to reduce everything into one? The number and name of the fields in all the XML is different.

    1 answer 1

    Use databases. Write a service that periodically downloads xml, processes them and stores entries in the database. And when the user goes to the page with the game displays the data from the database

    • Thank! A good option. I can not think about how often to download, when to delete outdated data, how to identify data that is outdated. How do I imagine, but I can not think over the algorithm of work. - Roman Romanyuk
    • For example, if you delete old records, and then fill the database with new ones. That while it is filled, someone from users not to receive the data. - Roman Romanyuk