I receive data on exchange rates through the API of various exchanges. In total, there are about 3000 courses, each contains price data for every minute throughout the year. Accordingly, about 1.5 billion records are received. The question is, where is all this good to store?
|
1 answer
can be saved to txt on a flash drive)) If in the case: Need to know how you are going to use this data, how often to get results. If this is for a program that must give out its course according to a specified date and currency, then the question arises: why do you download this entire database right away? the API of these exchanges is needed to give you exactly what you need at the moment. Want to download everything and no longer access the API? )) I did the logic like this: I made a request, checked if there was a result in the database, if it showed, if not I make a request to the exchange's API - I received a response, showed it, saved it in the database.
Turalllb turalllb
1,420 6 21
- I need to output these courses in the form of graphs on the site, the number of requests to the API is limited, for this I need to save the data and then output them from the database. And new data arrive every minute, respectively, they need to be added to the rest. - Oxpa 8:36 pm
- And the graphics are required precisely with per-minute resolution? If not, then you can average the data and thus reduce the volume dozens of times. - avp
- Well, I still do not understand where you still want to store data, if not in the database? After all, you will need a quick sample. For example, you need to display the schedule for the last week. Make a request for each hour during the week. I think these points are enough to draw a graph. Well, save to the database. In the database, too, is not difficult to store them. The name of the currency, plus the time already together are a unique identifier to select from the database. - Turalllb
- avp, I think the entire volume is not downloaded at all, what for do I need an API at all, and which API issues the entire database at once? The question is a little incorrectly posed, there is little information. - Turalllb 8:55 pm
- @Turalllb, of course the base is selected in parts. Once per hour (or daily) is selected by API. Once a day, new data can be averaged (say, for an interval of 1 hour) and stored in the database for graphs already. - avp
|