Storage and access to a large multi-dimensional dynamic array

Question

Can you please tell us how best to store a multidimensional dynamic array (ranging in size from 120GB to 8-20 TB), as well as access its elements without loading the entire array into RAM?

The main condition is a quick read access to the elements of the array, the rest does not matter. In extreme cases, you can any programming language, not just C #.

Initially, it is created in RAM, will be filled with data, then when the size of the OP is not enough, I need to save it and how to work with it from the hard disk, is it possible?
@Merlin, here you are, for sure, not new to programming, but it all turns upside down: when the size of the OP is not enough, I need to save it and how to work with it from the hard disk Well this is absurd!
C # is quite a flexible language, most of the “black” work it takes on, unlike C ++, for example.
But what you are trying to do, the C # developers could not have foreseen =)
And if so, on what equipment are you going to solve the problem and for how long?
(Recall how many hundreds of personal computers initially drove their MapReduce google?)
> what is the best way to store> a multidimensional dynamic array (> from 120GB to 8-20 TB in size), and also get access to its elements without loading the entire array into> RAM?
Quoted1>> The main condition is quick access> readings to the elements of the array;
In extreme cases, you can any> programming language, not just C # This is the problem, if you do not see it, please do not divorce demagoguery.

avp avp 37.4k 3 gold marks 35 silver marks 90 bronze marks · Accepted Answer · 2012-11-19T18:28:29

If the array is "dense", then quickly (although it depends on the method of use) will not work.

Discharged (the vast majority of elements are zeros (or some other predefined value)) can be stored in a hash table with a key of indexes.

avp

37.4k 3 gold marks 35 silver marks 90 bronze marks

Thank. And storage and access to the elements of the array is not in RAM, how to implement? - Merlin
Good question. For example, you can store in a DBMS. But most likely you need to come up with a file format (direct access) in which you can effectively store the hash table. There will simply be the problem of effective access to collisions. It is necessary to come up with such a format so that the elements of collisions fall into one “record” on the disk (in general, so that they can be read by one read). - avp
@Merlin, as I understood from other comments, you are going to use SSD. The idea is good, especially in terms of storing direct access structures, the correct approach to the format may be different compared to the HDD, but as far as I know there are pitfalls with performance. It is known that Oracle logs should not be placed on SSD (it turns out to be faster on HDD). True, I personally have not been able to experiment with them (SSD, etc.). - avp 8:04
@avp third-party DBs do not yet want to use, I will try as you have recommended to experiment with the file format. Tell me please, is this article suitable for the solution? habrahabr.ru/post/124900 I have never encountered file formats. - Merlin
2
Looked at article in Habré. I do not think this is your case. I meant binary file formats (without claims for cross-platform). Those. a format that is quickly (almost without conversions) read-written. Something with which you can work directly with the system read / write / lseek. Naturally, the addresses of the parts of the file read in the memory must be configured. Inside the parts you may be able to store certain offsets (indexes) and work with them, rather than directly with pointers. Generally speaking this, I probably mean more implementation in C. - avp

|

Storage and access to a large multi-dimensional dynamic array

1 answer 1

More articles: