The client downloads all the data (price lists) via csv , I save this file, and every time I need to, I give it out. Why is that? Because each price has its own structure. The files are very large, 200K each, and sometimes 1-2M lines each.

Is the practice of storing data in csv files normal, or is it better to use a database for such purposes?

What with the database , what with csv , php requires memory_limit in 1024 - otherwise a white screen.

    2 answers 2

    First we need to separate the concepts of storage and data exchange .

    For storage, it is most reasonable to use a database, if only because we have the SQL language and we can create flexible queries using it.

    But for data exchange (uploads, reports, alarm messages, etc.), the CSV format fits just fine. In addition to CSV, there is also a tab-delimited text and several less common formats.

    It must be said that all of them appeared in the era "before XML" (and even more so "before JSON"), but they proved themselves superbly in all sorts of integration solutions and data exchange between heterogeneous systems. In short, these formats are like Windows XP - they look old, and people love and use it.

    In the given case, the client is likely to store data in a relational database management system, and for you it simply unloads the prices. This is a common practice.

    Most modern DBMS contain tools (tools) for data import / export operations of their plain-text formats. Some of them are intuitive and easy to use, others (for example, the SQL Server Import and Export Wizard and traditionally the entire Microsoft product line) require dancing with tambourines and some effort.

    Well, any high-level PL has a complete set of tools to write your own converters and parsers for CSV / Tab-delimited files. The benefit of classes and add-ins on the web is full.

    • In the above case, everything is stored in csv and when some user of the site loads the page - the csv file is opened and read to display on the user's screen. The client simply exports from eksel to csv and uploads to the site for further storage. Nothing is stored in the database. About other formats and stories - heard, you can not explain) Thanks for the answer! - ka5itoshka

    If you are not going to analyze this data, then store it in CSV. By "analyze", I mean here a search by content, a comparison of prices and characteristics, etc. In any case, it will be reasonable to analyze the structure of your files and select several common fields (product, part number, price, stock status), as well as highlight The meta data set is the size and name of the file, who uploaded it and when. And this information is already stored in the database.

    About memory_limit : do you handle it somehow when loading a file? Maybe you should consider creating a processing queue:

    1. Client uploads file to server
    2. The file is placed in some temporary folder and a task is created to process it.
    3. A separate process (cron-task or a certain demon) polls the queue of tasks, finds the first unprocessed file, does something with it and marks this task as performed with one or another result.
    4. A notification is generated for the client, which is communicated to it in one way or another.
    • Thanks for the answer! No, analysis is not needed at all, there is no information needed for "us". When loading in any way. Only when the user enters the brand page, the csv file is read with the goods and then the memory_limit - because php bypasses all the fields, parses the csv row and forms the table into an array, and then this table is output to the page. - ka5itoshka
    • Rather, even the client simply downloads, and the end user simply browses the file. The client can still edit these files (the functionality is already there), and the user only opens the page that reads the csv and creates an array in a loop, which is then output. - ka5itoshka
    • one
      Then create static html pages from this data, store them in files and send them directly, and in the database store only links to these files. - avp
    • @ ka5itoshka: in this case, the creation of the background-processing of your CSV files is just appropriate: background conversion to static html-pages - Boris