This is not a holivar, but a specific task. There is a base for postgres. And something has become rather bad for her lately. We have at this stage - cron runs the python script, which scans the data for the graphs for the last day every 5 minutes. Total we have in the text file on the server 250k records (graph id, value, time). On php using ajax, I grab the data and parse, depending on the graph, what data to display. Like lepotta. FF loads data for 1s, but the donkey processes data for 10 seconds. So the performance gain is low.

Attention, experts, the question is how to optimize this business? It seems to me that the python should be faster in parsing the values ​​on the ID, but can php page run the script and how can I pass the tag values?

And if you do this with the help of PCP, then transfer the tag values ​​and the time with a getta or a post is a trifling matter. But this again will be parsed by the server and it seems to be from empty to empty? Who kind of did \ did?

UPD

For simplicity, I want to implement a scheme (request to the database from python -> create a file of ALL values ​​-> parsing php data \ python -> the end user from the php page accesses data only for its graphs) stumbled on the intrument of text file parsing. php - it seems easy, but what about performance? python is kind of a cool YaP, but how can I pass ids tags and is it even possible?

разница файла и запроса к базе

alt text

  • one
    So what's the problem? Is that a donkey loaded for a long time? So what does the DB have to do with it, if everything is normal in FF? Maybe see what's wrong with a donkey? And where does PHP or python come from? From the database, the database is taken, the PHP / Python itself does not parse anything. - BOPOH
  • if you rake out of the base, then this is the notorious 10 seconds. FF, of course, decides, but you have to support the donkey. - anunak

1 answer 1

  1. Hire a DB-admin who will configure the database, indexes, and others, once progers are lazy.
  2. To hire a DB-developer, who will write sql queries correctly, since progers are too lazy.
  3. Aggregate data for easy work.
  4. Start working directly from the database, not a gigabyte file and parsing it.
  5. Write api to get data from the client.
  6. Write a client in any language that displays a graph using lightweight plugins.

This is if I correctly understood your question. I work with a huge amount of data, and it works pretty fast, although I see that I can be faster than I do.

  • That, of course, is good. 1. The base is configured, many users are simple. The data is dynamic, and the admin says that indexing them is pointless. 2. The question is open, although I do not think that you can write easier. 3. The issue is controversial. 4. Working with DB is slower than with a file. 5. The item above. This was done before. 6. Used by js. At this stage, such a specific refatoring is not definitely needed. - anunak
  • Very interesting, in the database of dynamic data is it like? Are you changing the database structure on the fly and creating new tables or is your database written with disregard of normal forms with all the consequences? In addition, a well-made database with the cache will not work slower than parsing a text file with the same server using php. -------- The second point - what do you have there on JS written that the spread in the acceptance of data by the client is already 10 seconds for different browsers? As if the server has nothing to do with it, it is up to him to give the data, and he does it the same for both browsers (in theory, this is the way) - Alex Krass
  • des1roer.blogspot.ru/2014/07/highcharthighstock.html like this graphics build. but what about the donkey kernel and ff - here it is worth arguing. FF 2015 and 2007 2007 - the difference in technology at least (not 9). dynamic data - talking about the fact that they are added every 30 seconds - anunak
  • @anunak, you don't understand. IE / FF is a client, the server can only be associated with them by how much it sends data. The server is not interested - it is IE or FF. If everything works well for FF, and for IE it’s bad, then you have to look at it — why IE’s bad. Look at the client side, not the server. You can have a plugin or proxy on IE and it drives all the data through itself. If for FF the time was 10 seconds - then it would be possible to watch the server, and now you are trying to guess the sex of the child by the stars. And all your "optimization" - work with scrap in microelectronics. - BOPOH
  • >> 4. Working with the database is slower than with the file you tell the Apache, Microsoft and the Oracle. And then they, the poor, write their bases, because they think that working with files is, firstly, not convenient, and, secondly, slower. That's naive, right? - BOPOH