I need to create an Excel report and download it. The data is taken from SQL (approximately 200 thousand rows with 70 columns). The number of columns is not known in advance (otherwise I would just have stuffed everything into the structure and norms), since pivot is used for normal output, without it there would be about 20 columns for 1,400 thousand lines. The request itself takes about two minutes - bad, but it will do. Then there is an entry in each Excel cell separately, because I did not find any other way to transfer all this, here it is (using "github.com/360EntSecGroup-Skylar/excelize" and _ "github.com/denisenkom/go-mssqldb" // s ):

rows, err := db.Query("[Proc]", sql.Named("param", param)) if err != nil { log.Fatal(err) } defer rows.Close() ( colNames, err := rows.Columns() if err != nil { fmt.Println("error fetching column names\n", err) } length := len(colNames) // Create a interface slice filled with pointers to interface{}'s pointers := make([]interface{}, length) container := make([]interface{}, length) for i := range pointers { pointers[i] = &container[i] } rowcol := 9 col := 1 for i := 0; i < len(colNames); i++ { xxxfile.SetCellValue(sheetName, getExcelColumnName(col)+"8", colNames[i]) col++ } // Process sql rows for rows.Next() { column := 1 // Scan the sql rows into the interface{} slice err = rows.Scan(pointers...) if err != nil { fmt.Println("error scanning sql row\n", err) } // Here we range over our container and look at each column // and set some different options depending on the column type. for _, v := range container { xxxfile.SetCellValue(sheetName, getExcelColumnName(column)+strconv.Itoa(rowcol), v) column++ } rowcol++ } 

After the recording, each line is processed in a cycle (there is a group attribute in the line, according to which it is formatted and the style is assigned)

How to optimize it? Must be reduced to a few minutes

  • Questions: a) github.com/denisenkom/go-mssqldb - which version of MSSQL? b) Do you have direct access to the tables or only through the stored procedure [Proc] ? c) did you try to prepare a csv file instead of an excel file and then use it as a source? - Alex Yu
  • a) 14, b) is, c) no, now I will try - Jeltopuz

1 answer 1

Analysis

Consider the problem "How to optimize creation" of 20 columns by 1,400 thousand rows in excel "more carefully:

  1. DB - MSSQL 2017
  2. The data source is a stored procedure [Proc] . Access to the source tables used in [Proc] is. Execution time: ~ 2 minutes
  3. The excelize library is excelize
  4. The program organizes a cycle with a bypass of 20 columns and 200 thousand rows from the original data set.
  5. In a loop, there is a manipulation with getCellValue , setCellValue (the role of pointers and container not entirely clear
  6. The xxxfile.SetCellValue(sheetName, getExcelColumnName(column)+strconv.Itoa(rowcol), v) conversion itself is not entirely clear, but it is clear that it does not contain any rocket science processing algorithms

How to optimize for rows.Next(){ ... } ?

No The fastest operation is the one you do not perform.

According to the initial data received from OPA, we can recommend the following way:

  1. Take the procedure [Proc]
  2. Either we change it, or we write the output from it to the insert into .. exec [Proc] table, or we do view / function - what we consider the most appropriate at the MSSQL level
  3. We create a query in SQL that performs a conversion similar to getExcelColumnName(column)+strconv.Itoa(rowcol), v) . If problems arise, we create a question with mssql, t-sql mssql and wait for the help of the community (I am personally ready to help, I am convinced that there is something simple)
  4. We change the program on the goal, leaving only db.Query in it to the created request and writing to the Excel file without any processing
  5. We measure the execution time, we find that it has become less than a minute.

Ready to help in case of difficulties