How to organize multi-threaded file processing (opening, data collection, closing) from QFileInfoList array?

I found a sample code where you can just put it in 1 separate stream. And if for example files will be 20 or more, and you need to separate them, for example, into 4 streams.

dir.cd(pathFolder); dir.setFilter(QDir::Files | QDir::Hidden | QDir::NoSymLinks); QStringList nameFilter; nameFilter << "*.txt"; QFileInfoList list = dir.entryInfoList(nameFilter);

This is what happens during processing. It simply reads 1 line, checks the match and adds it to the QTableWidget.

  for (int i = 0; i < list.size(); ++i) { QFileInfo fileInfo = list.at(i); QFile file(fileInfo.filePath()); if (file.open(QIODevice::ReadOnly)) { QTextStream in(&file); QString line = in.readLine(); if(line.indexOf("str")!=-1) { ui->listWidget_ps->addItem(fileInfo.fileName()); } } }
  • Just like any other multithreaded processing. Use threads or built-in Qt thread pool: QRunnable and company. - ixSci
  • Try QtConcurrent :: run - Alexander Chernin
  • I’ll add that it’s impossible to call interface functions from a non-main thread, so it will be necessary to use signals to connect the workflow threads with the main one - Bearded Beaver

1 answer 1

The fastest way to do what you want without major code changes is to use openmp. To do this, we perform the following steps:

  1. "Enable" openmp, for this we add the following to the .pro file:

     QMAKE_CXXFLAGS += -fopenmp LIBS += -fopenmp

    after that, you may need to clean the project and manually pull qmake

  2. We declare a signal to deliver the found file names from the workflows to the main one:

     signals: void sendFileName(QString fileName);

  3. Replace in a loop the addition of a string in QListWidget to emit this signal:

      if(line.indexOf("str")!=-1) emit sendFileName(fileInfo.fileName());

  4. We connect the signal with the method of adding a line to the sheet (I did it in the designer of the test program window, but in fact I need to do it once before emitting the signal)

     connect(this, &MainWindow::sendFileName, ui->listWidget_ps, QOverload<const QString &>::of(&QListWidget::addItem));

    Pay attention to the QOverload construction, addItem not a slot, because of this, the "old" connection syntax will not work. Also, addItem has overloads that the “normal” new syntax does not know how to make friends with overloaded methods, QOverload used.

  5. Commands openmp to execute our loop in multithreaded mode:

     #pragma omp parallel for for (int i = 0; i < list.size(); ++i) { ...

According to my tests, the performance gain is about 4.5 times on a dataset of 1000 files.