The task is to develop a service that will use the folder on the local disk as an input and when files appear in it, the service must parse them, and then put the data into the database. The caveat is that the files come in "pairs" (01012015.xml + 01012015.txt) , and are dropped into the folder around the clock via FTP. I am developing a Windows Service (C #). I tried iteratively (every second) to call the EnumerateFiles method in a folder, group them, then parse and put the received data into the database. However, as a result, exceptions periodically pop up (either the file is not fully downloaded via FTP, or there is no pair for it, etc.). When adding multithreading errors even more. Interested in whether there are any ready-made solutions for such a case?

  • The question is a little off topic - why do you need multithreading, if all the same, everything is limited to "single threaded" I / O. - Kromster
  • Parsing files is faster than stacking data in the database, so at this point you can parallelize the download (opening several connections at once) - klutch1991
  • but what about files that have not yet loaded? - klutch1991
  • one
    I will leave here: msdn.microsoft.com/ru-ru/library/… - Pavel Mayorov
  • one
    @ klutch1991 react to everything and collect data in the list "to processing". Is duplication not a problem? - Kromster

1 answer 1

I had a similar problem.

In the main thread of the program do the interface and control.

Scanning a data folder can be left in 1 service flow - all the same, everything is limited to "single-threaded" I / O (and, accordingly, only an isolated question of scanning a folder is solved in it). But the handlers in the database should be made by workers who can be created, for example, N items and which should be distributed.

I would conjure up with an attempt to open a file for writing (or, as suggested by FileAccess.Read , FileShare.Read ). While it is not loaded, the OS will not allow such access to it. (there may be more correct solutions). And also check the pairing. And only then add the files to the list "for processing", from which they can be disassembled by workers (using the critical section).

  • No need to open it for writing, just FileAccess.Read, FileShare.Read - Pavel Mayorov
  • FileShare.Read is a sufficient guarantee that no one will write anything to this file at the same time - Pavel Mayorov
  • @PavelMayorov thanks for the hint! Do you mind including this in response? - Kromster
  • Gentlemen, can you tell me the minimum action algorithm? :) I don't need the code. Just send in the right direction. - klutch1991
  • @ klutch1991 you worked with threads, about a pool of tasks, workers, crit. you know? - Kromster