The task is to start a large number of processes (about 10 thousand). I make a test version for 100 processes. I do it using BackgroundWorker. Processes are created. In the working cycle of the process I draw a conclusion in a separate file for each process, the progress of the work with the time stamp, after which I take the process to sleep for 3 seconds. The problem is that the processes do not work simultaneously. According to the logs in the files, it is clear that at first several cycles carry out processes 10-12, then they turn on the same number of times. As a result, the last ten processes begin work more than a minute after the start of the first processes. Tried to do the same using new Task (my_task) .Start (). The result is the same. Tell me, please, what is needed for all tasks to be performed in parallel.

  • one
    Parallel execution will not appear from a simple Task application. You should not cast spells without understanding how they work. - VladD
  • Probably, after all not a process (Process), but a thread (Thread)? - Alexander Petrov
  • BackgroundWorker uses ThreadPool, just like Task. The pool has a size after which tasks are queued, waiting for threads to be released. So you need a path to Thread where memory consumption and overhead to switch threads will be. Well, at 10 thousand everything will be a stake. - vitidev

1 answer 1

When you do Sleep for a process, the process is not released and it is still occupied by Windows. It is the OS that does not have time to allocate many threads to you at the same time. Plus, you write to the file from each process - it is quite resource-intensive exercise.

Try instead of writing to a file during a separate process, throw the log into the thread-safe ConcurrentBag collection and save the common log after all threads work.

To optimize more, you need to see your code.

  • Logs in the files I made to check the suspicions that arose even before that. I don't need the files myself. Initially, the task was to send http requests to the server and it was clear from the server logs that everything did not work as intended. - Oleg Shubin
  • At the same time too few requests came. It seems that the problem is exactly what you described about Sleep. And how can you get around this? - Oleg Shubin
  • I don't think this can be bypassed. Here the reason is pretty well described: cyberforum.ru/csharp-beginners/thread638303.html But it seems to me that you generally went from the wrong end. If your task is to load test the site, then there are many programs for this, from quite good console ( github.com/tsliwowicz/go-wrk description of working with it - habrahabr.ru/post/322266 ) to the UI with the possibility of authorization ( jmeter.apache .org description - habrahabr.ru/post/165159 ) - Ivan Kramarchuk