Periodic performance drops when using linq to ConcurrentBag

Hello! I'm trying to deal with serious performance drops in the application.

Drawdowns about which there is a speech appear periodically. Mostly, the iterations are worked out rather quickly, but they are diluted with iterations, which take much longer. Since the application has a graphical display of information, it all looks very twitchy and requires adjustment.

Please look at the following code:

while (true) { var rng = new Random(1); var concurrenBag = new ConcurrentBag<ICollection<(int X, int Y)>>(); Parallel.For(0, 20000, i => { var entry = new List<(int X, int Y)>(); // здесь демонстрируются основные аспекты реального алгоритма var r = rng.Next(0, 3); // примерно 20к обработчиков возвращают коллекцию пары координат for (var j = 0; j < r; j++) // иногда возвращаются null'ы, иногда коллекция с 1 записью, чаще - с 2-мя { // все возвращаемые обработчиками значения идут в ConcurrentBag entry.Add((j, j * j)); } if (entry.Count == 0) entry = null; concurrenBag.Add(entry); }); var sw = Stopwatch.StartNew(); var results = concurrenBag.ToList().AsParallel().Where(x => x != null).SelectMany(x => x).Distinct().ToList(); // как выяснилось, просадки в производительности ведут на эту строчку var time = sw.ElapsedMilliseconds; Console.WriteLine($"CB count: {concurrenBag.Count:00000}, result count: {results.Count:00}, time: {time:000}"); Thread.Sleep(1000); } 

The above code leads to the following result in the console:

 CB count: 20000, result count: 02, time: 032 <- 32мс, для первой итерации с учетом инициализации и всего такого это норма CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 014 <- 14мс, уже не норма CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 015 <- как видно, повторяется каждые несколько итераций CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 019 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 014 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 008 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 004 CB count: 20000, result count: 02, time: 011 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 003 CB count: 20000, result count: 02, time: 004 

I think the idea is clear. In a real application, each "normal" iteration takes 10-15ms, and "subsiding" occur every 6-8 iterations and take up to 150ms

Initially, I assumed that the problem could be with business logic, but you can run the example above and get about the same results. Now I assume that something is wrong with the way I use Parallel.For , AsParallel() or ConcurrentBag , but I have no idea what exactly is wrong.