How can I limit Parallel.ForEach?
C#.NetAsynchronousparallel.foreachC# Problem Overview
I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.
Is there a way to limit thread number or any other limiter while running Parallel.ForEach?
Demo code:
Parallel.ForEach(listOfWebpages, webpage => {
Download(webpage);
});
The real task has nothing to do with webpages, so creative web crawling solutions won't help.
C# Solutions
Solution 1 - C#
You can specify a MaxDegreeOfParallelism
in a ParallelOptions
parameter:
Parallel.ForEach(
listOfWebpages,
new ParallelOptions { MaxDegreeOfParallelism = 4 },
webpage => { Download(webpage); }
);
MSDN: Parallel.ForEach
Solution 2 - C#
You can use ParallelOptions and set MaxDegreeOfParallelism to limit the number of concurrent threads:
Parallel.ForEach(
listOfwebpages,
new ParallelOptions{MaxDegreeOfParallelism=2},
webpage => {Download(webpage);});
Solution 3 - C#
Use another overload of Parallel.Foreach
that takes a ParallelOptions
instance, and set MaxDegreeOfParallelism
to limit how many instances execute in parallel.
Solution 4 - C#
And for the VB.net users (syntax is weird and difficult to find)...
Parallel.ForEach(listOfWebpages, New ParallelOptions() With {.MaxDegreeOfParallelism = 8}, Sub(webpage)
......end sub)