I have a Parallel.ForEach() async loop with which I download some webpages. My bandwidth is limited so I can download only x pages per time but Parallel.ForEach executes whole list of desired webpages.

Is there a way to limit thread number or any other limiter while running Parallel.ForEach?

Demo code:

    Parallel.ForEach(listOfWebpages, webpage =&gt; {
      Download(webpage);
    });

The real task has nothing to do with webpages, so creative web crawling solutions won&#39;t help.



How can I limit Parallel.ForEach?

I would like to handle a collection in parallel, but I&#39;m having trouble implementing it and I&#39;m therefore hoping for some help.

The trouble arises if I want to call a method marked async in C#, within the lambda of the parallel loop. For example:

    var bag = new ConcurrentBag&lt;object&gt;();
    Parallel.ForEach(myCollection, async item =&gt;
    {
      // some pre stuff
      var response = await GetData(item);
      bag.Add(response);
      // some post stuff
    }
    var count = bag.Count;
The problem occurs with the count being 0, because all the threads created are effectively just background threads and the `Parallel.ForEach` call doesn&#39;t wait for completion. If I remove the async keyword, the method looks like this:

    var bag = new ConcurrentBag&lt;object&gt;();
    Parallel.ForEach(myCollection, item =&gt;
    {
      // some pre stuff
      var responseTask = await GetData(item);
      responseTask.Wait();
      var response = responseTask.Result;
      bag.Add(response);
      // some post stuff
    }
    var count = bag.Count;
It works, but it completely disables the await cleverness and I have to do some manual exception handling.. (Removed for brevity).

How can I implement a `Parallel.ForEach` loop, that uses the await keyword within the lambda? Is it possible?

The prototype of the Parallel.ForEach method takes an `Action&lt;T&gt;` as parameter, but I want it to wait for my asynchronous lambda.

Parallel foreach with asynchronous lambda

In a metro app, I need to execute a number of WCF calls. There are a significant number of calls to be made, so I need to do them in a parallel loop.  The problem is that the parallel loop exits before the WCF calls are all complete.

How would you refactor this to work as expected?

    var ids = new List&lt;string&gt;() { &quot;1&quot;, &quot;2&quot;, &quot;3&quot;, &quot;4&quot;, &quot;5&quot;, &quot;6&quot;, &quot;7&quot;, &quot;8&quot;, &quot;9&quot;, &quot;10&quot; };
    var customers = new  System.Collections.Concurrent.BlockingCollection&lt;Customer&gt;();

    Parallel.ForEach(ids, async i =&gt;
    {
        ICustomerRepo repo = new CustomerRepo();
        var cust = await repo.GetCustomer(i);
        customers.Add(cust);
    });

    foreach ( var customer in customers )
    {
        Console.WriteLine(customer.ID);
    }

    Console.ReadKey();

Nesting await in Parallel.ForEach

What are the differences between using Parallel.ForEach or Task.Run() to start a set of tasks asynchronously?

Version 1:

    List&lt;string&gt; strings = new List&lt;string&gt; { &quot;s1&quot;, &quot;s2&quot;, &quot;s3&quot; };
    Parallel.ForEach(strings, s =&gt;
    {
        DoSomething(s);
    });

Version 2:

    List&lt;string&gt; strings = new List&lt;string&gt; { &quot;s1&quot;, &quot;s2&quot;, &quot;s3&quot; };
    List&lt;Task&gt; Tasks = new List&lt;Task&gt;();
    foreach (var s in strings)
    {
        Tasks.Add(Task.Run(() =&gt; DoSomething(s)));
    }
    await Task.WhenAll(Tasks);




Parallel.ForEach vs Task.Run and Task.WhenAll

I am a bit of confused about `Parallel.ForEach`.  
What is `Parallel.ForEach` and what does it exactly do?  
Please don&#39;t reference any MSDN link.  

Here&#39;s a simple example : 

    string[] lines = File.ReadAllLines(txtProxyListPath.Text);
    List&lt;string&gt; list_lines = new List&lt;string&gt;(lines);

    foreach (string line in list_lines)
    {
        //My Stuff
    }

How can I rewrite this example with `Parallel.ForEach`?


How can I convert this foreach code to Parallel.ForEach?

Erg, I&#39;m trying to find these two methods in the BCL using Reflector, but can&#39;t locate them. What&#39;s the difference between these two snippets?

A:

    IEnumerable&lt;string&gt; items = ...
    
    Parallel.ForEach(items, item =&gt; {
       ...
    });

B:

    IEnumerable&lt;string&gt; items = ...
    
    foreach (var item in items.AsParallel())
    {
       ...
    }

Are there different consequences of using one over the other? (Assume that whatever I&#39;m doing in the bracketed bodies of both examples is thread safe.)


Parallel.ForEach() vs. foreach(IEnumerable&lt;T&gt;.AsParallel())

**How do I break out of an [parallel.for][1] loop?**

I have a pretty complex statement which looks like the following: 

    Parallel.ForEach&lt;ColorIndexHolder&gt;(ColorIndex.AsEnumerable(),
        new Action&lt;ColorIndexHolder&gt;((ColorIndexHolder Element) =&gt;
        {
            if (Element.StartIndex &lt;= I &amp;&amp; Element.StartIndex + Element.Length &gt;= I)
            {
                Found = true;
                break;
            }
        }));


Using parallel class, I can optimize this process by far. However; I cannot figure out how to break the parallel loop? The `break;` statement throws following syntax error: 

&gt; No enclosing loops out of which to break or continue

  [1]: http://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.for.aspx

Break parallel.foreach?

I&#39;m calling a slow webservice in parallel. Things were great until I realized I need to get some information back from the service. But I don&#39;t see where to get the values back. I can&#39;t write to the database, HttpContext.Current appears to be null inside of a method called using Parallel.ForEach

Below is a sample program (in your mind, please imagine a slow web service instead of a string concatenation)


    using System;
    using System.Threading.Tasks;
    
    class Program
    {
        static void Main(string[] args)
        {
            WordMaker m = new WordMaker();
            m.MakeIt();
        }
        public class WordMaker
        {
            public void MakeIt()
            {
                string[] words = { &quot;ack&quot;, &quot;ook&quot; };
                ParallelLoopResult result = Parallel.ForEach(words, word =&gt; AddB(word));
                Console.WriteLine(&quot;Where did my results go?&quot;);
                Console.ReadKey();
            }
            public string AddB(string word)
            {
                return &quot;b&quot; + word;
            }
        }
    
    }



How do I collect return values from Parallel.ForEach?

I am using Parallel.ForEach and I am doing some database updates, now without setting MaxDegreeOfParallelism , a dual core processor machine results in sql client timeouts, where else quad core processor machine somehow does not timeout.

Now I have no control over what kind of processor cores are available where my code runs, but is there some settings I can change with MaxDegreeOfParallelism that will probably run less operations simultaneously and not result in timeouts?

I can increase timeouts but it isnt a good solution, if on lower CPU I can process less operations simultaneously, that will put less load on cpu.

Ok I have read all other posts and MSDN too, but will setting MaxDegreeOfParallelism to lower value make my quad core machines suffer?

For example, is there anyway to do something like, if CPU has two cores, then use 20, if CPU has four cores then 40?

What does MaxDegreeOfParallelism do?

I had such method:
    
    public async Task&lt;MyResult&gt; GetResult()
    {
    	MyResult result = new MyResult();
    
    	foreach(var method in Methods)
    	{
    		string json = await Process(method);

    		result.Prop1 = PopulateProp1(json);
            result.Prop2 = PopulateProp2(json);

    	}
    	
    	return result;
    }

Then I decided to use `Parallel.ForEach`:

    public async Task&lt;MyResult&gt; GetResult()
    {
    	MyResult result = new MyResult();
    
    	Parallel.ForEach(Methods, async method =&gt;
    	{
    		string json = await Process(method);	
	
    		result.Prop1 = PopulateProp1(json);
            result.Prop2 = PopulateProp2(json);
    	});
    	
    	return result;
    }
But now I&#39;ve got an error:

&gt;An asynchronous module or handler completed while an asynchronous operation was still pending.

All parallel.foreach Solutions on Gang of Coders

Total of 10 parallel.foreach Solutions