HttpClient - task was cancelled - How to get the exact error message?

I have the following test code. I always get the "Task was cancelled" error after looping 316934 or 361992 times.

If I am not wrong, there are two possible reasons why the task was cancelled a) HttpClient got timeout or b) too many tasks in queue and some tasks got time-out.

I couldn't find the documentation about the limitation in queueing the tasks. And I tried creating more than 500K tasks and no time-out. I guess the reason "b" might not be right.

Q1. Is there any other reason that I missed out?

Q2. If it's because HttpClient timeout, how can I get the exact exception message instead of "TaskCancellation" exception.

Q3. What would be the best way to fix it? Should I introduce the throttler?

Thanks!

var _httpClient = new HttpClient();
_httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept", "text/html,application/xhtml+xml,application/xml");
_httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Encoding", "gzip, deflate");
_httpClient.DefaultRequestHeaders.TryAddWithoutValidation("User-Agent", "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:19.0) Gecko/20100101 Firefox/19.0");
_httpClient.DefaultRequestHeaders.TryAddWithoutValidation("Accept-Charset", "ISO-8859-1");

int[] intArray = Enumerable.Range(0, 600000).ToArray();

var results = intArray                
    .Select(async t => {

        using (HttpRequestMessage requestMessage = new HttpRequestMessage(HttpMethod.Get, "http://www.google.com")) {
            log.Info(t);

            try {

                var response = await _httpClient.SendAsync(requestMessage);
                var responseContent = await response.Content.ReadAsStringAsync();

                return responseContent;
            }
            catch (Exception ex) {
                log.ErrorException(string.Format("SoeHtike {0}", Task.CurrentId), ex);
            }
            return null;
        }
    });

Task.WaitAll(results.ToArray());

Console.ReadLine();

Here is the step to replicate the issue.

1. Create a Console Project in VS 2012.

2. Please copy and paste my code in Main.

3. Put the breakpoint at this line " log.ErrorException(string.Format("SoeHtike {0}", Task.CurrentId), ex);"

Run the program in debug mode. Wait for a few minutes. (maybe 5 minutes? ) I just tested my code and I got the exception after 3 mins. If you have fiddler, you can monitor the requests so that you know the program is still running or not.

Feel free to let me know if you can't replicate the issue.

The default HttpClient.Timeout value is 100 seconds (00:01:40). If you do a timestamp in your catch block you will notice that tasks begin to get canceled at exactly that time. Apparently there is a limited number of HTTP requests you can do per second, others get queued. Queued requests get canceled on timeout. Out of all 600k of tasks I personally got only 2500 successful, others got canceled.

I also find it unlikely, that you will be able to run the whole 600000 of tasks. Many network drivers let through high number of requests only for a small time, and reduce that number to a very low value after some time. My network card allowed me to send only 921 requests within 36 seconds and dropped that speed to only one request per second. At that speed it will take a week to complete all the tasks.

If you are able to bypass that limitation, make sure you build the code for 64-bit platform as the app is very hungry for memory.

Don't dispose the instance of HttpClient you're using. Weird but fixed for me this problem.

just wanted to share I have had a similar code to load test our servers ,and its a high probability that your requests are timing out. You can set the timeout for your http request to max and see if it changes anything for you. I tried hitting our servers by creating various threads. And it increased the hits but they would all eventually timeout.And also you cannot set a timeout when hitting them on another thread.

I ran across this recently. As it turned out, when I started up the web application in debug mode, the start up URL was using https.

However, the URL for WebApi endpoint (in my config file) was using http. Once I update it to use https, I was able to call the WebApi endpoint.