WebClient.DownloadString results in mangled characters due to encoding issues, but the browser is OK

.NetUnicodeUtf 8Webclient

.Net Problem Overview


The following code:

var text = (new WebClient()).DownloadString("http://export.arxiv.org/api/query?search_query=au:Freidel_L*&start=0&max_results=20"));

results in a variable text that contains, among many other things, the string

> "$κ$-Minkowski space, scalar field, and the issue of Lorentz invariance"

However, when I visit that URL in Firefox, I get

> $κ$-Minkowski space, scalar field, and the issue of Lorentz invariance

which is actually correct. I also tried

var data = (new WebClient()).DownloadData("http://export.arxiv.org/api/query?search_query=au:Freidel_L*&start=0&max_results=20");
var text = System.Text.UTF8Encoding.Default.GetString(data);

but this gave the same problem.

I'm not sure where the fault lies here. Is the feed lying about being UTF8-encoded, and the browser is smart enough to figure that out, but not WebClient? Is the feed properly UTF8-encoded, but WebClient is failing in some other way? What can I do to mitigate this?

.Net Solutions


Solution 1 - .Net

It's not lying. You should set the webclient's encoding first before calling DownloadString.

using(WebClient webClient = new WebClient())
{
webClient.Encoding = Encoding.UTF8;
string s = webClient.DownloadString("http://export.arxiv.org/api/query?search_query=au:Freidel_L*&start=0&max_results=20");
}

As for why your alternative isn't working, it's because the usage is incorrect. Its should be:

System.Text.Encoding.UTF8.GetString()

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
QuestionDomenicView Question on Stackoverflow
Solution 1 - .NetLostInComputerView Answer on Stackoverflow