How I can get web page's content and save it into the string variable

C#asp.netScreen Scraping

C# Problem Overview


How I can get the content of the web page using ASP.NET? I need to write a program to get the HTML of a webpage and store it into a string variable.

C# Solutions


Solution 1 - C#

You can use the [WebClient][1]

Using System.Net;
    
WebClient client = new WebClient();
string downloadString = client.DownloadString("http://www.gooogle.com");

[1]: http://msdn.microsoft.com/en-us/library/system.net.webclient(v=vs.80).aspx "WebClient"

Solution 2 - C#

I've run into issues with Webclient.Downloadstring before. If you do, you can try this:

WebRequest request = WebRequest.Create("http://www.google.com");
WebResponse response = request.GetResponse();
Stream data = response.GetResponseStream();
string html = String.Empty;
using (StreamReader sr = new StreamReader(data))
{
    html = sr.ReadToEnd();
}
    

Solution 3 - C#

I recommend not using WebClient.DownloadString. This is because (at least in .NET 3.5) DownloadString is not smart enough to use/remove the BOM, should it be present. This can result in the BOM () incorrectly appearing as part of the string when UTF-8 data is returned (at least without a charset) - ick!

Instead, this slight variation will work correctly with BOMs:

string ReadTextFromUrl(string url) {
    // WebClient is still convenient
    // Assume UTF8, but detect BOM - could also honor response charset I suppose
    using (var client = new WebClient())
    using (var stream = client.OpenRead(url))
    using (var textReader = new StreamReader(stream, Encoding.UTF8, true)) {
        return textReader.ReadToEnd();
    }
}

Solution 4 - C#

Webclient client = new Webclient();
string content = client.DownloadString(url);

Pass the URL of page who you want to get. You can parse the result using htmlagilitypack.

Attributions

All content for this solution is sourced from the original question on Stackoverflow.

The content on this page is licensed under the Attribution-ShareAlike 4.0 International (CC BY-SA 4.0) license.

Content TypeOriginal AuthorOriginal Content on Stackoverflow
Questionkamiar3001View Question on Stackoverflow
Solution 1 - C#dhineshView Answer on Stackoverflow
Solution 2 - C#ScottView Answer on Stackoverflow
Solution 3 - C#user2246674View Answer on Stackoverflow
Solution 4 - C#Steven SpielbergView Answer on Stackoverflow