WebRequest and WebResponse has issues

[Complete]

WebRequest and WebResponse has issues

I wrote a C# program that uses WebRequest and WebResponse to perform a simple web crawler. I discovered something about web sites. Web browsers such as IE and FireFox offer the capacity to view the HTML source code. But it seems that html code that is sent to the browser is one thing and what the browser interprets and displays is something else. For example, if you run a google search in IE and run the same google search in FireFox, the content that you can see when you view the source in IE will NOT have the hyperlinks and content from the search results, but you can see the html hyperlinks and content from the search results when you view the source in FireFox. So my question is this. How do you specialise the WebRequest and WebResponse to show the content after it is processed by the browser instead of before?

[+Nik Louch Subscriber²]

I am not sure you understand what you're saying...

[ZakO]

It's not an 'issue', it's by design. It's not the responsibility of WebRequest/WebResponse to execute client-side scripts, no content loaded by XHR requests will be returned, just the original HTML document. I believe if you want to return the HTML after it's been modified by a client-side script you would need to use a full web browser control rather than a WebRequest.

[Eric Veteran]

*WebResponse and *WebRequest only retrieve and send data. You'll have to either parse it yourself or run it through the WebBrowser control to get the final rendered result.

[Complete]

I think I will have to use the WebBrowser class instead.

How do I expose the LoadCompleted method in the WebBrowser class in WPF C#?

I am trying to write a C# program in wpf that retrieves the content of a web page.

The first thing I tried was to try the WebRequest and WebResponse classes. This did not provide the actual displayed content. WebResponse reveils the HTML code that is sent to the browser. But I discovered that, while the page is being loaded by the browser, javascript can change what content is finally displayed in the browser.

So I decided to use the WebBrowser class.

Immediately I found that there are two WebBrowser classes. Thee is the one that is documented for WinForms and there is another that is documented for WPF. I need to understand the one documented for WPF. What I think I neeed to know what to do is to retrieve code after the "LoadCompleted" method is caused. But I do not know how to this and I cannot find any example demonstrating how this is done.

[Eric Veteran]

In whatever class you're hosting the control in (Page, Window, etc) you need to add a handler. You can either put it in the class's initialization routine as

myBrowser.LoadCompleted += WebBrowser_LoadCompleted

or put it in the XAML in the WebBrowser declaration.

<WebBrowser Name="myBrowser" LoadCompleted="WebBrowser_LoadCompleted"/>

[Complete]

I am getting close to solving this and having a working bit of code. As things stand right now, the call back function for LoadCompleted is not called with the code is stepped through.

Why doesn't this call back function get called?

Basically here is the code surrounding the declaratoini callback method:

webbrowser1 = new WebBrowser();
webbrowser1.LoadCompleted +=webbrowser1_LoadCompleted;
webbrowser1.Navigate(new Uri([url="http://www.google.com"]http://www.google.com[/url]"));

Should there be something more or are they in the wrong order?

The method, webbrowser1_LoadCompleted, is never called. I have put breakpoints in the callback method and the running program never reaches this method:

		 void webbrowser1_LoadCompleted(object sender, NavigationEventArgs e)
		{
			.
			.
			.
		}

I must be missing a reference. I do not know what one I am missing. Can you offer a suggestion?

By the way, that block above is placed there by the editor on this forum, it is not how my code actually looks like

[Eric Veteran]

I tested it with the following:

WebBrowser b = new WebBrowser();
b.Loaded += b_Loaded;
b.Navigated += b_Navigated;
b.LoadCompleted += b_LoadCompleted;
b.Navigate(http://microsoft.com);
[/CODE]

It seems to fire those three events in that order: Loaded, Navigated, then LoadCompleted. LoadCompleted doesn't fire until the entire page's content is completely downloaded.

Yeah, the code tags are dumb. There should be quotes around the Uri string.

[Complete]

I tested it with the following:

WebBrowser b = new WebBrowser();
b.Loaded += b_Loaded;
b.Navigated += b_Navigated;
b.LoadCompleted += b_LoadCompleted;
b.Navigate(http://microsoft.com);
[/CODE]

It seems to fire those three events in that order: Loaded, Navigated, then LoadCompleted. LoadCompleted doesn't fire until the entire page's content is completely downloaded.

Yeah, the code tags are dumb. There should be quotes around the Uri string.

I am close. It works but, at the same time it does not completly come through. When I use a google search page as a test. all the methods you mention are called, but the HTMLDocument I extract when these methods are fired contain the HTML from the home page of google.

Also, if I put breakpoints on these methods, at watch the output to see if the WebBroswer class loads the google results page, it does not. The page is blank until the program is idle.

What do you think?

[sathenzar]

If I am not mistaken it is because google uses ajax to load div content to show the results as you type and the load completed just will pull the html document that was originally loaded. If I use ajax requests and they change div content and then go to view source in any browser it will just show my original document.

[Complete]

using a few other methods and some "jigglery pokery" I finally got to the point where I could extract the HTMLDocument