{infiniteZest}
// Articles. Tutorials. Utilities.
Home  |   Search  |   Login  
Categories Skip Navigation Links
New / All
AJAX
Apple
ASP.NET
.NET
Git
Google / Android
Python / IronPython
Miscellaneous
SQL Server
Using Fiddler for debugging HTTP issues (including RewritePath issues)
Summary
Fiddler lets you capture the HTTP traffic on your computer very easily. You would be able to test how your site responds to requests from search engines very easily (by putting that user-agent string in the HTTP request). In addition, what’s discussed in this article can be used to debug things like the problems with RewritePath.
 
Table of Contents

What is Fiddler?

The issue and test scenario

The Request Builder

Figure 1: Fiddler Request Builder

What results should we expect?

Figure 2: Results from the HTTP Request

Inspecting the data that came back

Figure 3: Inspecting the returned data

Raw Tab

The Extensions

Syntax View

Figure 4: Nicely formatted code with Fiddler’s Syntax View

Web View

Figure 5. The web page UI from the WebView tab

Where to get Fiddler?

 

What is Fiddler?

Fiddler is an impressive HTTP/S debugging tool from Microsoft that can be used to simulate various user agents (i.e. browsers like Internet Explorer, FireFox, Safari, etc.) going against your web server without actually installing those browsers on the computer.

Now, to make it clear, this tool doesn’t display the UI for that particular user agent; so, you wouldn’t see how a particular page from your website looks and feels and behaves inside a particular browser. Fiddler also comes with extensions – which includes Web View Extension. This tab in the Fiddler shows you how the page would be rendered inside the Internet Explorer (see the difference: a particular browser vs. IE).

However, you can send a particular user agent string to your server and see what your server returns.

The issue and test scenario

At the minimum, your web server should respond without an error to all the legitimate user agent strings (major browsers, major search engine spiders, etc.) when a legitimate page is requested.

Use the following steps to test (details follow):

  • Use the Request Builder to input a URL and a User Agent
  • Check the Results pane for the result code
  • Check the tabs in Session Inspector to see what came from the server
  • Optionally, check the Performance Statistics tab to see how many bytes were sent and how many bytes were returned

The Request Builder

Using the Request Builder tab in Fiddler, enter your http query and the user agent you wish to test it for. The example values could be:

URL: http://www.infinitezest.com/articles/search-engine-issues-with-rewritepath.aspx

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

In the above case, we are testing how our web server (and the web framework ASP.NET 2.0) reacts to this particular user agent. In this case, the user agent is the Googlebot.

Figure 1: Fiddler Request Builder

Figure 1: Fiddler Request Builder

You need a whole bunch of real-world user-agent strings (both browsers and search engines) for the above test. You can get them from your web server logs. You can also get some popular user-agent strings from the following article:

Important User Agent Strings

What results should we expect?

If everything is working fine, you will see the HTTP OK result code 200. If the result code is 500, you should worry about it – basically the server is erroring out for that particular user agent string. Or, the error could be for a particular page, where you will see the error code for all the user agents.

Figure 2: Results from the HTTP Request

Figure 2: Results from the HTTP Request

There were some issues with the pages redirected with RewritePath on ASP.NET 2.0. The specific solution for that problem is discussed in the following article:

Search Engine Issues with RewritePath

If you have widgets, etc. on your computer that are making HTTP calls (in the figure above requests are being made to msn.com), the Fiddler will continue to capture that traffic as well. In general, you don’t to have to worry about them. If these entries bother you, you can simply select them and delete them (Delete is on the right-click menu).

Inspecting the data that came back

You can use the Session Inspector tab to see what the server sent to this particular request from this particular user agent as shown in the figure below.

Figure 3: Inspecting the returned data

Figure 3: Inspecting the returned data

The bottom portion of the Session Inspector has several tabs to facilitate viewing the returned data in different ways. Some notes on this:

Raw Tab

Here you can see all the data returned in its raw form, including the HTTP headers. This is where you realize ASP.NET (or whatever the framework you are using) does a whole lot for you. For example:

  • The __VIEWSTATE hidden input contains a gigantic string to facilitate the state management.
  • Various scripts written by you and generated by the framework
  • The HTML generated by the framework for the menus, master pages, etc.
  • The HTML for your page, of course

The tabs here are self-explanatory. Two tabs – Syntax View and Web View --come from adding extensions, which are discussed below.

In addition, there is a Performance Statistics tab that you can use in conjunction with the Session Inspector tab to see if you can cut down on the number bytes you are sending for this page.

The Extensions

In addition to the basic Fiddler, you can also install extensions. Two extensions I found useful for Fiddler are Syntax View and Web View. You can download these extensions from the Fiddler page.

Syntax View

As the name suggests, the code is colorized depending on the syntax. Makes it nice to read what came back from the server.

Figure 4: Nicely formatted code with Fiddler’s Syntax View

Figure 4: Nicely formatted code with Fiddler’s Syntax View

Web View

After installing the WebView extension, you see how the downloaded page would look like when drawn with the IE rendering engine.

Figure 5. The web page UI from the WebView tab

Figure 5. The web page UI from the WebView tab

Where to get Fiddler?

If you haven’t already downloaded Fiddler, you can get the latest version of Fiddler here:

http://www.fiddler2.com/Fiddler2/

Bookmark and Share This

More Articles With Similar Tags
icon-fiddler-googlebot-http-request-results.jpg
RewritePath used to service search engine friendly urls has problems with search engines. While this works fine with regular browsers, the request fails with the user agents of most all search engines. This article looks at the workarounds (fix from Microsoft was not available by early 2007)
With ID as a query string parameter (...aspx?ID=...) being a big no-no from the search engine perspective, we would need a way to convert the convenient fake urls into the ones our system understands (like using ID= in the url). There are several ways of accomplishing this; we will look at a couple of quick options.
Transferring control from one aspx page to another is a pretty common need. This piece looks at one scenario.
icon-user-agent-strings.jpg
This article lists the important User-Agent strings (i.e. the IDs of the browsers that contact your web server). You can use these strings to do quick tests of accessing your site via tools like Fiddler.
This article looks at some quick basics of using the Android Debug Bridge that lets you manage the Android emulator/device.
About  Contact  Privacy Policy  Site Map