Thursday 21 August 2008

How to Troubleshoot IIS6 'Server Too Busy' Error


Background:

“Server Too Busy” means HTTP status code 500 and that means, something is wrong with Web server rather website.

I am not going to discuss dying (when IIS7 is knocking the door) product in too depth. But please look intently at this diagram for 5 minutes before you read further.

Ohhh O, it is not 5 minutes yet!!!!

Ok so it seems that potentially there can be queuing at following places because,

1) HTTP.SYS is the first handler in kernel mode to validate the incoming request and the route the request to appropriate kernel mode queue.

2) Worker Process it self because of multiple AppDomains (in simple terms, it is multiple application inside same worker process roughly).

3) Application it self with I/O Thread pool.

Let’s examine the properties associated with each queue.

Mode

IIS Queues

Symptoms

Property

Location

Default

Kernel Mode

HTTP.sys

HTTP Error log indicates 503 – Queue Full

AppPoolQueueLength

Metabase

IIS 6 - 1000







Kernel Mode to User Mode transition

W3WP or Thread Pool

HTTP Error log indicates Timer Connection Idle

Please Ignore it for a while because it is a big post in it self!!!!







User Mode: Global

ASP.Net ISAPI Handler or Application Queue

HTTP 503 – Server Too Busy Error

AppRequestQueueLimit

Machine.config in .Net configuration Folder under <HTTPRunTime>

.Net 1.0 or 1.1 it is 100

.Net 2.0 or higher it is 5000







User Mode: Application

.Net Thread Pool

See the Details below.

MaxWorkerThreads

MinWorkerThreads

MaxIoThreads

MinFreeThreads

MinLocalRequestFreeThreads

Maxconnection

ExecutionTimeout

Machine.config under the element <ProcessModel> affects all applications using that .Net version.[1]

OR

In web.config for application specific

.Net 2.0 it is set to auto config so generally do not require modification.

.Net 1.0 or 1.1 it set to very minimal number and always require modification so scale up.

Symptoms For User Mode:Global Contention:

1. Event ID: 1003 – aspnet_isapi.dll reported itself as unhealthy for the following reason: ‘Deadlock Detected’

2. Event ID: 1013 – A Process serving application pool ‘udaypandya.com’ exceeded time limits during shut down.

3. “System.InvalidOperationException: There were not enough free threads in the ThreadPool object to complete the operation.”

4. In most extreme situation it gives “HttpException (0x80004005): Request timed out.”

Real world problem:

Client created a support ticket mentioning that server displays “Server Too Busy” error. I have enabled the performance monitoring and got the following output. Customer mentioned that application is using .Net version 2.0. This error is random and can not be reproduced on demand.

I went ahead and checked the HTTPErr log and got the following output:

Logparser Query: logparser -i:HTTPERR -o:DATAGRID "SELECT date, time, c-ip, s-ip, cs-uri, cs-method, sc-status, s-siteid, s-reason, s-queuename from C:\IISLogs\HTTPERR\HTTPERR\httperr42.log WHERE sc-status>500"

Here is a screenshot for Performance Monitor.

Can some one please help me out with what is wrong and how should I troubleshoot it further?

Uday Pandya


No comments: