Background:
“Server Too Busy” means HTTP status code 500 and that means, something is wrong with Web server rather website.
I am not going to discuss dying (when IIS7 is knocking the door) product in too depth. But please look intently at this diagram for 5 minutes before you read further.
Ohhh O, it is not 5 minutes yet!!!!
Ok so it seems that potentially there can be queuing at following places because,
1) HTTP.SYS is the first handler in kernel mode to validate the incoming request and the route the request to appropriate kernel mode queue.
2) Worker Process it self because of multiple AppDomains (in simple terms, it is multiple application inside same worker process roughly).
3) Application it self with I/O Thread pool.
Let’s examine the properties associated with each queue.
Mode | IIS Queues | Symptoms | Property | Location | Default |
Kernel Mode | HTTP.sys | HTTP Error log indicates 503 – Queue Full | AppPoolQueueLength | Metabase | IIS 6 - 1000 |
| | | | | |
Kernel Mode to User Mode transition | W3WP or Thread Pool | HTTP Error log indicates Timer Connection Idle | Please Ignore it for a while because it is a big post in it self!!!! | ||
| | | | | |
User Mode: Global | ASP.Net ISAPI Handler or Application Queue | HTTP 503 – Server Too Busy Error | AppRequestQueueLimit | Machine.config in .Net configuration Folder under <HTTPRunTime> | .Net 1.0 or 1.1 it is 100 .Net 2.0 or higher it is 5000 |
| | | | | |
User Mode: Application | .Net Thread Pool | See the Details below. | MaxWorkerThreads MinWorkerThreads MaxIoThreads MinFreeThreads MinLocalRequestFreeThreads Maxconnection ExecutionTimeout | Machine.config under the element <ProcessModel> affects all applications using that .Net version.[1] OR In web.config for application specific | .Net 2.0 it is set to auto config so generally do not require modification. .Net 1.0 or 1.1 it set to very minimal number and always require modification so scale up. |
Symptoms For User Mode:Global Contention:
1. Event ID: 1003 – aspnet_isapi.dll reported itself as unhealthy for the following reason: ‘Deadlock Detected’
2. Event ID: 1013 – A Process serving application pool ‘udaypandya.com’ exceeded time limits during shut down.
3. “System.InvalidOperationException: There were not enough free threads in the ThreadPool object to complete the operation.”
4. In most extreme situation it gives “HttpException (0x80004005): Request timed out.”
Real world problem:
Client created a support ticket mentioning that server displays “Server Too Busy” error. I have enabled the performance monitoring and got the following output. Customer mentioned that application is using .Net version 2.0. This error is random and can not be reproduced on demand.
I went ahead and checked the HTTPErr log and got the following output:
Logparser Query: logparser -i:HTTPERR -o:DATAGRID "SELECT date, time, c-ip, s-ip, cs-uri, cs-method, sc-status, s-siteid, s-reason, s-queuename from C:\IISLogs\HTTPERR\HTTPERR\httperr42.log WHERE sc-status>500"
Here is a screenshot for Performance Monitor.
Can some one please help me out with what is wrong and how should I troubleshoot it further?
Uday Pandya