Recommendation:
===============
==> In this case TCPIP.SYS driver was the cause for the BugCheck as the IRP was stuck at the TCPIP.SYS.
==> Apply MS KB 962238 to update TCPIP.SYS and AFD.SYS
==> Disable TCP Chimney Feature of Windows Server 2003 SP2. You can follow the article below to disable TCP Chimney either using the package mentioned in the article or you can refer to the workaround section for disabling it manually.
Debug Trace:
============
DRIVER_LEFT_LOCKED_PAGES_IN_PROCESS (cb)
Caused by a driver not cleaning up completely after an I/O.
When possible, the guilty driver’s name (Unicode string) is printed on
the bugcheck screen and saved in KiBugCheckDriver.
Arguments:
Arg1: b7bf92f2, The calling address in the driver that locked the pages or if the
IO manager locked the pages this points to the dispatch routine of
the top driver on the stack to which the IRP was sent.
Arg2: b7bfa933, The caller of the calling address in the driver that locked the
pages. If the IO manager locked the pages this points to the device
object of the top driver on the stack to which the IRP was sent.
Arg3: 8975d618, A pointer to the MDL containing the locked pages.
Arg4: 00000003, The number of locked pages.
Debugging Details:
——————
PEB is paged out (Peb.Ldr = 7ffd500c). Type “.hh dbgerr001” for details
PEB is paged out (Peb.Ldr = 7ffd500c). Type “.hh dbgerr001” for details
FAULTING_IP:
afd!AfdAllocateMdlChain+bf
b7bf92f2 8b45dc mov eax,dword ptr [ebp-24h]
DEFAULT_BUCKET_ID: DRIVER_FAULT
BUGCHECK_STR: 0xCB
PROCESS_NAME: cognos.cgi
CURRENT_IRQL: 0
LAST_CONTROL_TRANSFER: from 8084cc7b to 80827c83
STACK_TEXT:
b49b2c38 8084cc7b 000000cb b7bf92f2 b7bfa933 nt!KeBugCheckEx+0x1b
b49b2c68 8094c4eb ffffffe5 898cbcb8 f1256323 nt!MmCleanProcessAddressSpace+0x551
b49b2cf0 8094c569 f1256323 b49b2d4c 8082e0d6 nt!PspExitThread+0x5f1
b49b2cfc 8082e0d6 898cbcb8 b49b2d48 b49b2d3c nt!PsExitSpecialApc+0x1d
b49b2d4c 80889867 00000001 00000000 b49b2d64 nt!KiDeliverApc+0x1ae
b49b2d4c 7c8285ec 00000001 00000000 b49b2d64 nt!KiServiceExit+0x56
WARNING: Frame IP not in any known module. Following frames may be wrong.
0012f3bc 00000000 00000000 00000000 00000000 0x7c8285ec
STACK_COMMAND: .bugcheck ; kb
FOLLOWUP_IP:
afd!AfdAllocateMdlChain+bf
b7bf92f2 8b45dc mov eax,dword ptr [ebp-24h]
SYMBOL_NAME: afd!AfdAllocateMdlChain+bf
FOLLOWUP_NAME: MachineOwner
MODULE_NAME: afd
IMAGE_NAME: afd.sys
DEBUG_FLR_IMAGE_TIMESTAMP: 48f76164
FAILURE_BUCKET_ID: 0xCB_afd!AfdAllocateMdlChain+bf
BUCKET_ID: 0xCB_afd!AfdAllocateMdlChain+bf
Followup: MachineOwner
———
==>> Now, use the !search extension to find the virtual addresses that are associated with the MDL pointer.
13: kd> !search 8975d618
Debuggee is a kernel summary dump, some physical pages may not be present.
Searches will miss hits from those pages.
Searching PFNs in range 00000001 – 0022FFFE for [FFFFFFFF8975D618 – FFFFFFFF8975D618]
Pfn Offset Hit Va Pte
– – – – – – – – – – – – – – – – – – – – – – – – – – –
000008A6 0000054C 8975D618 808A654C C0404530
00105EA6 0000016C 8975D618 876A616C C043B530
876a6160+0xc : Irp — Io, IRP packets
00105EF2 00000828 8975D718 876F2828 C043B790
876f2000+0x828 : Io (Free) — general IO allocations
00106EE8 00000F70 8975D698 886E8F70 C0443740
886e8f60+0x10 : AfdC (Protected) — Afd connection structure
00107E22 000008DC 8975D698 896228DC C044B110
896228a8+0x34 : File (Protected) — File objects
00107FC3 00000EE4 8975D698 897C3EE4 C044BE18
897c3e18+0xcc : TCPA — TCP/IP network protocol
00107FC3 00000EEC 8975D698 897C3EEC C044BE18
897c3e18+0xd4 : TCPA — TCP/IP network protocol
00107FC3 00000EF4 8975D698 897C3EF4 C044BE18
897c3e18+0xdc : TCPA — TCP/IP network protocol
00107FC3 00000F04 8975D698 897C3F04 C044BE18
897c3e18+0xec : TCPA — TCP/IP network protocol
00107FC3 00000F1C 8975D698 897C3F1C C044BE18
897c3e18+0x104 : TCPA — TCP/IP network protocol
001085BD 00000B68 8975D698 89DBDB68 C044EDE8
89dbda78+0xf0 : TCPT — TCB pool
001085BD 00000BEC 8975D698 89DBDBEC C044EDE8
89dbda78+0x174 : TCPT — TCB pool
00108E75 00000034 8975D618 8A675034 C04533A8
8a675000+0x34 : Dump (LargePool) — Bugcheck dump allocations
00109211 00000CD8 8975D618 8AA11CD8 C0455088
8aa11cc8+0x10 : MmLk — Bugcheck dump allocations
0010A79A 00000318 8975D618 8BF9A318 C045FCD0
8bf9a2f0+0x28 : TCPr — TCP request pool
001DDAE0 0000082C 8975D618 B49B282C C05A4D90
001DDAE0 000008BC 8975D618 B49B28BC C05A4D90
001DDAE0 00000C2C 8975D618 B49B2C2C C05A4D90
001DDAE0 00000C4C 8975D618 B49B2C4C C05A4D90
Search done.
==>> For each virtual address (VA) found, look for an IRP signature. Do this by using the dd command with the VA minus one DWORD.
13: kd> dd B49B2C4C-4 l4
b49b2c48 b7bfa933 8975d618 00000003 87ac10e0
13: kd> dd B49B2C2C-4 l4
b49b2c28 b7bfa933 8975d618 00000003 00000000
13: kd> dd B49B28BC-4 l4
b49b28b8 b7bf92f2 8975d618 00000000 01330180
13: kd> dd B49B282C-4 l4
b49b2828 b7bfa933 8975d618 00000003 bad8b242
13: kd> dd 8BF9A318-4 l4
8bf9a314 876a6228 8975d618 00000000 00000000
13: kd> dd 8AA11CD8-4 l4
8aa11cd4 87708550 8975d618 007dd000 00000003
13: kd> dd 8A675034-4 l4
8a675030 b7bfa933 8975d618 00000003 45474100
13: kd> dd 89DBDBEC-4 l4
89dbdbe8 b7bfaf81 8975d698 00000000 00000000
13: kd> dd 89DBDB68-4 l4
89dbdb64 b7bfb4bb 8975d698 00000000 00000000
13: kd> dd 897C3F1C-4 l4
897c3f18 b7bfaf81 8975d698 00000000 00000000
13: kd> dd 897C3F04-4 l4
897c3f00 b7c08f11 8975d698 00000000 00000000
13: kd> dd 897C3EF4-4 l4
897c3ef0 b7bfb4bb 8975d698 00000000 00000000
13: kd> dd 897C3EEC-4 l4
897c3ee8 b7c077bb 8975d698 b7bfb4bb 8975d698
13: kd> dd 897C3EE4-4 l4
897c3ee0 b7bf9ebb 8975d698 b7c077bb 8975d698
13: kd> dd 896228DC-4 l4
896228d8 00000000 8975d698 00000000 00000000
13: kd> dd 886E8F70-4 l4
886e8f6c 00111030 8975d698 891005e0 8aa33030
13: kd> dd 876F2828-4 l4
876f2824 61436d4d 8975d718 00000000 808ab4d4
13: kd> dd 876A616C-4 l4
876a6168 01d80006 8975d618 00000000 00000000 ==>> $ Matches signature
13: kd> dd 808A654C-4 l4
808a6548 b7bfa933 8975d618 00000003 00000000
==>> After a VA with an IRP signature has been found, use the !irp extension to find out what driver is pending on this IRP.
13: kd> !irp 876a6168
Irp is active with 4 stacks 3 is current (= 0x876a6220)
Mdl=8975d618: No System Buffer: Thread 00000000: Irp stack trace.
cmd flg cl Device File Completion-Context
[ 0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
[ 0, 0] 0 0 00000000 00000000 00000000-00000000
Args: 00000000 00000000 00000000 00000000
>[ f, 8] 0 e1 8aa33030 88c532a0 b7bfbb72-8acf31b0 Success Error Cancel pending
\Driver\Tcpip afd!AfdRestartBufferReceiveWithUserIrp
Args: 00002000 00010020 00000000 00000000
[ e, 5] 5 0 8ac68030 894e4c28 00000000-00000000
\Driver\AFD
Args: 00002000 00000000 20000020 00000000
==>> From the stack below, TCP Chimney feature is enabled on the server, disabling it will be good step to disable that.
13: kd> dd tcpip!DisableTCPChimney L1
b7c950d0 00000000
13: kd> dd tcpip!DisableRSS L1
b7c950d4 00000000
13: kd> dd tcpip!UseDma L1
b7c95fa0 00000101
13: kd> dd tcpip!DisableTaskOffload L1
b7c95e54 00000000
Reference Microsoft KB articles:
MS09-008: Vulnerabilities in DNS and WINS Server could allow spoofing
http://support.microsoft.com/kb/962238
An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
http://support.microsoft.com/default.aspx?scid=kb;EN-US;948496
Don’t forget to leave your comments 🙂
Thanks,
Aresh