100% CPU Usage

[andyrite]

Hi

I'm having problems with my two windows 2003 servers.
During busy periods throughout the day the servers CPU utilisation
jumps up to 100%. I can take up to 10 minutes for people to log on.

The servers are connected to an iSCSI array for the storage. Both have
plenty of free space. The hardware manufacturers (eonstor) have had a
look and are happy that the array is setup correctly. Both servers
have a QLOGIC HBA connecting them to the array.

I've used process explorer when the high utilisation is happening

Here are the results


Process PID CPU Description Company Name
System Idle Process 0 0.38
Interrupts n/a 1.54 Hardware Interrupts
DPCs n/a 11.15 Deferred Procedure Calls
System 4 75.38
smss.exe 1456 Windows NT Session Manager Microsoft Corporation
csrss.exe 1536 0.77 Client Server Runtime Process Microsoft
Corporation
winlogon.exe 1576 Windows NT Logon Application Microsoft
Corporation
services.exe 1648 Services and Controller app Microsoft
Corporation
svchost.exe 1836 Generic Host Process for Win32 Services
Microsoft Corporation
wmiprvse.exe 4004 WMI Microsoft Corporation
svchost.exe 348 Generic Host Process for Win32 Services
Microsoft Corporation
SavService.exe 432 9.23 Performs virus scanning and disinfection
functions Sophos Plc
svchost.exe 880 Generic Host Process for Win32 Services
Microsoft Corporation
svchost.exe 896 Generic Host Process for Win32 Services
Microsoft Corporation
svchost.exe 944 Generic Host Process for Win32 Services
Microsoft Corporation
spoolsv.exe 1408 Spooler SubSystem App Microsoft Corporation
msdtc.exe 1480 MS DTCconsole program Microsoft Corporation
beremote.exe 1956 Backup Exec Remote Agent for Windows NT/2000
Symantec Corporation
cisvc.exe 2044 Content Index service Microsoft Corporation
cidaemon.exe 5880 Indexing Service filter daemon Microsoft
Corporation
dfssvc.exe 264 0.38 Windows NT Distributed File System Service
Microsoft Corporation
dns.exe 476 Domain Name System (DNS) Server Microsoft
Corporation
svchost.exe 572 Generic Host Process for Win32 Services
Microsoft Corporation
FCM.exe 660 Ranger Suite Sentinel Products Ltd
FCR.exe 856 Ranger Suite Sentinel Products Ltd
ismserv.exe 924 Windows NT Intersite Messaging Service Microsoft
Corporation
ntfrs.exe 1192 File Replication Service Microsoft Corporation
portmap.exe 1276 ONC/RPC Portmap service registration helper
svchost.exe 2008 Generic Host Process for Win32 Services
Microsoft Corporation
RngrSvr.exe 2192 Ranger Suite Sentinel Products Ltd
SAVAdminService.exe 2208 Sophos Administrator Service Sophos Plc
ManagementAgentNT.exe 2272 Sophos Agent Sophos Plc
ALsvc.exe 2488 Sophos AutoUpdate Service. Sophos Plc
RouterNT.exe 2540 Sophos Message Router Sophos Plc
smhstart.exe 2712 HP System Management Homepage Service Hewlett-
Packard Company
hpsmhd.exe 2860 HP System Management Homepage Hewlett-Packard
Company
hpsmhd.exe 3988 HP System Management Homepage Hewlett-Packard
Company
CpqRcmc.exe 2788 HP ProLiant Remote Monitor Service Hewlett-
Packard Company
iqlremote.exe 2888
RgrAccss.exe 2980 Ranger Suite Sentinel Products Ltd
RgrLic.exe 3008 Ranger Suite Sentinel Products Ltd
sysdown.exe 3100 HP ProLiant System Shutdown Service Compaq
Computer Corporation
svchost.exe 3136 Generic Host Process for Win32 Services
Microsoft Corporation
dmadmin.exe 3336 Logical Disk Manager Adminstrative Service
Microsoft Corporation
lsass.exe 1660 0.77 LSA Shell Microsoft Corporation
explorer.exe 3448 Windows Explorer Microsoft Corporation
cpqteam.exe 3568 CPQTEAM.EXE (32-bit) Hewlett-Packard Company
LanRanger.exe 3636 Ranger Suite Sentinel Products Ltd
RangerToolbar.exe 3652 Ranger Suite Sentinel Products Ltd
ALMon.exe 3668 Component to show AutoUpdate's GUI elements. Sophos
Plc
taskmgr.exe 4184 Windows TaskManager Microsoft Corporation
GhostSrv.exe 5964 Symantec GhostCast Server for Windows Symantec
Corporation
procexp.exe 4856 0.38 Sysinternals Process Explorer Sysinternals

Both servers are running active directory. There are approx 500
workstations attached to these servers. (Dual 3.2Ghz, 5GB RAM.)

Anybody got any ideas on this? I've searched everywhere and found no
answer

[Herb Martin]

"andyrite" <andy@ndonline.ac.uk> wrote in message
news:1173095192.664025.321340@30g2000cwc.googlegroups.com...
> Hi
>
> I'm having problems with my two windows 2003 servers.
> During busy periods throughout the day the servers CPU utilisation
> jumps up to 100%. I can take up to 10 minutes for people to log on.
>
> The servers are connected to an iSCSI array for the storage. Both have
> plenty of free space. The hardware manufacturers (eonstor) have had a
> look and are happy that the array is setup correctly. Both servers
> have a QLOGIC HBA connecting them to the array.
>
> I've used process explorer when the high utilisation is happening

What is running on these servers (out of the ordinary)? Are they DCs?
Do they have other services running beyond the standard? What about
AV software (much of that is notorious for CPU usage)? etc?

What user processes are running? Which seem to be unresponsive?

Sort by CPU USAGE (Task Manager) or other tool and try to determine
which process(es) is(are) the culprits....

Many times a server will stop responding due to IE (or Firefox), or just due
to Windows Explorer getting hosed. Such process can be killed with
TaskKill.exe or Task Manager (TaskKill is better for more control and
capability).

Even Svchost.exe processes (with HIGHER PIDs) can usually be killed.
(Early PIDs will sometimes take down the system.)



> Here are the results

It's virtually impossible to read that output without proper formatting and
by not sorting on CPU USAGE it is even harder.


> Process PID CPU Description Company Name
> System Idle Process 0 0.38

Such problems are occasionally difficult to isolate so you may need to
experiment with selective Task KILLing.

--
Herb Martin, MCSE, MVP
http://www.LearnQuick.Com
(phone on web site)

[andyrite]

Hi thansk for the quick respone.

It is the system process that it taking up the cpu time. When i narrow
it down in process exporer srv.sys has a few hundred threads open and
taking all the cpu time.
Nothing else is seems to be taking up the cpu time.
Both servers are domain controllers.
Both are running sophos anti-virus 6.5.3
Both are running ranger suite. http://www.rangersuite.com/
One is running as a print server

[Herb Martin]

"andyrite" <andy@ndonline.ac.uk> wrote in message
news:1173099325.460604.102330@t69g2000cwt.googlegroups.com...
> Hi thansk for the quick respone.
>
> It is the system process that it taking up the cpu time. When i narrow
> it down in process exporer srv.sys has a few hundred threads open and
> taking all the cpu time.

How many file/print clients? How active?

> Nothing else is seems to be taking up the cpu time.
> Both servers are domain controllers.
> Both are running sophos anti-virus 6.5.3

Is this new or upgraded? Most people don't have this problem so
I would look here first....

> Both are running ranger suite. http://www.rangersuite.com/

Ditto. new or upgraded etc?

> One is running as a print server

Look especially for things that have recently changed -- no matter
how innocuous seeming.

How are the RAM etc resources?

Which apps have the highest Pages per Second? (Sometimes that
will drive CPU Usage higher.)

What about checking the System Log for errors? Hardware interupt
problems may cause such problems....

--
Herb Martin, MCSE, MVP
http://www.LearnQuick.Com
(phone on web site)

[andyrite]

> How many file/print clients? How active?
Theres about 500 machines connected. Each user has there my documents
mapped to a server drive and i'm using roaming profiles. The mostly
use word documents.
There are about 30 printers setup.

Sophos has been installed for a while. If i disable it on the server
it has no effect.
I have been in contact with ranger and they insist that it isn't there
software that is causing the problem.

The ram is fine. each server is only using 768mb

> Which apps have the highest Pages per Second? (Sometimes that
> will drive CPU Usage higher.)

How do i do that?

> What about checking the System Log for errors? Hardware interupt
> problems may cause such problems....

The error log is fine.

The only error is this when the server is at 100%

Event Type: Warning
Event Source: Srv
Event Category: None
Event ID: 2012
Date: 29-06-2006
Time: 12:39:42
User: N/A
Computer: "SERVER NAME"
Description:
While transmitting or receiving data, the server encountered a network
error.
Occassional errors are expected, but large amounts of these indicate a
possible error in your
network configuration. The error status code is contained within the
returned data
(formatted as Words) and may point you towards the problem.

[Herb Martin]

"andyrite" <andy@ndonline.ac.uk> wrote in message
news:1173176443.235663.196790@v33g2000cwv.googlegroups.com...
>
>> How many file/print clients? How active?
> Theres about 500 machines connected. Each user has there my documents
> mapped to a server drive and i'm using roaming profiles. The mostly
> use word documents.
> There are about 30 printers setup.
>
> Sophos has been installed for a while. If i disable it on the server
> it has no effect.
> I have been in contact with ranger and they insist that it isn't there
> software that is causing the problem.
>
> The ram is fine. each server is only using 768mb
>
>> Which apps have the highest Pages per Second? (Sometimes that
>> will drive CPU Usage higher.)
>
> How do i do that?

Task manager is the easier way -- PerfMon is the more difficult (for apps)
but gives a bit more control and shows a graph which can be usefu.

>> What about checking the System Log for errors? Hardware interupt
>> problems may cause such problems....
>
> The error log is fine.


> The only error is this when the server is at 100%
>
> Event Type: Warning
> Event Source: Srv
> Event Category: None
> Event ID: 2012
> Date: 29-06-2006
> Time: 12:39:42
> User: N/A
> Computer: "SERVER NAME"
> Description:
> While transmitting or receiving data, the server encountered a network
> error.
> Occassional errors are expected, but large amounts of these indicate a
> possible error in your
> network configuration. The error status code is contained within the
> returned data
> (formatted as Words) and may point you towards the problem.

It's worth checking all hardware and the NIC drivers. ESPECIALLY the
drop cable and any switch/concentrator port.

Drop cables cause problems surprisingly often and get very little attention.

--
Herb Martin, MCSE, MVP
http://www.LearnQuick.Com
(phone on web site)