High CPU Usage After KB958644

[jason.hill]

Hi,

We installed the critical update KB958644 on a Windows Server 2008
machine 2 days ago and have experienced a massive increase in CPU
utilisation, apparently for the System process. Our performance
monitor reports typically showed the web server running in the 5-15%
CPU utilisation range for the whole day even under fairly heavy load.
For the last 2 days, the CPU graph now looks like an earthquake has
hit and we have had to reboot the server both days to reset the CPU to
normal.

Today, the graph was showing 90%+ CPU utilisation and we were not able
to RDP to the box and all web sites were non-responsive so had to get
the ISP to do a hard reboot. The graph showed the server in this 90%+
CPU usage state for several hours before we were made aware of the
problem so it was not just a temporary problem.

Has anyone else had problems with this security update? If we have
this problem again tomorrow, then I think we will have to uninstall
the update and ensure we are as secure as possible with the
workarounds in the technet article.

Thanks,

Jason

[Leythos]

In article <0e3cd5ab-0a93-45e7-8a5b-37f04b1a1b80
@e38g2000prn.googlegroups.com>, jason.hill2@gmail.com says...
> Hi,
>
> We installed the critical update KB958644 on a Windows Server 2008
> machine 2 days ago and have experienced a massive increase in CPU
> utilisation, apparently for the System process. Our performance
> monitor reports typically showed the web server running in the 5-15%
> CPU utilisation range for the whole day even under fairly heavy load.
> For the last 2 days, the CPU graph now looks like an earthquake has
> hit and we have had to reboot the server both days to reset the CPU to
> normal.
>
> Today, the graph was showing 90%+ CPU utilisation and we were not able
> to RDP to the box and all web sites were non-responsive so had to get
> the ISP to do a hard reboot. The graph showed the server in this 90%+
> CPU usage state for several hours before we were made aware of the
> problem so it was not just a temporary problem.
>
> Has anyone else had problems with this security update? If we have
> this problem again tomorrow, then I think we will have to uninstall
> the update and ensure we are as secure as possible with the
> workarounds in the technet article.

Not seen on any of our servers, but you don't mention what you're
running for a server.

--
- Igitur qui desiderat pacem, praeparet bellum.
- Calling an illegal alien an "undocumented worker" is like calling a
drug dealer an "unlicensed pharmacist"
spam999free@rrohio.com (remove 999 for proper email address)

[Jason Hill]

We are running a quad core Xeon Clovertown with 4GB RAM and Win 2008
Datacenter.

CPU usage was a bit out of control again today so we uninstalled the update.
Seems to have stabilised a little although still around 2-4X what we have
been seeing for the same traffic volumes over the last 3 months.

We have been running performance monitor reports with the System process
being tracked and it is using 75%+ of the overall CPU utilisation which
doesn't quite look right. It would account for the overall increase that we
have seen but we are not sure what is causing it.

Very coincidental that it all started happening after that update but things
are still a little weird even after we uninstalled it.

Could this be a hardware issue? Not seeing anything in the logs but maybe
there are some diags we can run to check this out?

"Leythos" wrote:

> In article <0e3cd5ab-0a93-45e7-8a5b-37f04b1a1b80
> @e38g2000prn.googlegroups.com>, jason.hill2@gmail.com says...
> > Hi,
> >
> > We installed the critical update KB958644 on a Windows Server 2008
> > machine 2 days ago and have experienced a massive increase in CPU
> > utilisation, apparently for the System process. Our performance
> > monitor reports typically showed the web server running in the 5-15%
> > CPU utilisation range for the whole day even under fairly heavy load.
> > For the last 2 days, the CPU graph now looks like an earthquake has
> > hit and we have had to reboot the server both days to reset the CPU to
> > normal.
> >
> > Today, the graph was showing 90%+ CPU utilisation and we were not able
> > to RDP to the box and all web sites were non-responsive so had to get
> > the ISP to do a hard reboot. The graph showed the server in this 90%+
> > CPU usage state for several hours before we were made aware of the
> > problem so it was not just a temporary problem.
> >
> > Has anyone else had problems with this security update? If we have
> > this problem again tomorrow, then I think we will have to uninstall
> > the update and ensure we are as secure as possible with the
> > workarounds in the technet article.
>
> Not seen on any of our servers, but you don't mention what you're
> running for a server.
>
> --
> - Igitur qui desiderat pacem, praeparet bellum.
> - Calling an illegal alien an "undocumented worker" is like calling a
> drug dealer an "unlicensed pharmacist"
> spam999free@rrohio.com (remove 999 for proper email address)
>

[Thee Chicago Wolf]

>We are running a quad core Xeon Clovertown with 4GB RAM and Win 2008
>Datacenter.
>
>CPU usage was a bit out of control again today so we uninstalled the update.
>Seems to have stabilised a little although still around 2-4X what we have
>been seeing for the same traffic volumes over the last 3 months.
>
>We have been running performance monitor reports with the System process
>being tracked and it is using 75%+ of the overall CPU utilisation which
>doesn't quite look right. It would account for the overall increase that we
>have seen but we are not sure what is causing it.
>
>Very coincidental that it all started happening after that update but things
>are still a little weird even after we uninstalled it.
>
>Could this be a hardware issue? Not seeing anything in the logs but maybe
>there are some diags we can run to check this out?

Double check and make sure the NIC driver is fully up to date. Since
the update was for netapi32.dll, it's possible some kind of change was
made that might be affecting offloading or processing to the NIC. I'm
not sure how or if 2008 server offloads any processing to the NIC but
it might and it *could* be that. I am thinking in terms of what the
SNP does for 2003 server.

Go grab Process Explorer for a more detailed look at the specific
process that is eating up your cycles. Worse case scenario, you could
open a case with MS to see if they've seen this yet.

- Thee Chicago Wolf

[Jason Hill]

Another update was released on 26 Oct that is related to performance and
seems to have resolved the problem for the time being.

http://www.microsoft.com/downloads/d...DisplayLang=en

"Thee Chicago Wolf" wrote:

> >We are running a quad core Xeon Clovertown with 4GB RAM and Win 2008
> >Datacenter.
> >
> >CPU usage was a bit out of control again today so we uninstalled the update.
> >Seems to have stabilised a little although still around 2-4X what we have
> >been seeing for the same traffic volumes over the last 3 months.
> >
> >We have been running performance monitor reports with the System process
> >being tracked and it is using 75%+ of the overall CPU utilisation which
> >doesn't quite look right. It would account for the overall increase that we
> >have seen but we are not sure what is causing it.
> >
> >Very coincidental that it all started happening after that update but things
> >are still a little weird even after we uninstalled it.
> >
> >Could this be a hardware issue? Not seeing anything in the logs but maybe
> >there are some diags we can run to check this out?
>
> Double check and make sure the NIC driver is fully up to date. Since
> the update was for netapi32.dll, it's possible some kind of change was
> made that might be affecting offloading or processing to the NIC. I'm
> not sure how or if 2008 server offloads any processing to the NIC but
> it might and it *could* be that. I am thinking in terms of what the
> SNP does for 2003 server.
>
> Go grab Process Explorer for a more detailed look at the specific
> process that is eating up your cycles. Worse case scenario, you could
> open a case with MS to see if they've seen this yet.
>
> - Thee Chicago Wolf
>

[Thee Chicago Wolf]

>Another update was released on 26 Oct that is related to performance and
>seems to have resolved the problem for the time being.
>
>http://www.microsoft.com/downloads/d...DisplayLang=en
>

Thanks for the heads up.

- Thee Chicago Wolf

[Jason Hill]

I spoke too soon. The problem still remains and requires a reboot of the
server because it becomes totally unresponsive.

"Thee Chicago Wolf" wrote:

> >Another update was released on 26 Oct that is related to performance and
> >seems to have resolved the problem for the time being.
> >
> >http://www.microsoft.com/downloads/d...DisplayLang=en
> >
>
> Thanks for the heads up.
>
> - Thee Chicago Wolf
>

[Thee Chicago Wolf]

>I spoke too soon. The problem still remains and requires a reboot of the
>server because it becomes totally unresponsive.

Did you already use Process Explorer to see what app is eating up
cycles? What about BIOS and firmware, up to date as well? Intel
Chipset drivers installed?

- Thee Chicago Wolf

[Kerry Brown]

"Jason Hill" <JasonHill@discussions.microsoft.com> wrote in message
news:4953444D-19A6-4B5B-86D0-DEA189D5B256@microsoft.com...
>I spoke too soon. The problem still remains and requires a reboot of the
> server because it becomes totally unresponsive.
>


Is the server a DNS server running other roles as well? Is it possible that
the port conflict problem talked about here may be involved?

http://support.microsoft.com/kb/956188

I've seen this happen on one server. Installing KB953230 didn't initially
cause any problems. A later update did. Figuring out which port DNS was
clobbering then reserving that port with the registry changes in the above
kb fixed the problem.

--
Kerry Brown
MS-MVP - Windows Desktop Experience: Systems Administration
http://www.vistahelp.ca/phpBB2/
http://vistahelpca.blogspot.com/

[jason.hill]

On Oct 31, 1:57 am, "Kerry Brown" <ke...@kdbNOSPAMsys-tems.c*a*m>
wrote:
> "Jason Hill" <JasonH...@discussions.microsoft.com> wrote in message
>
> news:4953444D-19A6-4B5B-86D0-DEA189D5B256@microsoft.com...
>
> >I spoke too soon. The problem still remains and requires a reboot of the
> > server because it becomes totally unresponsive.
>
> Is the server a DNS server running other roles as well? Is it possible that
> the port conflict problem talked about here may be involved?
>
> http://support.microsoft.com/kb/956188
>
> I've seen this happen on one server. Installing KB953230 didn't initially
> cause any problems. A later update did. Figuring out which port DNS was
> clobbering then reserving that port with the registry changes in the above
> kb fixed the problem.
>
> --
> Kerry Brown
> MS-MVP - Windows Desktop Experience: Systems Administrationhttp://www.vistahelp.ca/phpBB2/http://vistahelpca.blogspot.com/

We have isolated our problem to the AVG network filter driver. When
this is enabled, CPU usage is much higher than usual and when we kick
off a big download, CPU usage sky rockets.

When we disable the AVG network filter driver, CPU usage returns to
normal and does not spiral out of control when a big download is
started.

Our hosting provider has logged a support request with AVG and I will
post back any updates.

Jason

[Thee Chicago Wolf]

>We have isolated our problem to the AVG network filter driver. When
>this is enabled, CPU usage is much higher than usual and when we kick
>off a big download, CPU usage sky rockets.
>
>When we disable the AVG network filter driver, CPU usage returns to
>normal and does not spiral out of control when a big download is
>started.
>
>Our hosting provider has logged a support request with AVG and I will
>post back any updates.

Hopefully it'll be just as easy as updating to a patched version and
you guess will be good. I was almost going to suggest you try the
hotfix from the following KB article:
http://support.microsoft.com/kb/955860

- Thee Chicago Wolf

[gram]

We have had problems in connecting via remote desktop since applying the
patch.


"jason.hill" <jason.hill2@gmail.com> wrote in message
news:0e3cd5ab-0a93-45e7-8a5b-37f04b1a1b80@e38g2000prn.googlegroups.com...
> Hi,
>
> We installed the critical update KB958644 on a Windows Server 2008
> machine 2 days ago and have experienced a massive increase in CPU
> utilisation, apparently for the System process. Our performance
> monitor reports typically showed the web server running in the 5-15%
> CPU utilisation range for the whole day even under fairly heavy load.
> For the last 2 days, the CPU graph now looks like an earthquake has
> hit and we have had to reboot the server both days to reset the CPU to
> normal.
>
> Today, the graph was showing 90%+ CPU utilisation and we were not able
> to RDP to the box and all web sites were non-responsive so had to get
> the ISP to do a hard reboot. The graph showed the server in this 90%+
> CPU usage state for several hours before we were made aware of the
> problem so it was not just a temporary problem.
>
> Has anyone else had problems with this security update? If we have
> this problem again tomorrow, then I think we will have to uninstall
> the update and ensure we are as secure as possible with the
> workarounds in the technet article.
>
> Thanks,
>
> Jason