Windows 2003 WSUS Clients Hanging at "Windows Is Shutting Down..." After Scheduled Patch Install and Auto-Reboot

I have seen this topic posted several times but it looks
like most of them were never answered or abandoned some time ago, so I
thought I would post a new thread. We have 150+ Windows Servers that
we patch in groups of 25-30 whenever updates are released by
Microsoft. We are currently using WSUS 2.0 and GPO's to automate the
patching process and scheduling of installs/reboots for these servers.
Over the past several months, we have seen that 10-15 servers in each
of these groups will hang at the "Windows is shutting down..." screen
after applying updates and attempting to auto-reboot. The affected
servers are random at best as far as we can tell (they are usually
different each patch interval).

We have antivirus scanning policies in place to prevent real-time
scanning of the following directories on every server:
C:\Pagefile.sys
C:\System Volume Information All files and sub-directories
C:\Windows\Microsoft.Net All files and sub-directories
C:\Windows\SoftwareDistribution All files and sub-directories
C:\Windows\System32\config All files and sub-directories
C:\Windows\System32\CatRoot2 All files and sub-directories

We are currently using CA eTrust Antivirus, and these problems started
occurring after this software had been in place for a couple of
years.

We have developed a means (sort of) of being able to detect which
servers are in this "stuck" state by looking at our WSUS reports on
Monday mornings after a scheduled Saturday patching. Any servers which
still appear to have "needed" updates we attempt to RDP into. When we
do that on these servers, we receive only a gray screen and never any
logon windows. However, as soon as we touch the server via KVM or
Avocent, it will instantly complete the shut down process and reboot
normally. But if we don't do this, they will remain in this "stuck"
state indefinitely.

We have opened a case with Microsoft, but they have not been able to
determine a cause and/or solution as of yet. They recommended that we
disable all screen savers on our servers as an attempted resolution,
but there we run into a different problem. On a known-good GPO,
whenever we attempt to turn on the settings to disable screen savers
on our servers the settings will never take effect. It's as if the
screen saver settings portion of the GPO are non-existent. Apparently
this is another documented issue that I've seen several posts on with
no resolution.

If anyone has any thoughts and or information on how to resolve the
servers that get stuck at "Windows is shutting down..." issue, it
would be immensely appreciated.

Actually, the question has been answered a gazillion times. Those without
resolutions is because they've been unwilling or unable to find the actual
cause, which has *NEVER* been anything associated with the Windows Server
Update Services subsystems.

[a] There's an update for Windows 2000 systems demonstrating this behavior.
Find it and install it.

[b] Otherwise, your AV inquiry is on the right track.

Well, all of the affected servers are Windows 2003 Servers - 2003 SP1,
SP2, x64, x32, Enterprise and Standard we have seen affected by this
behavior.

None of our Windows 2000 machines (the very few we have left), have
been affected.

Anyone else have thoughts/ideas? I run through an automated patch
install/reboot on a server last night and it also hung at the "Windows
is Shutting Down..." screen. When I attempted to launch Remote Task
Manager to see what processes were running, it said the RPC Server was
unavailable. This kept me from looking at the Services snap-in and the
Event Viewer snap-in when I attempted to connect to them remotely.

Again, the instant I touched it via our Avocent console (KVM) it shut
down and rebooted normally. But it will stay "stuck" like that
indefinitely until someone touches it via console or KVM.

After it rebooted I looked at the Event Log and this is the last event
recorded prior to the reboot:

"The process winlogon.exe has initiated the restart of computer
SIEDFS0 on behalf of user NT AUTHORITY\SYSTEM for the following
reason: No title for this reason could be found
Reason Code: 0x80020002
Shutdown Type: restart
Comment: "

Microsoft says this might be related to screen savers....??

I've seen this infrequently on our Windows XP machines but never on a server.
The usual scapegoat is anti-virus software; is there any installed?

Gets a bit tricky, but according to the documentation a service that doesn't
accept the shutdown control should continue to run in this scenario. So if you
have any Windows-savvy programmers on hand it should be possible to write a
service that keeps a dialog box open on the Winlogon desktop and launches a
command-line window on demand.

Does the problem happen every time one of the servers is restarted? One of the
reasons it's been hard for me to diagnose what's going on is that it only occurs
very rarely.

An organization I am working with is experiencing the same issue with
both windows 2003 and windows 2000 boxes. We have been trying various
ways to replicate the issue but cannot seem to find a way to make it
happen consistently. Microsoft recommended we update the WSUS client
to version 3 or disable the screen saver. Since we can't consistently
reproduce the problem we can not test the solutions they have provided
us. Have you had any luck and getting the problem to appear
consistently? In our case usually a reboot fixes the problem and we
can't get it to reoccur.

Yes - we are using CA's eTrust Antivirus 8. The problem is that we
have been using this product for a few years now, and have not had
this problem up until several months ago. I do not believe an upgrade
has taken place, but I can't rule that out until I can talk to our
Data Security team. But I do know that as far as scanning exclusions
go, we are excluding the typical Windows system folders from realtime
scans, as well as any Windows Update related folders.

I downloaded a remote task manager utility to see if I could pull up a
list of running tasks on one of these servers when it became "stuck",
but at that point the RPC server service is already stopped, so I
can't connect with it; nor pull up Event Viewer remotely via MMC.

It doesn't appear each time a server is restarted. We only see the
issue when we activate our GPO to download and install updates
automatically. When the auto-install is done, the server tries to
restart but gets "stuck" until we touch it with a KVM/console. It's
always a different combination of servers.... some are good one week,
next week they get stuck

We have not been able to reproduce the issue consistently. As best as
I can tell, it appears to be random. Microsoft PSS also suggested that
we update to the 3.0 client as well as disable screen savers. I
manually modified the Local Security Policy on 45 servers this past
weekend to disable screen savers, and we still saw the issue on a
handful of them.

As I mentioned before, it seems to be pretty random and is occuring on
Windows 2000 Server, Windows 2003 SP1, Windows 2003 SP2, Windows 2003
x64, as well as both Standard and Enterprise versions of 2003.

I haven't come up with a way to deploy the AU 3.0 client to all of our
servers, but since the screen saver disabling doesn't appear to have
any effect this will likely be my next move. Our next patch window
won't occur until July 10th, so if I can't find a way to auto-deploy
the 3.0 client I will manually install it on our set of servers that
will be patched first following the next "Patch Tuesday" release. I'll
keep you posted on the results of that. If you happen to notice
anything on your end as well, please let me know! Thanks!

I originally thought of doing that, but then remembered that we
are seeing this issue on VM's as well as physical servers; so I don't
think it is a KVM related issue. However, the same problem exists on
the VM's: if we attempt to RDP into a "stuck" server, all we get is a
blank gray screen. We have to open up VMWare Virtual Infrastructure
Client, navigate to the VM in question, and click the 'Console' tab.
As soon as we touch the server using this 'Console' tab, the server
will complete its shutdown and reboot.

I appreciate the info Gus and will look into that.... outside of our
VM servers, all of our hardware is Dell PowerEdge 2600 - 2950 using
Broadcom NetExtreme NICs. I will look into the TOE disabling to see if
we even have that installed as an option with our NICs.

[stfconsulting]

Did anyone ever figure this out? Working with MSFT now and they have no idea. Here is the run down of our experience with the problem:

We see this problem on some of our Windows 2003 servers when ever they reboot for a Scheduled Update. The server will complete the shutdown process and log the shutdown event 6006. When we use the Remote Access Controller to gain access to the server all we see is a grey screen. We have to powercycle the server to bring it back online. We have worked with Microsoft on the issue through the Private News Groups and here is their response. The problem is really annoying because there are issues that occur on the client side with Outlook when they have not been able to talk to the server for a few hours. Some of our customers that have BES also need to have that server restarted because it lost contact with Exchange for so long. We have tried added the SYSTEM account to the User Rights Assignment and that did not help. I guess we can experiment with the screen saver settings next. None of the other suggestions apply to us. The log that you see below is from a brand new Poweredge 2900 with the firmware fully up to date and the patches up to date (SP2).

Issue:
Servers cannot shutdown properly after Windows Update finishes installation

Suggestion:
We found that System account was not configured with Shut Down The System rights. We used the Group Policy to assign the rights to the System account.

Additional information:
Below is some known shutdown issues:
1) Do not use the "Do not require CTRL+ALT+DEL" policy as this is known to cause shutdown issues. The DisableCAD registry key and the local policy will be deleted by the .REG file above, but the policy must be manually checked.
The policy is under \Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options\Interactive logon: Do not require CTRL+ALT+DEL

2) Having a default screen saver. Disable the default screen saver and just use the monitor power save feature. Without the screen saver, the power save will still work, and is the best way to prevent burn-in. The registry settings above will automatically remove the screensaver for you.

3) The System user account must have the SEShutdown user right for Automatic Updates to reboot the server.
The policy is understand \Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment\Shut down the system.

Please note: If a 3rd party service or process is being used to shutdown the machine, make sure the user account for it has this privilege.

4) Citrix servers may not shutdown properly due to orphaned winlogon and csrss processes. In this case, we need to contact Citrix for further suggestions/solutions.

It is my pleasure to work with you on this issue and if you have any concerns in the future, please feel free to post back.

We currently have this same problem occuring now...

your message thread was interesting but it actually didn't say how to manage to restart the windows server 2003, thus beeing able to login to the server again...

how can we fix thiw problem when the server is stuck, ctrl-alt-del doesn't have any reactions?

[vertaylor]

We've experienced this same issue with our Windows 2003 servers. After scheduled updates from WSUS, several of them would initiate restart and just hang. It would then require someone going to the physical console to move a mouse to allow the server to complete the restart.

After alot of guessing/testing we determined is that it was the Windows logon screensaver that was somehow interfering with the restart. After disabling the logon screensaver on our servers, we have not had this issue again.

Steps to disable the screensaver are on Microsoft's site: http://support.microsoft.com/kb/185348