Issue failover MS SQL cluster

[defstar]

Windows 2003 server running SQL 2005
Failover does not work: not manually and probably also not by shutting down the active Node

I have looked at the cluster log, although there are errors i cannot figure out what's going on.
I have filtered all the lines that contain the word 'error': During this time i first tried a manual failover and then i shut down the active node.

17:27:25.187 WARN Network Name <Cluster Name>: Unable to read ResourceData parameter, error=2
17:27:25.187 WARN Network Name <Cluster Name>: Unable to read CreatingDC parameter, error=2
17:27:25.203 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read ResourceData parameter, error=2
17:27:25.203 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read CreatingDC parameter, error=2
17:27:25.281 WARN Network Name <VDISPATCH Network Name>: Unable to read ResourceData parameter, error=2
17:27:25.281 WARN Network Name <VDISPATCH Network Name>: Unable to read CreatingDC parameter, error=2
17:33:31.029 ERR Volume Manager Disk Group <quorum_sqlcluster>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
17:36:12.562 WARN Network Name <VDISPATCH Network Name>: Unable to read ResourceData parameter, error=2
17:36:12.562 WARN Network Name <VDISPATCH Network Name>: Unable to read CreatingDC parameter, error=2
17:36:12.577 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read ResourceData parameter, error=2
17:36:12.577 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read CreatingDC parameter, error=2
17:36:43.401 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
17:36:43.401 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
17:36:50.915 ERR Volume Manager Disk Group <quorum_sqlcluster>: Dg-Guid is Null000008b8.000008d4::2009/06/25-17:36:58.429 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
17:37:01.519 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
17:37:01.519 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
17:37:13.490 ERR Volume Manager Disk Group <quorum_sqlcluster>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
17:37:35.599 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
17:37:35.599 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
17:37:43.161 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
17:37:46.251 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
17:37:46.251 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
17:38:01.280 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
17:38:01.280 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
17:38:08.810 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
17:38:11.899 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
17:38:11.899 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
17:38:26.960 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
17:38:26.960 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
17:38:34.491 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
17:38:37.580 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
17:38:37.580 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
17:38:37.580 INFO [FM] FmpRmOfflineResource: RmOffline() for b803a691-6e9b-4901-b6c1-a83ae79870dd returned error 997
17:38:37.580 INFO [FM] FmpRmOfflineResource: RmOffline() for 0fdef525-a65b-4d3b-96ad-dfcfead832c8 returned error 997
17:42:01.990 ERR Volume Manager Disk Group <quorum_sqlcluster>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
17:42:24.188 ERR Volume Manager Disk Group <quorum_sqlcluster>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
17:42:24.188 ERR [FM] Failed to arbitrate quorum resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff, error 5006.
17:42:24.188 ERR [RGP] Node 1: REGROUP ERROR: arbitration failed.
17:42:24.188 ERR [NM] Halting this node due to membership or communications error. Halt code = 1000
17:42:24.204 ERR [CS] Halting this node to prevent an inconsistency within the cluster. Error status = 5892
17:43:59.593 ERR [FM] Failed to get quorum disk signature, error 2.
17:44:21.827 ERR Volume Manager Disk Group <quorum_sqlcluster>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
17:44:21.827 ERR [FM] FmGetQuorumResource failed, error 5006.
17:46:27.135 WARN Network Name <Cluster Name>: Unable to read ResourceData parameter, error=2
17:46:27.135 WARN Network Name <Cluster Name>: Unable to read CreatingDC parameter, error=2
17:46:27.151 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read ResourceData parameter, error=2
17:46:27.151 WARN Network Name <SQL Network Name(sqlcluster)>: Unable to read CreatingDC parameter, error=2
17:46:27.214 WARN Network Name <VDISPATCH Network Name>: Unable to read ResourceData parameter, error=2
17:46:27.214 WARN Network Name <VDISPATCH Network Name>: Unable to read CreatingDC parameter, error=2

Hopefully someone has an idea what's going on.

[RockOn]

Before moving the group, the Resource Manager sends an Offline request to all resource DLLs of the failing group. In this case, the SQL Server and SQL Executive resource DLLs do not handle this situation correctly and return error code 160 back to the Resource Manager. Starting with Windows NT 4.0 Service Pack 4, MSCS no longer ignores this error code and thus fails to move the resource group.

A supported fix is now available from Microsoft: ftp://ftp.microsoft.com/bussys/sql/transfer/sql421i.exe for Intel systems and ftp://ftp.microsoft.com/bussys/sql/transfer/sql421a.exe for Alpha systems. A password is required to use these files; the password is 6.50.421.

[defstar]

thanks RockOn, but....

This fix seems to be for windows NT 4.0:
http://support.microsoft.com/kb/216674

There is no error code 160 either as far as i can see.

I can download these files wihtout using a password btw
but it is not the solution i am afraid

[defstar]

I have filtered the log for the passages with what seems to be the most useful information: (changed servernames/IP addresses)

000004c8.00000780::2009/06/25-17:33:11.711 INFO [FM] FmsTakeGroupRequest: To take group '1055be19-08c2-40bf-9a0c-934d58ffc9d3'.
000004c8.00000780::2009/06/25-17:33:11.711 INFO [FM] FmpTakeGroupRequest: To take group '1055be19-08c2-40bf-9a0c-934d58ffc9d3'.
000004c8.00000780::2009/06/25-17:33:11.711 INFO [MM] MmSetQuorumOwner(1,1), old owner 0.
000008b8.000008d4::2009/06/25-17:33:11.725 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: >>> Entering LDM_RESArbitrate
000008b8.000005b8::2009/06/25-17:33:11.725 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: >>> Entering LDM_RESArbitrateThread
000008b8.000005b8::2009/06/25-17:33:18.278 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000008b8.000005b8::2009/06/25-17:33:24.654 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000008b8.000005b8::2009/06/25-17:33:24.654 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: Calling ArbitrateDg API
000008b8.000005b8::2009/06/25-17:33:31.029 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000008b8.000005b8::2009/06/25-17:33:31.029 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000008b8.000008d4::2009/06/25-17:33:31.043 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
000004c8.00000780::2009/06/25-17:33:31.043 INFO [MM] MmSetQuorumOwner(0,0), old owner 1.
000004c8.00000780::2009/06/25-17:33:31.043 INFO [FM] FmpTakeGroupRequest: MM did not select local node 1 as the arbitration winner, Status 5006
000004c8.00000780::2009/06/25-17:33:31.043 INFO [FM] FmpTakeGroupRequest: Exit for group <1055be19-08c2-40bf-9a0c-934d58ffc9d3>, Status = 1237...


000004c8.0000077c::2009/06/25-17:36:12.562 INFO [GUM] GumSendUpdate: queuing update type 0 context 8
000008b8.00000b0c::2009/06/25-17:36:12.562 INFO Network Name <VDISPATCH Network Name>: Bringing resource online...
000008b8.00000b0c::2009/06/25-17:36:12.562 WARN Network Name <VDISPATCH Network Name>: Unable to read ResourceData parameter, error=2
000004c8.0000077c::2009/06/25-17:36:12.562 INFO [GUM] GumSendUpdate: Dispatching seq 30792 type 0 context 8 to node 1
000008b8.00000b0c::2009/06/25-17:36:12.562 WARN Network Name <VDISPATCH Network Name>: Unable to read CreatingDC parameter, error=2
000004c8.0000077c::2009/06/25-17:36:12.562 INFO [GUM] GumSendUpdate: completed update seq 30792 type 0 context 8


000004c8.0000077c::2009/06/25-17:36:12.577 INFO [CP] CppResourceNotify for resource SQL Network Name(SQLSERVER)
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [FM] FmpRmOnlineResource: called InterlockedIncrement on gdwQuoBlockingResources for resource b803a691-6e9b-4901-b6c1-a83ae79870dd
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [GUM] GumSendUpdate: queuing update type 0 context 8
000008b8.00000db4::2009/06/25-17:36:12.577 INFO Network Name <SQL Network Name(SQLSERVER)>: Bringing resource online...
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [GUM] GumSendUpdate: Dispatching seq 30796 type 0 context 8 to node 1
000008b8.00000db4::2009/06/25-17:36:12.577 WARN Network Name <SQL Network Name(SQLSERVER)>: Unable to read ResourceData parameter, error=2
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [GUM] GumSendUpdate: completed update seq 30796 type 0 context 8
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [FM] FmpPropagateResourceState: resource b803a691-6e9b-4901-b6c1-a83ae79870dd pending event.
000008b8.00000db4::2009/06/25-17:36:12.577 WARN Network Name <SQL Network Name(SQLSERVER)>: Unable to read CreatingDC parameter, error=2
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [FM] FmpRmOnlineResource: Resource b803a691-6e9b-4901-b6c1-a83ae79870dd pending
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [FM] FmpRmOnlineResource: Returning. Resource b803a691-6e9b-4901-b6c1-a83ae79870dd, state 129, status 997.
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [FM] FmpOnlineWaitingTree, online for resource b803a691-6e9b-4901-b6c1-a83ae79870dd returned = 997.
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [GUM] GumSendUpdate: queuing update type 0 context 11
000004c8.0000077c::2009/06/25-17:36:12.577 INFO [GUM] GumSendUpdate: Dispatching seq 30797 type 0 context 11 to node 1


000004c8.0000091c::2009/06/25-17:36:35.484 INFO [GUM] s_GumUpdateNode: dispatching seq 30821 type 0 context 11
000004c8.0000091c::2009/06/25-17:36:35.484 INFO [GUM] s_GumUpdateNode: completed update seq 30821 type 0 context 11
000008b8.00000c18::2009/06/25-17:36:35.886 ERR Volume Manager Disk Group <SQL Disk S:>: Dg-Guid is Null000008b8.00000c18::2009/06/25-17:36:35.886 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: Calling OnlineDg API
000008b8.00000b0c::2009/06/25-17:36:36.079 INFO Network Name <VDISPATCH Network Name>: Registered workstation name VDISPATCH on transport \Device\NetBt_If1.
000008b8.00000b0c::2009/06/25-17:36:36.079 INFO Network Name <VDISPATCH Network Name>: Network Name VDISPATCH is now online


000004c8.00000780::2009/06/25-17:36:36.208 INFO [GUM] s_GumUpdateNode: completed update seq 30828 type 0 context 13
000004c8.0000091c::2009/06/25-17:36:36.208 INFO [FM] FmsTakeGroupRequest: To take group '1055be19-08c2-40bf-9a0c-934d58ffc9d3'.
000004c8.0000091c::2009/06/25-17:36:36.208 INFO [FM] FmpTakeGroupRequest: To take group '1055be19-08c2-40bf-9a0c-934d58ffc9d3'.
000004c8.0000091c::2009/06/25-17:36:36.208 INFO [MM] MmSetQuorumOwner(1,1), old owner 0.
000008b8.000005b4::2009/06/25-17:36:36.208 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: >>> Entering LDM_RESArbitrate
000008b8.00000eb4::2009/06/25-17:36:36.208 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: >>> Entering LDM_RESArbitrateThread
000008b8.00000c18::2009/06/25-17:36:43.401 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
000008b8.00000c18::2009/06/25-17:36:43.401 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
000008b8.00000c18::2009/06/25-17:36:43.401 INFO [RM] RmpSetResourceStatus, Posting state 4 notification for resource <SQL Disk S:>
000008b8.00000c18::2009/06/25-17:36:43.401 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: <<< Exiting LDM_RESOnlineThread
000004c8.000008cc::2009/06/25-17:36:43.401 INFO [FM] NotifyCallBackRoutine: enqueuing event
000004c8.0000017c::2009/06/25-17:36:43.401 INFO [CP] CppResourceNotify for resource SQL Disk S:


000004c8.0000077c::2009/06/25-17:36:43.417 INFO [FM] TerminateResource: c6e3bd0a-7066-4076-adc8-cd01d00be670 depends on 8f1f4232-ef39-42d6-9752-0ac8c6997463. Terminating first
000004c8.0000077c::2009/06/25-17:36:43.417 INFO [FM] TerminateResource: a3aa013c-c5fe-4729-b5c7-87c9043f5cb8 depends on 8f1f4232-ef39-42d6-9752-0ac8c6997463. Terminating first
000008b8.000008d4::2009/06/25-17:36:43.417 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESTerminate: >>> Entering LDM_RESTerminate
000008b8.000008d4::2009/06/25-17:36:43.417 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESterminate: Disk group is not offline - calling LDM_RESDoTerminate. State = 4
000008b8.000008d4::2009/06/25-17:36:43.417 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: >>> Entering LDM_RESDoTerminate
000008b8.00000eb4::2009/06/25-17:36:50.915 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000008b8.000008d4::2009/06/25-17:36:58.429 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
000008b8.000008d4::2009/06/25-17:37:01.519 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
000008b8.000008d4::2009/06/25-17:37:01.519 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
000008b8.000008d4::2009/06/25-17:37:01.519 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: <<< Exiting LDM_RESDoTerminate
000008b8.000008d4::2009/06/25-17:37:01.519 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESTerminate: <<< Exiting LDM_RESTerminate
000004c8.0000077c::2009/06/25-17:37:01.519 INFO [CP] CppResourceNotify for resource SQL Disk S:
000004c8.0000077c::2009/06/25-17:37:01.519 INFO [FM] RmTerminateResource: 8f1f4232-ef39-42d6-9752-0ac8c6997463 is now offline
000004c8.0000077c::2009/06/25-17:37:01.519 INFO [FM] RestartResourceTree, Restart resource 8f1f4232-ef39-42d6-9752-0ac8c6997463


000004c8.0000077c::2009/06/25-17:37:12.847 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:37:13.361 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000008b8.00000eb4::2009/06/25-17:37:13.490 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000008b8.00000eb4::2009/06/25-17:37:13.490 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000008b8.000005b4::2009/06/25-17:37:13.490 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
000004c8.0000091c::2009/06/25-17:37:13.490 INFO [MM] MmSetQuorumOwner(0,0), old owner 1.
000004c8.0000091c::2009/06/25-17:37:13.490 INFO [FM] FmpTakeGroupRequest: MM did not select local node 1 as the arbitration winner, Status 5006
000004c8.0000091c::2009/06/25-17:37:13.490 INFO [FM] FmpTakeGroupRequest: Exit for group <1055be19-08c2-40bf-9a0c-934d58ffc9d3>, Status = 1237...
000004c8.00000654::2009/06/25-17:37:13.538 INFO [GUM] s_GumUpdateNode: dispatching seq 30830 type 0 context 13


000004c8.00000654::2009/06/25-17:37:39.139 INFO [FM] Gum update resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3, state 2, current state 2.
000004c8.00000654::2009/06/25-17:37:39.139 INFO [GUM] s_GumUpdateNode: completed update seq 30859 type 0 context 8
000004c8.00000780::2009/06/25-17:37:39.139 INFO [GUM] s_GumUpdateNode: dispatching seq 30860 type 0 context 11
000004c8.00000780::2009/06/25-17:37:39.139 INFO [GUM] s_GumUpdateNode: completed update seq 30860 type 0 context 11
000008b8.000008d4::2009/06/25-17:37:43.161 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
000008b8.000008d4::2009/06/25-17:37:46.251 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
000008b8.000008d4::2009/06/25-17:37:46.251 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
000008b8.000008d4::2009/06/25-17:37:46.251 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: <<< Exiting LDM_RESDoTerminate
000008b8.000008d4::2009/06/25-17:37:46.251 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESTerminate: <<< Exiting LDM_RESTerminate
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [CP] CppResourceNotify for resource SQL Disk S:
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [FM] RmTerminateResource: 8f1f4232-ef39-42d6-9752-0ac8c6997463 is now offline
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [FM] RestartResourceTree, Restart resource 8f1f4232-ef39-42d6-9752-0ac8c6997463

000004c8.0000077c::2009/06/25-17:37:46.251 INFO [GUM] GumSendUpdate: completed update seq 30866 type 0 context 11
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [GUM] GumSendUpdate: queuing update type 0 context 11
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [GUM] GumSendUpdate: Dispatching seq 30867 type 0 context 11 to node 1
000004c8.0000077c::2009/06/25-17:37:46.251 INFO [GUM] GumSendUpdate: completed update seq 30867 type 0 context 11
000008b8.000007f8::2009/06/25-17:37:53.765 ERR Volume Manager Disk Group <SQL Disk S:>: Dg-Guid is Null000008b8.000007f8::2009/06/25-17:37:53.765 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: Calling OnlineDg API
000008b8.000007f8::2009/06/25-17:38:01.280 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
000008b8.000007f8::2009/06/25-17:38:01.280 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
000008b8.000007f8::2009/06/25-17:38:01.280 INFO [RM] RmpSetResourceStatus, Posting state 4 notification for resource <SQL Disk S:>
000008b8.000007f8::2009/06/25-17:38:01.280 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: <<< Exiting LDM_RESOnlineThread
000004c8.000008cc::2009/06/25-17:38:01.280 INFO [FM] NotifyCallBackRoutine: enqueuing event
000004c8.0000017c::2009/06/25-17:38:01.280 INFO [CP] CppResourceNotify for resource SQL Disk S:
000004c8.0000017c::2009/06/25-17:38:01.280 INFO [FM] FmpRmDoHandleCriticalResourceStateChange: call InterlockedDecrement on gdwQuoBlockingResources, Resource 8f1f4232-ef39-42d6-9752-0ac8c6997463
000004c8.0000077c::2009/06/25-17:38:01.280 WARN [FM] FmpHandleResourceTransition: Resource Name = 8f1f4232-ef39-42d6-9752-0ac8c6997463 [SQL Disk S:] old state=129 new state=4
000004c8.0000077c::2009/06/25-17:38:01.280 WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
000004c8.0000077c::2009/06/25-17:38:01.280 INFO [GUM] GumSendUpdate: queuing update type 0 context 8
000004c8.0000077c::2009/06/25-17:38:01.280 INFO [GUM] GumSendUpdate: Dispatching seq 30868 type 0 context 8 to node 1


000004c8.0000077c::2009/06/25-17:38:01.280 INFO [FM] FmpPropagateResourceState: resource c6e3bd0a-7066-4076-adc8-cd01d00be670 offline event.
000004c8.0000077c::2009/06/25-17:38:01.280 INFO [FM] RmTerminateResource: c6e3bd0a-7066-4076-adc8-cd01d00be670 is now offline
000004c8.0000077c::2009/06/25-17:38:01.280 INFO [FM] TerminateResource: a3aa013c-c5fe-4729-b5c7-87c9043f5cb8 depends on 8f1f4232-ef39-42d6-9752-0ac8c6997463. Terminating first
000008b8.00000c14::2009/06/25-17:38:01.280 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESTerminate: >>> Entering LDM_RESTerminate
000008b8.00000c14::2009/06/25-17:38:01.280 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESterminate: Disk group is not offline - calling LDM_RESDoTerminate. State = 4
000008b8.00000c14::2009/06/25-17:38:01.296 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: >>> Entering LDM_RESDoTerminate
000008b8.00000c14::2009/06/25-17:38:08.810 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: stop res failed, error. 0 1
000008b8.00000c14::2009/06/25-17:38:11.899 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: vxio stop res, error. 24
000008b8.00000c14::2009/06/25-17:38:11.899 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: *** Unable to terminate resource, error 24
000008b8.00000c14::2009/06/25-17:38:11.899 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESDoTerminate: <<< Exiting LDM_RESDoTerminate
000008b8.00000c14::2009/06/25-17:38:11.899 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESTerminate: <<< Exiting LDM_RESTerminate
000004c8.0000077c::2009/06/25-17:38:11.899 INFO [CP] CppResourceNotify for resource SQL Disk S:
000004c8.0000077c::2009/06/25-17:38:11.899 INFO [FM] RmTerminateResource: 8f1f4232-ef39-42d6-9752-0ac8c6997463 is now offline
000004c8.0000077c::2009/06/25-17:38:11.899 INFO [FM] RestartResourceTree, Restart resource 8f1f4232-ef39-42d6-9752-0ac8c6997463


000004c8.0000077c::2009/06/25-17:38:11.899 INFO [GUM] GumSendUpdate: queuing update type 0 context 11
000004c8.0000077c::2009/06/25-17:38:11.899 INFO [GUM] GumSendUpdate: Dispatching seq 30876 type 0 context 11 to node 1
000004c8.0000077c::2009/06/25-17:38:11.916 INFO [GUM] GumSendUpdate: completed update seq 30876 type 0 context 11
000008b8.00000f60::2009/06/25-17:38:19.446 ERR Volume Manager Disk Group <SQL Disk S:>: Dg-Guid is Null000008b8.00000f60::2009/06/25-17:38:19.446 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: Calling OnlineDg API
000008b8.00000f60::2009/06/25-17:38:26.960 ERR Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: *** Unable to bring resource online Error: Unknown disk group
000008b8.00000f60::2009/06/25-17:38:26.960 ERR Volume Manager Disk Group <SQL Disk S:>: GetMountVolumeInfo Error: 1.
000008b8.00000f60::2009/06/25-17:38:26.960 INFO [RM] RmpSetResourceStatus, Posting state 4 notification for resource <SQL Disk S:>
000004c8.000008cc::2009/06/25-17:38:26.960 INFO [FM] NotifyCallBackRoutine: enqueuing event
000008b8.00000f60::2009/06/25-17:38:26.960 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESOnlineThread: <<< Exiting LDM_RESOnlineThread
000004c8.0000017c::2009/06/25-17:38:26.960 INFO [CP] CppResourceNotify for resource SQL Disk S:
000004c8.0000017c::2009/06/25-17:38:26.960 INFO [FM] FmpRmDoHandleCriticalResourceStateChange: call InterlockedDecrement on gdwQuoBlockingResources, Resource 8f1f4232-ef39-42d6-9752-0ac8c6997463
000004c8.0000077c::2009/06/25-17:38:26.960 WARN [FM] FmpHandleResourceTransition: Resource Name = 8f1f4232-ef39-42d6-9752-0ac8c6997463 [SQL Disk S:] old state=129 new state=4
000004c8.0000077c::2009/06/25-17:38:26.960 WARN [FM] FmpHandleResourceTransition: Resource failed, post a work item
000004c8.0000077c::2009/06/25-17:38:26.960 INFO [GUM] GumSendUpdate: queuing update type 0 context 8
000004c8.0000077c::2009/06/25-17:38:26.960 INFO [GUM] GumSendUpdate: Dispatching seq 30877 type 0 context 8 to node 1
000004c8.0000077c::2009/06/25-17:38:26.960 INFO [GUM] GumSendUpdate: completed update seq 30877 type 0 context 8



004c8.0000077c::2009/06/25-17:41:40.439 INFO [CP] CppResourceNotify for resource SQL Network Name(SQLSERVER)
000004c8.0000017c::2009/06/25-17:41:40.439 INFO [FM] FmpRmDoHandleCriticalResourceStateChange: call InterlockedDecrement on gdwQuoBlockingResources, Resource 537affa1-bb80-4a32-8957-e2fe045353b3
000004c8.0000077c::2009/06/25-17:41:40.439 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:40.945 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:41.451 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:41.957 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:42.463 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:42.969 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:43.475 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:43.981 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:44.487 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:44.993 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:45.499 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:46.005 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:46.511 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:47.017 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000008b8.00000634::2009/06/25-17:41:47.222 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000004c8.0000077c::2009/06/25-17:41:47.523 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:48.029 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:48.535 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:49.041 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:49.546 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:50.052 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000077c::2009/06/25-17:41:50.558 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000004c8.0000084c::2009/06/25-17:41:50.922 INFO [RGP] Node 1: RGP Event : 0x7, 0x2, 0x6, 0x0.



000004c8.00000848::2009/06/25-17:42:01.816 INFO [RGP] Node 1: RGP Unicast: 0x2, 0x0, 0x0, 0x0.
000008b8.00000634::2009/06/25-17:42:01.990 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000008b8.00000634::2009/06/25-17:42:01.990 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000008b8.00000c10::2009/06/25-17:42:01.990 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
000008b8.00000ddc::2009/06/25-17:42:01.990 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: >>> Entering LDM_RESArbitrate
000004c8.00000b18::2009/06/25-17:42:01.990 INFO [MM] MmSetQuorumOwner(0,0), old owner 1.
000004c8.00000b18::2009/06/25-17:42:01.990 WARN [MM] MmSetQuorumOwner: regroup is in progress, forcing the new value in.
000004c8.00000b18::2009/06/25-17:42:01.990 INFO [FM] FmpTakeGroupRequest: MM did not select local node 1 as the arbitration winner, Status 5006
000004c8.00000b18::2009/06/25-17:42:01.990 INFO [FM] FmpTakeGroupRequest: Exit for group <1055be19-08c2-40bf-9a0c-934d58ffc9d3>, Status = 1237...
000008b8.00000b28::2009/06/25-17:42:01.990 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: >>> Entering LDM_RESArbitrateThread
000004c8.00000848::2009/06/25-17:42:02.100 INFO [RGP] Node 1: RGP Send packets: 0x3, 0xc0008000, 0x80008000, 0x0.
000004c8.00000848::2009/06/25-17:42:02.100 INFO [RGP] Node 1: RGP Unicast: 0x1, 0x0, 0x0, 0x0.


000004c8.00000848::2009/06/25-17:42:16.647 INFO [RGP] Node 1: RGP Unicast: 0x1, 0x0, 0x0, 0x0.
000004c8.00000848::2009/06/25-17:42:16.647 INFO [RGP] Node 1: RGP Incoming pkt: 0x7fff, 0x6, 0x3, 0x1.
000004c8.00000848::2009/06/25-17:42:16.647 INFO [RGP] Node 1: RGP recv pkt : 0x60003, 0x80008000, 0x80000000, 0x0.
000004c8.00000848::2009/06/25-17:42:16.647 INFO [RGP] Node 1: RGP Unicast: 0x2, 0x0, 0x0, 0x0.
000008b8.00000b28::2009/06/25-17:42:16.757 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000008b8.00000b28::2009/06/25-17:42:16.757 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: Calling ArbitrateDg API
000004c8.00000848::2009/06/25-17:42:16.963 INFO [RGP] Node 1: RGP Send packets: 0x3, 0xc0008000, 0x80008000, 0x0.
000004c8.00000848::2009/06/25-17:42:16.963 INFO [RGP] Node 1: RGP Unicast: 0x1, 0x0, 0x0, 0x0.
000004c8.00000848::2009/06/25-17:42:16.963 INFO [RGP] Node 1: RGP Incoming pkt: 0x7fff, 0x6, 0x3, 0x1.
000004c8.00000848::2009/06/25-17:42:16.963 INFO [RGP] Node 1: RGP recv pkt : 0x60003, 0x80008000, 0x80000000, 0x0.



000004c8.0000077c::2009/06/25-17:42:24.078 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000008b8.00000b28::2009/06/25-17:42:24.188 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000008b8.00000b28::2009/06/25-17:42:24.188 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000008b8.00000ddc::2009/06/25-17:42:24.188 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [MM] MmSetQuorumOwner(0,0), old owner 0.
000004c8.00000a8c::2009/06/25-17:42:24.188 WARN [MM] MmSetQuorumOwner: regroup is in progress, forcing the new value in.
000004c8.00000a8c::2009/06/25-17:42:24.188 ERR [FM] Failed to arbitrate quorum resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff, error 5006.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [NM] We do not own the quorum resource, status 6.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Event : 0x8, 0x1, 0x3, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP (re)starting: 0x7, 0x7f09, 0x1, 0xff.



000004c8.00000848::2009/06/25-17:42:23.920 INFO [RGP] Node 1: RGP recv pkt : 0x60003, 0x80008000, 0x80000000, 0x0.
000004c8.00000848::2009/06/25-17:42:23.920 INFO [RGP] Node 1: RGP Unicast: 0x2, 0x0, 0x0, 0x0.
000004c8.0000077c::2009/06/25-17:42:24.078 INFO [FM] FmpRmOnlineResource: release quolock/group lock and wait on ghQuoOnlineEvent
000008b8.00000b28::2009/06/25-17:42:24.188 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000008b8.00000b28::2009/06/25-17:42:24.188 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000008b8.00000ddc::2009/06/25-17:42:24.188 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [MM] MmSetQuorumOwner(0,0), old owner 0.
000004c8.00000a8c::2009/06/25-17:42:24.188 WARN [MM] MmSetQuorumOwner: regroup is in progress, forcing the new value in.
000004c8.00000a8c::2009/06/25-17:42:24.188 ERR [FM] Failed to arbitrate quorum resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff, error 5006.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [NM] We do not own the quorum resource, status 6.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Event : 0x8, 0x1, 0x3, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP (re)starting: 0x7, 0x7f09, 0x1, 0xff.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP masks : 0xc0008000, 0x80008000, 0x80000000, 0x8000.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [NM] Holding I/O.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Hold all IO : 0x0, 0x0, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: Self Isolation: 0x0, 0x0, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: Ignorescreen->: 0x7fff, 0x4000, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Send packets: 0x1, 0xc0000000, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Unicast: 0x1, 0x0, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Incoming pkt: 0x4000, 0x7, 0x1, 0x1.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP recv pkt : 0x70001, 0x0, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 INFO [RGP] Node 1: RGP Unicast: 0x2, 0x0, 0x0, 0x0.
000004c8.00000a8c::2009/06/25-17:42:24.188 ERR [RGP] Node 1: REGROUP ERROR: arbitration failed.
000004c8.00000a8c::2009/06/25-17:42:24.188 ERR [NM] Halting this node due to membership or communications error. Halt code = 1000
000004c8.0000084c::2009/06/25-17:42:24.188 INFO [ClMsg] Received delete address event, address b596a8c0
000004c8.00000394::2009/06/25-17:42:24.188 INFO [NM] Processing PnP delete event for address 192.168.110.181.
000004c8.00000394::2009/06/25-17:42:24.188 INFO [NM] Deleted address does not correspond to a cluster interface.
000004c8.0000084c::2009/06/25-17:42:24.188 INFO [ClMsg] Received delete address event, address b696a8c0
000004c8.00000394::2009/06/25-17:42:24.188 INFO [NM] Processing PnP delete event for address 192.168.110.156.
000004c8.00000394::2009/06/25-17:42:24.188 INFO [NM] Deleted address does not correspond to a cluster interface.
000004c8.0000084c::2009/06/25-17:42:24.188 INFO [ClMsg] Received unhandled event type 8 node 0 network 0
000004c8.00000a8c::2009/06/25-17:42:24.204 ERR [CS] Halting this node to prevent an inconsistency within the cluster. Error status = 5892
000008b8.000008bc::2009/06/25-17:42:24.315 WARN [RM] Going away, Status = 1, Shutdown = 0.
000008b8.000008bc::2009/06/25-17:42:24.315 INFO [RM] Posting shutdown notification.
000008b8.00000c14::2009/06/25-17:42:24.315 INFO [RM] NotifyChanges shutting down.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Microsoft Search Service Instance <SQL Server Fulltext>: Instance <> successfully closed

000008b8.000008bc::2009/06/25-17:42:24.821 INFO Network Name <VDISPATCH Network Name>: ResID 783424 closed.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Generic Service <VDISPATCH Service>: Close request.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Generic Service <VDISPATCH Service>: Terminate request.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO [RM] RundownResources, terminate resource <VDISPATCH IP Address>.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <VDISPATCH IP Address>: Terminating resource...
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <VDISPATCH IP Address>: NBT interface \Device\NetBt_If1 (instance 0x49338B3A) is no longer valid, status 2.
000008b8.000008bc::2009/06/25-17:42:24.821 WARN IP Address <VDISPATCH IP Address>: Failed to delete NBT interface information from database, status 1722.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <VDISPATCH IP Address>: IP interface 5 (instance 0x49388C0D) is no longer valid.
000008b8.000008bc::2009/06/25-17:42:24.821 WARN IP Address <VDISPATCH IP Address>: Failed to delete IP interface information from database, status 1722.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <VDISPATCH IP Address>: Address 192.168.120.136 on adapter HP NC7781 Gigabit Server Adapter #2 offline.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <VDISPATCH IP Address>: Resource closed.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Network Name <SQL Network Name(SQLSERVER)>: ResID 666976 closed.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO [RM] RundownResources, terminate resource <SQL IP Address1(SQLSERVER)>.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <SQL IP Address1(SQLSERVER)>: Terminating resource...
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <SQL IP Address1(SQLSERVER)>: NBT interface \Device\NetBt_If2 (instance 0x49338B3A) is no longer valid, status 2.
000008b8.000008bc::2009/06/25-17:42:24.821 WARN IP Address <SQL IP Address1(SQLSERVER)>: Failed to delete NBT interface information from database, status 1722.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <SQL IP Address1(SQLSERVER)>: IP interface 4 (instance 0x49388BED) is no longer valid.
000008b8.000008bc::2009/06/25-17:42:24.821 WARN IP Address <SQL IP Address1(SQLSERVER)>: Failed to delete IP interface information from database, status 1722.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <SQL IP Address1(SQLSERVER)>: Address 192.168.123.76 on adapter HP NC7781 Gigabit Server Adapter #2 offline.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <Cluster IP Address>: All resources offline - cleaning up
000008b8.000008ec::2009/06/25-17:42:24.821 ERR IP Address <Cluster IP Address>: WorkerThread: GetClusterNotify failed with status 6.
000008b8.000008ec::2009/06/25-17:42:24.821 INFO IP Address <Cluster IP Address>: WorkerThread terminating
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <SQL IP Address1(SQLSERVER)>: Resource closed.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESClose: >>> Entering LDM_RESClose
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESClose: RESOURCE IS CLOSED SUCCESSFULLY
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <SQL Disk S:>: LDM_RESClose: <<< Exiting LDM_RESClose
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: >>> Entering LDM_RESClose
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: RESOURCE IS CLOSED SUCCESSFULLY
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: <<< Exiting LDM_RESClose
000008b8.000008bc::2009/06/25-17:42:24.821 INFO Network Name <Cluster Name>: ResID 609904 closed.
000008b8.000008bc::2009/06/25-17:42:24.821 INFO IP Address <Cluster IP Address>: Resource closed.
000008b8.000008c4::2009/06/25-17:42:24.821 INFO [RM] RmpPollerThread: Asked to exit...
000008b8.000008c4::2009/06/25-17:42:24.821 INFO [RM] PollerThread stopping. Shutdown = 1, Status = 1, WaitFailed = 0, NotifyEvent address = 196.



00000c18.00000e9c::2009/06/25-17:43:25.107 INFO [CS] Cluster Service started - Cluster Node Version 4.3790
00000c18.00000e9c::2009/06/25-17:43:25.107 INFO OS Version 5.2.3790 (ADS 03000112L)
00000c18.00000e9c::2009/06/25-17:43:25.107 INFO Local Time is 2009/06/25-19:43:25.107
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [CS] Service Starting...
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [INIT] ClusterInitialize called to start cluster.
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [EP] Initialization...
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [DM] Initialization
00000c18.00000ec8::2009/06/25-17:43:25.107 ERR [DM] DmInitialize: The hive was loaded- rollback, unload and reload again
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [DM] DmpRestartFlusher: Entry
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [DM] DmpUnloadHive: unloading the hive
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [Qfs] QfsSetFileAttributes C:\WINDOWS\Cluster\CLUSDB.BKP$ 80, status 2
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [Qfs] QfsDeleteFile C:\WINDOWS\Cluster\CLUSDB.BKP$, status 2
00000c18.00000ec8::2009/06/25-17:43:25.107 INFO [DM] Loading cluster database from C:\WINDOWS\Cluster\CLUSDB
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [DM] DmpStartFlusher: Entry
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [DM] DmpStartFlusher: thread created
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Initializing...
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Local node name = SPAARNE.
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Local node ID = 1.
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Creating object for node 1 (SPAARNE)
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Initializing networks.
00000c18.00000ec8::2009/06/25-17:43:25.217 INFO [NM] Initializing network interfaces.
00000c18.00000ec8::2009/06/25-17:43:25.265 WARN [NM] Failed to open cluster parameters key, status 2.
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [NM] Initialization complete.
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [FM] Starting worker thread...
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [API] Initializing
00000c18.00000eec::2009/06/25-17:43:25.265 INFO [FM] Worker thread running
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [LM] LmInitialize Entry.
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [LM] TimerActInitialize Entry.
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [CS] Service Domain Account = DGR\S_Cluster_Spalm
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [CS] Initializing RPC server.
00000c18.00000ec8::2009/06/25-17:43:25.265 INFO [INIT] CspRegisterDynamicLRPCEndpoint: Successfully registered LRPC endpoint with EP mapper
00000c18.00000ec8::2009/06/25-17:43:25.280 INFO [INIT] Attempting to join cluster SPALM
00000c18.00000ec8::2009/06/25-17:43:25.280 WARN [JOIN] Spawning thread to connect to sponsor 192.168.110.123
00000c18.00000ec8::2009/06/25-17:43:25.280 WARN [JOIN] Spawning thread to connect to sponsor 192.168.100.87
00000c18.000003ec::2009/06/25-17:43:25.280 WARN [JOIN] Asking 192.168.100.87 to sponsor us after delay of 0 milliseconds.
00000c18.00000ec8::2009/06/25-17:43:25.280 WARN [JOIN] Spawning thread to connect to sponsor SWALM
00000c18.00000ec8::2009/06/25-17:43:25.280 WARN [JOIN] Spawning thread to connect to sponsor 192.168.110.123
00000c18.00000edc::2009/06/25-17:43:26.290 WARN [JOIN] Asking 192.168.145.111 to sponsor us after delay of 1000 milliseconds.
00000c18.00000ac8::2009/06/25-17:43:26.290 WARN [JOIN] Asking SWALM to sponsor us after delay of 1000 milliseconds.
00000c18.00000ef4::2009/06/25-17:43:27.299 WARN [JOIN] Asking 192.168.110.134 to sponsor us after delay of 2000 milliseconds.
00000c18.000003ec::2009/06/25-17:43:41.427 INFO [JOIN] Sponsor 192.168.100.87 is not available (JoinVersion), status=1722.
00000c18.000003ec::2009/06/25-17:43:41.427 WARN [JOIN] JoinVersion data for sponsor 192.168.100.87 is invalid, status 1722.
00000c18.00000edc::2009/06/25-17:43:58.584 INFO [JOIN] Sponsor 192.168.110.123 is not available (JoinVersion), status=1722.
00000c18.00000ac8::2009/06/25-17:43:58.584 INFO [JOIN] Sponsor SWALM is not available (JoinVersion), status=1722.
00000c18.00000edc::2009/06/25-17:43:58.584 WARN [JOIN] JoinVersion data for sponsor 192.168.110.145 is invalid, status 1722.
00000c18.00000ac8::2009/06/25-17:43:58.584 WARN [JOIN] JoinVersion data for sponsor SWALM is invalid, status 1722.
00000c18.00000ef4::2009/06/25-17:43:59.593 INFO [JOIN] Sponsor 192.168.130.180 is not available (JoinVersion), status=1722.
00000c18.00000ef4::2009/06/25-17:43:59.593 WARN [JOIN] JoinVersion data for sponsor 192.168.120.134 is invalid, status 1722.
00000c18.00000ec8::2009/06/25-17:43:59.593 INFO [JOIN] Got out of the join wait, CsJoinThreadCount = 1.
00000c18.00000ec8::2009/06/25-17:43:59.593 ERR [JOIN] Unable to connect to any sponsor node.
00000c18.00000ec8::2009/06/25-17:43:59.593 WARN [INIT] Failed to join cluster, status 53
00000c18.00000ec8::2009/06/25-17:43:59.593 INFO [INIT] Attempting to form cluster SPALM
00000c18.00000ec8::2009/06/25-17:43:59.593 INFO [API] Online read only
00000c18.00000ec8::2009/06/25-17:43:59.593 ERR [FM] Failed to get quorum disk signature, error 2.
000001e0.000001e4::2009/06/25-17:43:59.609 INFO [RM] Main: Initializing.
000001e0.000001e4::2009/06/25-17:43:59.609 INFO [RM] RmpGetDynamicEndpointName: Successfully got LRPC endpoint info, EP name is LRPC000001e0.00000001
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Resmon LRPC EP name is LRPC000001e0.00000001
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Creating group 1055be19-08c2-40bf-9a0c-934d58ffc9d3
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Initializing group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 from the registry.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Name for Group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 is 'Cluster Group'.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 preferred owner 1.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 contains Resource 9df7facf-35e6-47af-abe1-1dcc02a50b46.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Creating resource 9df7facf-35e6-47af-abe1-1dcc02a50b46
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Initializing resource 9df7facf-35e6-47af-abe1-1dcc02a50b46 from the registry.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Name for Resource 9df7facf-35e6-47af-abe1-1dcc02a50b46 is 'Cluster IP Address'.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleEntry: adding node 1 as possible host for resource 9df7facf-35e6-47af-abe1-1dcc02a50b46.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpQueryResTypeInfo: Calling FmpAddPossibleNodeToList for restype IP Address
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: adding node 1 to resource type's possible node list
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: Warning, node 2 not found
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] All dependencies for resource 9df7facf-35e6-47af-abe1-1dcc02a50b46 created.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 contains Resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Creating resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Initializing resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3 from the registry.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Name for Resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3 is 'Cluster Name'.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleEntry: adding node 1 as possible host for resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpQueryResTypeInfo: Calling FmpAddPossibleNodeToList for restype Network Name
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: adding node 1 to resource type's possible node list
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: Warning, node 2 not found
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3 depends on 9df7facf-35e6-47af-abe1-1dcc02a50b46. Creating...
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] CreateResource, Opened existing resource 9df7facf-35e6-47af-abe1-1dcc02a50b46
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] All dependencies for resource 8d7e7ba9-8ef3-461d-9c03-97ce6430fdf3 created.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Group 1055be19-08c2-40bf-9a0c-934d58ffc9d3 contains Resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Creating resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Initializing resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff from the registry.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Name for Resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff is 'Quorum_Server'.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleEntry: adding node 1 as possible host for resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpQueryResTypeInfo: Calling FmpAddPossibleNodeToList for restype Volume Manager Disk Group
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: adding node 1 to resource type's possible node list
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpAddPossibleNodeToList: Warning, node 2 not found
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Found the quorum resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] All dependencies for resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff created.
00000c18.00000ec8::2009/06/25-17:43:59.609 ERR [FM] Arbitrate for quorum resource id 3e2fae0c-5fa8-424f-b8b1-ab655da215ff.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Initializing resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff from the registry.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] Name for Resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff is 'Quorum_Server'.
00000c18.00000ec8::2009/06/25-17:43:59.609 INFO [FM] FmpRmCreateResource: creating resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff in shared resource monitor
000001e0.00000d6c::2009/06/25-17:43:59.624 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESOpen: RESOURCE OPENED SUCCESSFULLY
000001e0.00000d6c::2009/06/25-17:43:59.624 INFO Volume Manager Disk Group <Quorum_Server>: <<< Exiting LDM_RESOpen
00000c18.00000ec8::2009/06/25-17:43:59.624 INFO [FM] FmpRmCreateResource: created resource 3e2fae0c-5fa8-424f-b8b1-ab655da215ff, resid 900368
00000c18.00000ec8::2009/06/25-17:43:59.624 INFO [MM] MmSetQuorumOwner(1,1), old owner 0.
000001e0.000005d0::2009/06/25-17:43:59.624 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: >>> Entering LDM_RESArbitrate
000001e0.00000564::2009/06/25-17:43:59.624 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: >>> Entering LDM_RESArbitrateThread
000001e0.00000564::2009/06/25-17:44:07.099 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000001e0.00000564::2009/06/25-17:44:14.463 ERR Volume Manager Disk Group <Quorum_Server>: Dg-Guid is Null000001e0.00000564::2009/06/25-17:44:14.463 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: Calling ArbitrateDg API
000001e0.00000564::2009/06/25-17:44:21.827 ERR Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread - *** ArbitrateDg API failed - resource is not available - error: Unknown disk group
000001e0.00000564::2009/06/25-17:44:21.827 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrateThread: <<< Exiting LDM_RESArbitrateThread
000001e0.000005d0::2009/06/25-17:44:21.827 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESArbitrate: <<< Exiting LDM_RESArbitrate
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [MM] MmSetQuorumOwner(0,0), old owner 1.
00000c18.00000ec8::2009/06/25-17:44:21.827 ERR [FM] FmGetQuorumResource failed, error 5006.
00000c18.00000ec8::2009/06/25-17:44:21.827 ERR [INIT] ClusterForm: Could not get quorum resource. No fixup attempted. Status = 5086
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [INIT] Cleaning up failed form attempt.
00000c18.00000ec8::2009/06/25-17:44:21.827 ERR [INIT] Failed to form cluster, status 5086.
00000c18.00000ec8::2009/06/25-17:44:21.827 ERR [CS] ClusterInitialize failed 5086
00000c18.00000ec8::2009/06/25-17:44:21.827 WARN [INIT] The cluster service is shutting down.
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [EVT] EvShutdown
00000c18.00000ec8::2009/06/25-17:44:21.827 WARN [FM] Shutdown: Failover Manager requested to shutdown groups.
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [FM] FmpCleanupGroups: Entry
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [FM] FmpCleanupGroups: Exit
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [Dm] DmShutdown
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [DM] DmpShutdownFlusher: Entry
00000c18.00000ec8::2009/06/25-17:44:21.827 INFO [DM] DmpShutdownFlusher: Setting event
00000c18.00000b44::2009/06/25-17:44:21.827 INFO [DM] DmpRegistryFlusher: got 0
00000c18.00000b44::2009/06/25-17:44:21.827 INFO [DM] DmpRegistryFlusher: exiting
00000c18.00000ec8::2009/06/25-17:44:21.842 WARN [MM] MMLeave is called when rgp=NULL.
00000c18.00000ec8::2009/06/25-17:44:21.842 ERR [CS] Service Stopped. exit code = 5086

000001e0.000001e4::2009/06/25-17:44:22.063 WARN [RM] Going away, Status = 1, Shutdown = 0.
000001e0.000001e4::2009/06/25-17:44:22.063 INFO [RM] Posting shutdown notification.
000001e0.000007e8::2009/06/25-17:44:22.063 INFO [RM] NotifyChanges shutting down.
000001e0.000001e4::2009/06/25-17:44:22.568 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: >>> Entering LDM_RESClose
000001e0.000001e4::2009/06/25-17:44:22.568 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: RESOURCE IS CLOSED SUCCESSFULLY
000001e0.000001e4::2009/06/25-17:44:22.568 INFO Volume Manager Disk Group <Quorum_Server>: LDM_RESClose: <<< Exiting LDM_RESClose
000001e0.000001dc::2009/06/25-17:44:22.568 INFO [RM] RmpPollerThread: Asked to exit...
000001e0.000001dc::2009/06/25-17:44:22.568 INFO [RM] PollerThread stopping. Shutdown = 1, Status = 1, WaitFailed = 0, NotifyEvent address = 196.
00000154.00000f40::2009/06/25-17:46:23.187 INFO