Cluster Service Not Starting on Win2003 - HELP!

Here's the problem I'm running into. I've got two nodes... one active and
the other passive. Both are running Windows 2003 SP1. Yesterday, the active
node started alerting me of the following error: The server was unable to
allocate from the system paged pool because the pool was empty. The active
node is still up and it's file share resources are all available but I'm
worried that it'll crash soon due to a possible memory leak.

I rebooted the passive node before I wanted to manually fail over the active
to the passive, however after the reboot, the passive node could not start up
the Cluster Service. The passive node was reporting that it could not
contact the active node hence could not join the cluster. The cluster
service account is fine. I can also ping the heartbeat interface on the
active node from the passive so I feel network communication is working
properly.

I ran clussvc -debug and here was the output.
C:\WINDOWS\Cluster>clussvc -debug
[DM] DmInitialize: The hive was loaded- rollback, unload and reload again
[NM] Failed to open cluster parameters key, status 2.
[JOIN] Attempting join with sponsor 172.30.132.23.
[MM] MmQuorumArbitrationTimeout 60.
[NMJOIN] Member node 1 failed to create binding to us, status 5
[NMJOIN] Member node 1 failed to create binding to us, status 5
[NMJOIN] Member node 1 failed to create binding to us, status 5
[JOIN] NmJoinCluster failed, status 5.
[ClMsg] Receive datagram failed, status 995
[ClMsg] Post of receive datagram failed, status 10038
[INIT] Failed to join cluster, status 5
[CS] ClusterInitialize failed 5
[INIT] The cluster service is shutting down.
[FM] Shutdown: Failover Manager requested to shutdown groups.
[MM] MMLeave is called when rgp=NULL.
[CS] Service Stopped. exit code = 5

Anyone have any ideas why my passive node cannot communicate with my active
node and join the cluster? Is it because the active node's memory pool is
empty? I tried to get Dell support to help but they can't figure out why the
passive can't join the cluster.

net helpmsg 5 = "Access is Denied"

Check out your cluster service account permissions with this KB
http://support.microsoft.com/?id=269229

-CT

"ikaikaokaina" wrote:

> Here's the problem I'm running into. I've got two nodes... one active and
> the other passive. Both are running Windows 2003 SP1. Yesterday, the active
> node started alerting me of the following error: The server was unable to
> allocate from the system paged pool because the pool was empty. The active
> node is still up and it's file share resources are all available but I'm
> worried that it'll crash soon due to a possible memory leak.
>
> I rebooted the passive node before I wanted to manually fail over the active
> to the passive, however after the reboot, the passive node could not start up
> the Cluster Service. The passive node was reporting that it could not
> contact the active node hence could not join the cluster. The cluster
> service account is fine. I can also ping the heartbeat interface on the
> active node from the passive so I feel network communication is working
> properly.
>
> I ran clussvc -debug and here was the output.
> C:\WINDOWS\Cluster>clussvc -debug
> [DM] DmInitialize: The hive was loaded- rollback, unload and reload again
> [NM] Failed to open cluster parameters key, status 2.
> [JOIN] Attempting join with sponsor 172.30.132.23.
> [MM] MmQuorumArbitrationTimeout 60.
> [NMJOIN] Member node 1 failed to create binding to us, status 5
> [NMJOIN] Member node 1 failed to create binding to us, status 5
> [NMJOIN] Member node 1 failed to create binding to us, status 5
> [JOIN] NmJoinCluster failed, status 5.
> [ClMsg] Receive datagram failed, status 995
> [ClMsg] Post of receive datagram failed, status 10038
> [INIT] Failed to join cluster, status 5
> [CS] ClusterInitialize failed 5
> [INIT] The cluster service is shutting down.
> [FM] Shutdown: Failover Manager requested to shutdown groups.
> [MM] MMLeave is called when rgp=NULL.
> [CS] Service Stopped. exit code = 5
>
> Anyone have any ideas why my passive node cannot communicate with my active
> node and join the cluster? Is it because the active node's memory pool is
> empty? I tried to get Dell support to help but they can't figure out why the
> passive can't join the cluster.

It's one of the things I checked first. I logged into the passive node with
the domain cluster service account just to verify that I had the password set
correctly. That was how I was able to run the clussvc -debug.

Someone please pass me the dumb hat. In a feeble attempt to get that
cluster service started on the passive node, I uninstalled the cluster
service on that passive node and cannot reinstall the service back. When I
go to the Control Panel, select Add/Remove Programs, Select Add/Remove
Windows Components, the Cluster Service isn't listed. So now I've got an
active node with memory issues and a passive node that doesn't even have the
Clustering Services installed.

"Charles Tolento" wrote:

> net helpmsg 5 = "Access is Denied"
>
> Check out your cluster service account permissions with this KB
> http://support.microsoft.com/?id=269229
>
> -CT
>
> "ikaikaokaina" wrote:
>
> > Here's the problem I'm running into. I've got two nodes... one active and
> > the other passive. Both are running Windows 2003 SP1. Yesterday, the active
> > node started alerting me of the following error: The server was unable to
> > allocate from the system paged pool because the pool was empty. The active
> > node is still up and it's file share resources are all available but I'm
> > worried that it'll crash soon due to a possible memory leak.
> >
> > I rebooted the passive node before I wanted to manually fail over the active
> > to the passive, however after the reboot, the passive node could not start up
> > the Cluster Service. The passive node was reporting that it could not
> > contact the active node hence could not join the cluster. The cluster
> > service account is fine. I can also ping the heartbeat interface on the
> > active node from the passive so I feel network communication is working
> > properly.
> >
> > I ran clussvc -debug and here was the output.
> > C:\WINDOWS\Cluster>clussvc -debug
> > [DM] DmInitialize: The hive was loaded- rollback, unload and reload again
> > [NM] Failed to open cluster parameters key, status 2.
> > [JOIN] Attempting join with sponsor 172.30.132.23.
> > [MM] MmQuorumArbitrationTimeout 60.
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [JOIN] NmJoinCluster failed, status 5.
> > [ClMsg] Receive datagram failed, status 995
> > [ClMsg] Post of receive datagram failed, status 10038
> > [INIT] Failed to join cluster, status 5
> > [CS] ClusterInitialize failed 5
> > [INIT] The cluster service is shutting down.
> > [FM] Shutdown: Failover Manager requested to shutdown groups.
> > [MM] MMLeave is called when rgp=NULL.
> > [CS] Service Stopped. exit code = 5
> >
> > Anyone have any ideas why my passive node cannot communicate with my active
> > node and join the cluster? Is it because the active node's memory pool is
> > empty? I tried to get Dell support to help but they can't figure out why the
> > passive can't join the cluster.

Thanks. I got the problem resolved. Here's what I had to do. I restarted
the Cluster server on the hanging active node. By hanging I mean I couldn't
log into the system via Terminal Services or via the console. I could
however, get access to the system through remote Computer Management (so I
could see Services, Shares, etc.) Once the Cluster service was re-started I
could then re-add the old passive node to the cluster. Once that was added,
I manually failed over the active node resources to the passive node. After
all the resources were moved over I could then do a hard power cycle on that
hung system. Now the old active node is back and is the new passive node.

"Charles Tolento" wrote:

> net helpmsg 5 = "Access is Denied"
>
> Check out your cluster service account permissions with this KB
> http://support.microsoft.com/?id=269229
>
> -CT
>
> "ikaikaokaina" wrote:
>
> > Here's the problem I'm running into. I've got two nodes... one active and
> > the other passive. Both are running Windows 2003 SP1. Yesterday, the active
> > node started alerting me of the following error: The server was unable to
> > allocate from the system paged pool because the pool was empty. The active
> > node is still up and it's file share resources are all available but I'm
> > worried that it'll crash soon due to a possible memory leak.
> >
> > I rebooted the passive node before I wanted to manually fail over the active
> > to the passive, however after the reboot, the passive node could not start up
> > the Cluster Service. The passive node was reporting that it could not
> > contact the active node hence could not join the cluster. The cluster
> > service account is fine. I can also ping the heartbeat interface on the
> > active node from the passive so I feel network communication is working
> > properly.
> >
> > I ran clussvc -debug and here was the output.
> > C:\WINDOWS\Cluster>clussvc -debug
> > [DM] DmInitialize: The hive was loaded- rollback, unload and reload again
> > [NM] Failed to open cluster parameters key, status 2.
> > [JOIN] Attempting join with sponsor 172.30.132.23.
> > [MM] MmQuorumArbitrationTimeout 60.
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [NMJOIN] Member node 1 failed to create binding to us, status 5
> > [JOIN] NmJoinCluster failed, status 5.
> > [ClMsg] Receive datagram failed, status 995
> > [ClMsg] Post of receive datagram failed, status 10038
> > [INIT] Failed to join cluster, status 5
> > [CS] ClusterInitialize failed 5
> > [INIT] The cluster service is shutting down.
> > [FM] Shutdown: Failover Manager requested to shutdown groups.
> > [MM] MMLeave is called when rgp=NULL.
> > [CS] Service Stopped. exit code = 5
> >
> > Anyone have any ideas why my passive node cannot communicate with my active
> > node and join the cluster? Is it because the active node's memory pool is
> > empty? I tried to get Dell support to help but they can't figure out why the
> > passive can't join the cluster.