Cluster problems, how to recover?

[Sigitas Skublickas]

Hello,

I have Windows 2003 cluster server with two nodes. I was hosting SQL 2005
failover on top of it. For some reason this morning I'm not able to connect
to the
cluster administrator console anymore from any node. If I try to do so,
after quite
some time I get an error:

"The Cluster Service on the node "SQLCluster" cannot be started. The network
path was not found. Error ID: -2147024843 (80070035)."

I have verified that the cluster service is started on both nodes. I tried
restarting it
for testing purpose but still can’t get to the console. Servers and storage
reboot did
not help as well. If I go to windows explorer I can get only into quorum
volume,
while the other two volumes are “listed” only as a drive letters on BOTH
nodes. I
can’t ping cluster IP resources from network. I have noticed that the quorum
volume fails over OK to another node if I reboot one node, but the other two
volumes with my SQL stuff totally dead no matter what I try to do. When I
was rebooting nodes windows event viewer did not record anything interesting
about dead volumes or cluster itself other then the events that the “….server
was removed from the active server cluster membership…” and “…connection has
been lost with the other node”….

The storage device HP 1500 StorageWorks looks ok, no warnings are shown in
the LCD panels. I’m able to connect to the controllers via HP ACU and see
arrays fine.

Any ideas how to troubleshoot this situation?

[Jeff Hughes [MSFT]]

Do a start/run and type

cluadmin .

That's cluadmin followed by a space then a period '.'
That should open up cluster administrator. You'll probably find that your
cluster name and IP (and possibly everything else) are offline/failed. If
they are offline, just manually bring them online. If any resources are
failed, troubleshoot accordingly.

--
Jeff Hughes, MCSE
Support Escalation Engineer
Microsoft Enterprise Platforms Support (Server Core/Cluster)

"Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote in
message news:96384475-32AB-4754-BB71-4C9D5AEE661E@microsoft.com...
> Hello,
>
> I have Windows 2003 cluster server with two nodes. I was hosting SQL 2005
> failover on top of it. For some reason this morning I'm not able to
> connect
> to the
> cluster administrator console anymore from any node. If I try to do so,
> after quite
> some time I get an error:
>
> "The Cluster Service on the node "SQLCluster" cannot be started. The
> network
> path was not found. Error ID: -2147024843 (80070035)."
>
> I have verified that the cluster service is started on both nodes. I tried
> restarting it
> for testing purpose but still can’t get to the console. Servers and
> storage
> reboot did
> not help as well. If I go to windows explorer I can get only into quorum
> volume,
> while the other two volumes are “listed” only as a drive letters on BOTH
> nodes. I
> can’t ping cluster IP resources from network. I have noticed that the
> quorum
> volume fails over OK to another node if I reboot one node, but the other
> two
> volumes with my SQL stuff totally dead no matter what I try to do. When I
> was rebooting nodes windows event viewer did not record anything
> interesting
> about dead volumes or cluster itself other then the events that the
> “….server
> was removed from the active server cluster membership…” and “…connection
> has
> been lost with the other node”….
>
> The storage device HP 1500 StorageWorks looks ok, no warnings are shown in
> the LCD panels. I’m able to connect to the controllers via HP ACU and see
> arrays fine.
>
> Any ideas how to troubleshoot this situation?
>

[Sigitas Skublickas]

Mr. Jeff,

Thanks for your quick response!

Looks like it was a case.... the only thing that I'm curious about is why
did the resources go to offline by themselves? I'm the only one who has
access to the cluster and I know for sure that I did not bring them offline.
Do you know if Microsoft has some kind of guide or Technet article about
cluster troubleshooting/recovery?

Thanks!
Sigitas Skublickas

"Jeff Hughes [MSFT]" wrote:

> Do a start/run and type
>
> cluadmin .
>
> That's cluadmin followed by a space then a period '.'
> That should open up cluster administrator. You'll probably find that your
> cluster name and IP (and possibly everything else) are offline/failed. If
> they are offline, just manually bring them online. If any resources are
> failed, troubleshoot accordingly.
>
> --
> Jeff Hughes, MCSE
> Support Escalation Engineer
> Microsoft Enterprise Platforms Support (Server Core/Cluster)
>
> "Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote in
> message news:96384475-32AB-4754-BB71-4C9D5AEE661E@microsoft.com...
> > Hello,
> >
> > I have Windows 2003 cluster server with two nodes. I was hosting SQL 2005
> > failover on top of it. For some reason this morning I'm not able to
> > connect
> > to the
> > cluster administrator console anymore from any node. If I try to do so,
> > after quite
> > some time I get an error:
> >
> > "The Cluster Service on the node "SQLCluster" cannot be started. The
> > network
> > path was not found. Error ID: -2147024843 (80070035)."
> >
> > I have verified that the cluster service is started on both nodes. I tried
> > restarting it
> > for testing purpose but still can’t get to the console. Servers and
> > storage
> > reboot did
> > not help as well. If I go to windows explorer I can get only into quorum
> > volume,
> > while the other two volumes are “listed” only as a drive letters on BOTH
> > nodes. I
> > can’t ping cluster IP resources from network. I have noticed that the
> > quorum
> > volume fails over OK to another node if I reboot one node, but the other
> > two
> > volumes with my SQL stuff totally dead no matter what I try to do. When I
> > was rebooting nodes windows event viewer did not record anything
> > interesting
> > about dead volumes or cluster itself other then the events that the
> > “….server
> > was removed from the active server cluster membership…” and “…connection
> > has
> > been lost with the other node”….
> >
> > The storage device HP 1500 StorageWorks looks ok, no warnings are shown in
> > the LCD panels. I’m able to connect to the controllers via HP ACU and see
> > arrays fine.
> >
> > Any ideas how to troubleshoot this situation?
> >

[Jeff Hughes [MSFT]]

There is nothing the cluster does to offline resources by themselves. It's
either user initiated or the cluster service was started with the
/fixquorum. Cluster troubleshooting is too broad a topic to have just an
article or guide on but you can start here:

Windows Server 2003 R2 Enterprise Edition – Cluster Server Resource Center
http://www.microsoft.com/windowsserv...resources.mspx


Hope this helps.

Jeff Hughes, MCSE
Support Escalation Engineer
Microsoft Enterprise Platforms Support (Server Core/Cluster)

"Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote in
message news:5B66D9D3-5713-475E-A268-032172F2113B@microsoft.com...
> Mr. Jeff,
>
> Thanks for your quick response!
>
> Looks like it was a case.... the only thing that I'm curious about is why
> did the resources go to offline by themselves? I'm the only one who has
> access to the cluster and I know for sure that I did not bring them
> offline.
> Do you know if Microsoft has some kind of guide or Technet article about
> cluster troubleshooting/recovery?
>
> Thanks!
> Sigitas Skublickas
>
> "Jeff Hughes [MSFT]" wrote:
>
>> Do a start/run and type
>>
>> cluadmin .
>>
>> That's cluadmin followed by a space then a period '.'
>> That should open up cluster administrator. You'll probably find that your
>> cluster name and IP (and possibly everything else) are offline/failed. If
>> they are offline, just manually bring them online. If any resources are
>> failed, troubleshoot accordingly.
>>
>> --
>> Jeff Hughes, MCSE
>> Support Escalation Engineer
>> Microsoft Enterprise Platforms Support (Server Core/Cluster)
>>
>> "Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote
>> in
>> message news:96384475-32AB-4754-BB71-4C9D5AEE661E@microsoft.com...
>> > Hello,
>> >
>> > I have Windows 2003 cluster server with two nodes. I was hosting SQL
>> > 2005
>> > failover on top of it. For some reason this morning I'm not able to
>> > connect
>> > to the
>> > cluster administrator console anymore from any node. If I try to do so,
>> > after quite
>> > some time I get an error:
>> >
>> > "The Cluster Service on the node "SQLCluster" cannot be started. The
>> > network
>> > path was not found. Error ID: -2147024843 (80070035)."
>> >
>> > I have verified that the cluster service is started on both nodes. I
>> > tried
>> > restarting it
>> > for testing purpose but still can’t get to the console. Servers and
>> > storage
>> > reboot did
>> > not help as well. If I go to windows explorer I can get only into
>> > quorum
>> > volume,
>> > while the other two volumes are “listed” only as a drive letters on
>> > BOTH
>> > nodes. I
>> > can’t ping cluster IP resources from network. I have noticed that the
>> > quorum
>> > volume fails over OK to another node if I reboot one node, but the
>> > other
>> > two
>> > volumes with my SQL stuff totally dead no matter what I try to do. When
>> > I
>> > was rebooting nodes windows event viewer did not record anything
>> > interesting
>> > about dead volumes or cluster itself other then the events that the
>> > “….server
>> > was removed from the active server cluster membership…” and
>> > “…connection
>> > has
>> > been lost with the other node”….
>> >
>> > The storage device HP 1500 StorageWorks looks ok, no warnings are shown
>> > in
>> > the LCD panels. I’m able to connect to the controllers via HP ACU and
>> > see
>> > arrays fine.
>> >
>> > Any ideas how to troubleshoot this situation?
>> >

[Sigitas Skublickas]

Thanks Jeff. I really appreciate your help.

"Jeff Hughes [MSFT]" wrote:

> There is nothing the cluster does to offline resources by themselves. It's
> either user initiated or the cluster service was started with the
> /fixquorum. Cluster troubleshooting is too broad a topic to have just an
> article or guide on but you can start here:
>
> Windows Server 2003 R2 Enterprise Edition – Cluster Server Resource Center
> http://www.microsoft.com/windowsserv...resources.mspx
>
>
> Hope this helps.
>
> Jeff Hughes, MCSE
> Support Escalation Engineer
> Microsoft Enterprise Platforms Support (Server Core/Cluster)
>
> "Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote in
> message news:5B66D9D3-5713-475E-A268-032172F2113B@microsoft.com...
> > Mr. Jeff,
> >
> > Thanks for your quick response!
> >
> > Looks like it was a case.... the only thing that I'm curious about is why
> > did the resources go to offline by themselves? I'm the only one who has
> > access to the cluster and I know for sure that I did not bring them
> > offline.
> > Do you know if Microsoft has some kind of guide or Technet article about
> > cluster troubleshooting/recovery?
> >
> > Thanks!
> > Sigitas Skublickas
> >
> > "Jeff Hughes [MSFT]" wrote:
> >
> >> Do a start/run and type
> >>
> >> cluadmin .
> >>
> >> That's cluadmin followed by a space then a period '.'
> >> That should open up cluster administrator. You'll probably find that your
> >> cluster name and IP (and possibly everything else) are offline/failed. If
> >> they are offline, just manually bring them online. If any resources are
> >> failed, troubleshoot accordingly.
> >>
> >> --
> >> Jeff Hughes, MCSE
> >> Support Escalation Engineer
> >> Microsoft Enterprise Platforms Support (Server Core/Cluster)
> >>
> >> "Sigitas Skublickas" <SigitasSkublickas@discussions.microsoft.com> wrote
> >> in
> >> message news:96384475-32AB-4754-BB71-4C9D5AEE661E@microsoft.com...
> >> > Hello,
> >> >
> >> > I have Windows 2003 cluster server with two nodes. I was hosting SQL
> >> > 2005
> >> > failover on top of it. For some reason this morning I'm not able to
> >> > connect
> >> > to the
> >> > cluster administrator console anymore from any node. If I try to do so,
> >> > after quite
> >> > some time I get an error:
> >> >
> >> > "The Cluster Service on the node "SQLCluster" cannot be started. The
> >> > network
> >> > path was not found. Error ID: -2147024843 (80070035)."
> >> >
> >> > I have verified that the cluster service is started on both nodes. I
> >> > tried
> >> > restarting it
> >> > for testing purpose but still can’t get to the console. Servers and
> >> > storage
> >> > reboot did
> >> > not help as well. If I go to windows explorer I can get only into
> >> > quorum
> >> > volume,
> >> > while the other two volumes are “listed” only as a drive letters on
> >> > BOTH
> >> > nodes. I
> >> > can’t ping cluster IP resources from network. I have noticed that the
> >> > quorum
> >> > volume fails over OK to another node if I reboot one node, but the
> >> > other
> >> > two
> >> > volumes with my SQL stuff totally dead no matter what I try to do. When
> >> > I
> >> > was rebooting nodes windows event viewer did not record anything
> >> > interesting
> >> > about dead volumes or cluster itself other then the events that the
> >> > “….server
> >> > was removed from the active server cluster membership…” and
> >> > “…connection
> >> > has
> >> > been lost with the other node”….
> >> >
> >> > The storage device HP 1500 StorageWorks looks ok, no warnings are shown
> >> > in
> >> > the LCD panels. I’m able to connect to the controllers via HP ACU and
> >> > see
> >> > arrays fine.
> >> >
> >> > Any ideas how to troubleshoot this situation?
> >> >