Failed P2V - Active Directory USN rollback issue

Hi all,

Could someone please put my mind to rest - this has been driving me mad
since last week.

Last Tuesday whilst I was out of the office, my IT Manager attempted a P2V
conversion of our last physical Active Directory server. The P2V conversion
failed and so he brought the physical box back online. Since then, site
replication has been messed up.

From what I've been reading, the easiest way to recover from this is

1) Transfer FSMO roles to another DC (This AD server was our operations
master!)
2) Demote DC1 as a domain controller
3) Shutdown
4) Logon to healthy DC and clean meta-data for DC1 from AD
5) Start DC1 and run dcpromo/install AD
6) Wait for AD synch to take place
7) Transfer FSMO roles back to DC1

Could someone just confirm this is the right process to follow in this
situation?

For added info, below is USN output from repladmin.

3150641d-59d4-4428-a107-2d9917e666e9 @ USN 17209 @ Time 2007-06-12
17:11:12
Default-First-Site-Name\MMCGADS005 @ USN 980309 @ Time 2008-11-11 17:16:06
Houston\MMCHADS001 @ USN 933113 @ Time 2008-11-11 15:43:21
5592a03f-f358-40b8-87cc-c80f322bcf8b @ USN 1144891 @ Time 2008-05-07
14:57:36
Houston\MMCHADS002 @ USN 658236 @ Time 2008-11-11 15:47:20
88907d25-4d8e-4906-bb20-e0472eca1147 @ USN 1123895 @ Time 2007-05-23
16:05:15
95615a60-7172-453f-8785-afdfcee5e100 @ USN 20679 @ Time 2006-10-29
13:40:26
Default-First-Site-Name\MMCGADS001 @ USN 4362071 @ Time 2008-11-17 12:34:50

Hello cyacomini,

See here about:
http://support.microsoft.com/kb/875495

Your steps are the correct one's as you can see in the article. Make also
sure that you have GC's and DNS server available on other DC's.

Best regards

Meinolf Weber
Disclaimer: This posting is provided "AS IS" with no warranties, and confers
no rights.
** Please do NOT email, only reply to Newsgroups
** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm


> USN 17209 @ Time 2007-06-12
>

Ah yes - global catalog and DNS services need to be moved too.

Can you tell me if this will cause any problems with Exchange Server 2003?

I notice in AD Sites & Services there is "Active Directory Connector" ADC
Service listed under the same server.

"Meinolf Weber" wrote:

> Hello cyacomini,
>
> See here about:
> http://support.microsoft.com/kb/875495
>
> Your steps are the correct one's as you can see in the article. Make also
> sure that you have GC's and DNS server available on other DC's.
>
> Best regards
>
> Meinolf Weber
> Disclaimer: This posting is provided "AS IS" with no warranties, and confers
> no rights.
> ** Please do NOT email, only reply to Newsgroups
> ** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm
>
>
> > USN 17209 @ Time 2007-06-12
> >
>
>
>

Hello cyacomini,

If i am not wrong this ADC is from upgrading from Exchange 5.5.

In ESM make sure it points in the recipient update service to a running GC/DC
not that one you will remove.

Also in a single forest domain, yours sounds like that, you can make all
DC's GC. Or do you have multiple domains/child domains?

Best regards

Meinolf Weber
Disclaimer: This posting is provided "AS IS" with no warranties, and confers
no rights.
** Please do NOT email, only reply to Newsgroups
** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm


> Ah yes - global catalog and DNS services need to be moved too.
>
> Can you tell me if this will cause any problems with Exchange Server
> 2003?
>
> I notice in AD Sites & Services there is "Active Directory Connector"
> ADC Service listed under the same server.
>
> "Meinolf Weber" wrote:
>
>> Hello cyacomini,
>>
>> See here about:
>> http://support.microsoft.com/kb/875495
>> Your steps are the correct one's as you can see in the article. Make
>> also sure that you have GC's and DNS server available on other DC's.
>>
>> Best regards
>>
>> Meinolf Weber
>> Disclaimer: This posting is provided "AS IS" with no warranties, and
>> confers
>> no rights.
>> ** Please do NOT email, only reply to Newsgroups
>> ** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm
>>> USN 17209 @ Time 2007-06-12
>>>

Sorry - 3rd and last question (I hope)

I notice in the repadmin output there are entries containing just GUID's and
no server name. These are all dates 2006/2007 which is before I started work
for this organisation so not sure why these should be listed there at all.

?

"cyacomini" wrote:

> Ah yes - global catalog and DNS services need to be moved too.
>
> Can you tell me if this will cause any problems with Exchange Server 2003?
>
> I notice in AD Sites & Services there is "Active Directory Connector" ADC
> Service listed under the same server.
>
> "Meinolf Weber" wrote:
>
> > Hello cyacomini,
> >
> > See here about:
> > http://support.microsoft.com/kb/875495
> >
> > Your steps are the correct one's as you can see in the article. Make also
> > sure that you have GC's and DNS server available on other DC's.
> >
> > Best regards
> >
> > Meinolf Weber
> > Disclaimer: This posting is provided "AS IS" with no warranties, and confers
> > no rights.
> > ** Please do NOT email, only reply to Newsgroups
> > ** HELP us help YOU!!! http://www.blakjak.demon.co.uk/mul_crss.htm
> >
> >
> > > USN 17209 @ Time 2007-06-12
> > >
> >
> >
> >

Replied inline...


> If i am not wrong this ADC is from upgrading from Exchange 5.5.

Yes, probably from an upgrade that took place just before I started work here.

>
> In ESM make sure it points in the recipient update service to a running GC/DC
> not that one you will remove.

Thank you - noted

> Also in a single forest domain, yours sounds like that, you can make all
> DC's GC. Or do you have multiple domains/child domains?

Yes, I think it is a single forest. In AD S&S I can see under sites

Default-First-Site-Name (Which is our Head Office)
Houston (Which is our sales office)

We are on a single domain, but with 2 defined 'sites' on different subnets,
both of which are managed from the same domain (companyname.local)

Hello cyacomini,
Did you run this specific command ? repadmin /showutdvec dc_name dc=your_domain,dc=com


It is just odd that things aren't similar as far as guid's and names. I
not real experienced here but the holes are way off, like this has been going
on for some time and want to make sure that something else isn't giving improper
results. Also I don't see consistent guid's if you ran this against two
different dc's (In my experience) the output should line up (GUID wise) in
the results. Unless you pruned the output this seems odd. Just wanting
to check before you push forward.

Hi Paul,

Yes - I've run that specific command. It was run from the DC which failed
the P2V process so it's from the same physical box I plan to demote/promote.

Here is the output again - I have left domain/server names as is this time.


Microsoft Windows [Version 5.2.3790]
(C) Copyright 1985-2003 Microsoft Corp.

D:\Documents and Settings\user>repadmin /showutdvec mclglaads001 dc=mclaren
,dc=local
Caching GUIDs.
...
3150641d-59d4-4428-a107-2d9917e666e9 @ USN 17209 @ Time 2007-06-12
17:11:12
Default-First-Site-Name\MCLGLAADS005 @ USN 980309 @ Time 2008-11-11
17:16:06
Houston\MCLHOUADS001 @ USN 933113 @ Time 2008-11-11
15:43:21
5592a03f-f358-40b8-87cc-c80f322bcf8b @ USN 1144891 @ Time 2008-05-07
14:57:36
Houston\MCLHOUADS002 @ USN 658236 @ Time 2008-11-11
15:47:20
88907d25-4d8e-4906-bb20-e0472eca1147 @ USN 1123895 @ Time 2007-05-23
16:05:15
95615a60-7172-453f-8785-afdfcee5e100 @ USN 20679 @ Time 2006-10-29
13:40:26
Default-First-Site-Name\MCLGLAADS001 @ USN 4362373 @ Time 2008-11-17
13:41:37

The work to convert all of the DC's was started before I began work for this
company so I couldn't say for sure to be honest.

I would assume it was a straight P2V conversion though as the same process
was used here resulting in the borked AD box.

There are 4 DC's in total within the company, 3 of which are virtualised
already over the last few months. The last physical box contained FSMO,
Global Catalog etc which I think is why it was left until last.

"it looks like a mess to me."

Yes, my thoughts exactly.

The problem I have here is that I started working for this company not that
long ago and most of the work to convert the DC's had already been completed
when I joined. So I'm in the position where I need to pick apart what I'm
left with to try and find out what exactly has gone wrong.

This was only flagged to me last Thursday after a collegue tried to P2V the
last DC so I've only been looking at that particular server so far. Looking
at these outputs worries me more though because I'm also seeing USN rollbacks
on multiple servers!

Any advice on how to proceed?

Hello cyacomini,

I would do it this way:

Choose the DC which have the latest objects created which you know and start
with that one as the DC that will stay, make it GC and move all 5 FSMO roles
to it if possible. If not you have to seize them.

Then disconnect all other DC's from the domain and run dcpromo /forceremoval
on them. If all DC's are only member servers disjoin them from the domain
to workgroup.
Then cleanup AD database on the last existing one according to this for all
demoted DC's:
http://support.microsoft.com/kb/555846/en-us

Also cleanup DNS from all old entries of the removed servers.

After that run dcdiag /v, netdiag /v to check for errors.

If all are removed you can start to promote them again.

After every new server is promoted run dcdiag /v, netdiag /v and repadmin
/showrepl to check for errors.
Sounds like a bunch of fun!

Do all of the other DC's need to be disjoined from the domain through-out
the process or can they be done 1-by-1 to minimise downtime?

I'm assuming when the DC's are promoted again they will synch with the DC
holding the 5 FSMO roles? If that is the case, it would make things much
easier for me - 2 of the DC's are in another office running on VM's. They are
in a remote location so physically getting to them is not an option...

Hello cyacomini,
I would accept that there are issues and demote three of the four. Get all
the fsmo roles to the server you ultimately choose as the master and press
on from there. You need to get it down to one dc before you start promoting,
otherwise I believe you could be back in the same situation.

Check out an article I have on Decommissioning a DC on my website. You should
make sure to follow the tips in there and I would take two backups of your
dc that you plan on keeping prior to demoting. Expect users to have short
term difficulties once things are replicating properly, since things are
out of sync right now.

Hello cyacomini,

Don't do it one by one, because one change, for example user changes a password
which you will not recognize, creates a new USN, which has to be replicated
and because your replication is not correct it will end in the same situation.

Unfortunaly you will have a small downtime for the users i think. And it
can happen that they call your helpdesk, that maybe there password is not
longer working.

Best regards

Hi guys..

I completed the work on our 3 'broken' domain controllers this morning.

The demotion and metadata cleanup's went well with no errors encountered.
However, when promoting the servers to DC's again an error appeared and the
promotion failed.

"the source server is currently rejecting replication requests"

In order to get around this, from the 'master' DC I had to run the following
from a CMD prompt

repadmin /options SERVERNAME -disable_inbound_repl

and

repadmin /options SERVERNAME -disable_outbound_repl


Once that had been run, I attempted to promote the other DC's and all was
well.

So, I'm now in the situation where I have 4 heatlhy DC's all completing
replication without any problems.

Once again, thank you to the both of you for assistance here - it's very
much appreciated.

Hello cyacomini,
You are welcome but I would do the following weekly (I do).

Run diagnostics against your Active Directory domain.

If you don't have the support tools installed, install them from your server
install disk.
d:\support\tools\setup.exe

Run dcdiag, netdiag and repadmin in verbose mode.
-> DCDIAG /V /C /D /E /s:yourdcname > c:\dcdiag.log
-> repadmin.exe /showrepl dc* /verbose /all /intersite > c:\repl.txt
-> dnslint /ad /s "ip address of your dc"

**Note: Using the /E switch in dcdiag will run diagnostics against ALL dc's
in the forest. If you have significant numbers of DC's this test could generate
significant detail and take a long time. You also want to take into account
slow links to dc's will also add to the testing time.

If you download a gui script I wrote it should be simple to set and run (DCDiag
and NetDiag). It also has the option to run individual tests without having
to learn all the switch options. The details will be output in notepad text
files that pop up automagically.

The script is located on my website at http://www.pbbergs.com/windows/downloads.htm

Just select both dcdiag and netdiag make sure verbose is set. (Leave the
default settings for dcdiag as set when selected)

When complete search for fail, error and warning messages.

Description and download for dnslint
http://support.microsoft.com/kb/321045