NTDS ISAM 467 database corruption

[low40p]

Hi :) I have this event id NTDS ISAM 467 database corruption error nightmare
about some days ago and I can not get rid of it :(

All DC's are Windows Server 2003 SP2, I have 3 sites. In every site I have 2
DC's. All DC's are GC. At site A is located FSMO DC and Exchange 2003 std.
server. On DC's at this site A I do not have 467 errors. These error is
showing up only on DC's at sites B and C. But detailes of descrpition of
these error I did not see after searching.

Here is the error:
Event Type: Error
Event Source: NTDS ISAM
Event Category: Database Corruption
Event ID: 467
Description:
NTDS (416) NTDSA: Index LCL_ABVIEW_index0000081A of table datatable is
corrupted (0).

After that 467 error I received also these errors:

Event Type: Warning
Event Source: NTDS General
Event Category: Internal Processing
Event ID: 1173
User: NT AUTHORITY\ANONYMOUS LOGON
Description:
Internal event: Active Directory has encountered the following exception and
associated parameters.

Exception:
e0010004
Parameter:
0

Additional Data
Error value:
-1414
Internal ID:
2080490

and error 2041 also internal processing. After that error I have:

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 1084
User: NT AUTHORITY\ANONYMOUS LOGON
Description:
Internal event: Active Directory could not update the following object with
changes received from the following source domain controller. This is because
an error occurred during the application of the changes to Active Directory
on the domain controller.

Object:
CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
Objects,DC=domain,DC=sld,DC=tld
Object GUID:
dfdd5116-b09d-4f6b-9721-59e93a04379c
Source domain controller:
2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld

Synchronization of the local domain controller with the source domain
controller is blocked until this update problem is corrected.

This operation will be tried again at the next scheduled replication.

User Action
Restart the local domain controller if this condition appears to be related
to low system resources (for example, low physical or virtual memory).

Additional Data
Error value:
8451 The replication operation encountered a database error.

Event Type: Error
Event Source: NTDS Replication
Event Category: Replication
Event ID: 2108
User: NT AUTHORITY\ANONYMOUS LOGON
Description:
This event contains REPAIR PROCEDURES for the 1084 event which has
previously been logged. This message indicates a specific issue with the
consistency of the Active Directory database on this replication destination.
A database error occurred while applying replicated changes to the following
object. The database had unexpected contents, preventing the change from
being made.

Object:
CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
Objects,DC=domain,DC=sld,DC=tld
Object GUID:
dfdd5116-b09d-4f6b-9721-59e93a04379c
Source domain controller:
2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld

User Action

.... (deleted because is well known ;)

Additional Data
Primary Error value:
8451 The replication operation encountered a database error.
Secondary Error value:
-1414 JET_errSecondaryIndexCorrupted, Secondary index is corrupt. The
database must be defragmented

and also errors NTDS KCC 1566, 1311, 1865 about replication and topology.

Only thing I tryed is defragmentation of AD database on PDC Root domain
controler in DSRM mode at site A with ntdsutil and it pass ok without errors.
I did not try semantic analysis etc.

Does anyone have suggestion what to do next ? How can I find faulty DC ? Is
it at site A or on other sites. Is it only solution to build whole domain and
network again ?

So far all network activities are ok, users can log in to network but
replication is blocked.

[Jorge Silva]

check if no errors on the Disks or any other hardware, also have a look at
your Antiirus configuration and the possibility of server infection.
Then Some possible solutions:
*Solution1:
Perform restore operation to resolve the issue with your lastest Systate
BACKUPS.

*Solution 2:
In case this is not the only DC in the domain, you can simply rebuild it.
At the same time, you will need to perform the steps below before
re-promoting the server:
1. Seize FSMO roles to the existing DC in the domain. For the detailed
steps, you can refer to the following Microsoft Knowledge Base article:
Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller
http://support.microsoft.com/?id=255504
2. Remove remnant entries of the corrupted DC from AD database.
How to remove data in Active Directory after an unsuccessful domain
controller demotion
http://support.microsoft.com/?id=216498
--
I hope that the information above helps you.
Have a Nice day.

Jorge Silva
MCSE, MVP Directory Services
"low40p" <low40p@discussions.microsoft.com> wrote in message
news:0D236337-5209-4114-A9A1-3703853821B5@microsoft.com...
> Hi :) I have this event id NTDS ISAM 467 database corruption error
> nightmare
> about some days ago and I can not get rid of it :(
>
> All DC's are Windows Server 2003 SP2, I have 3 sites. In every site I have
> 2
> DC's. All DC's are GC. At site A is located FSMO DC and Exchange 2003 std.
> server. On DC's at this site A I do not have 467 errors. These error is
> showing up only on DC's at sites B and C. But detailes of descrpition of
> these error I did not see after searching.
>
> Here is the error:
> Event Type: Error
> Event Source: NTDS ISAM
> Event Category: Database Corruption
> Event ID: 467
> Description:
> NTDS (416) NTDSA: Index LCL_ABVIEW_index0000081A of table datatable is
> corrupted (0).
>
> After that 467 error I received also these errors:
>
> Event Type: Warning
> Event Source: NTDS General
> Event Category: Internal Processing
> Event ID: 1173
> User: NT AUTHORITY\ANONYMOUS LOGON
> Description:
> Internal event: Active Directory has encountered the following exception
> and
> associated parameters.
>
> Exception:
> e0010004
> Parameter:
> 0
>
> Additional Data
> Error value:
> -1414
> Internal ID:
> 2080490
>
> and error 2041 also internal processing. After that error I have:
>
> Event Type: Error
> Event Source: NTDS Replication
> Event Category: Replication
> Event ID: 1084
> User: NT AUTHORITY\ANONYMOUS LOGON
> Description:
> Internal event: Active Directory could not update the following object
> with
> changes received from the following source domain controller. This is
> because
> an error occurred during the application of the changes to Active
> Directory
> on the domain controller.
>
> Object:
> CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
> Objects,DC=domain,DC=sld,DC=tld
> Object GUID:
> dfdd5116-b09d-4f6b-9721-59e93a04379c
> Source domain controller:
> 2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld
>
> Synchronization of the local domain controller with the source domain
> controller is blocked until this update problem is corrected.
>
> This operation will be tried again at the next scheduled replication.
>
> User Action
> Restart the local domain controller if this condition appears to be
> related
> to low system resources (for example, low physical or virtual memory).
>
> Additional Data
> Error value:
> 8451 The replication operation encountered a database error.
>
> Event Type: Error
> Event Source: NTDS Replication
> Event Category: Replication
> Event ID: 2108
> User: NT AUTHORITY\ANONYMOUS LOGON
> Description:
> This event contains REPAIR PROCEDURES for the 1084 event which has
> previously been logged. This message indicates a specific issue with the
> consistency of the Active Directory database on this replication
> destination.
> A database error occurred while applying replicated changes to the
> following
> object. The database had unexpected contents, preventing the change from
> being made.
>
> Object:
> CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
> Objects,DC=domain,DC=sld,DC=tld
> Object GUID:
> dfdd5116-b09d-4f6b-9721-59e93a04379c
> Source domain controller:
> 2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld
>
> User Action
>
> ... (deleted because is well known ;)
>
> Additional Data
> Primary Error value:
> 8451 The replication operation encountered a database error.
> Secondary Error value:
> -1414 JET_errSecondaryIndexCorrupted, Secondary index is corrupt. The
> database must be defragmented
>
> and also errors NTDS KCC 1566, 1311, 1865 about replication and topology.
>
> Only thing I tryed is defragmentation of AD database on PDC Root domain
> controler in DSRM mode at site A with ntdsutil and it pass ok without
> errors.
> I did not try semantic analysis etc.
>
> Does anyone have suggestion what to do next ? How can I find faulty DC ?
> Is
> it at site A or on other sites. Is it only solution to build whole domain
> and
> network again ?
>
> So far all network activities are ok, users can log in to network but
> replication is blocked.

[Brad Rutkowski]

-1414 is JET_errSecondaryIndexCorrupted, doing an offline defrag via DSRM
will rebuild the indexes and perhaps resolve the issue.

http://msdn2.microsoft.com/en-us/library/ms683086.aspx

There is an error with a secondary index. This may be from physical
corruption (such as disk or memory corruption). It may also be returned when
attaching a database last modified on an older operating system and a
secondary index is over a column with Unicode data. See the remarks for more
information on indexes over Unicode data. Secondary indexes are completely
rebuilt when a database is defragmented with an offline utility using the
following command: esentutl -d.
--
This post is provided as-is and is my opinion, check the facts before doing
anything in production.

BTW I used err.exe to get the actual error:
http://blogs.technet.com/brad_rutkow...-is-admin.aspx


"Jorge Silva" wrote:

> check if no errors on the Disks or any other hardware, also have a look at
> your Antiirus configuration and the possibility of server infection.
> Then Some possible solutions:
> *Solution1:
> Perform restore operation to resolve the issue with your lastest Systate
> BACKUPS.
>
> *Solution 2:
> In case this is not the only DC in the domain, you can simply rebuild it.
> At the same time, you will need to perform the steps below before
> re-promoting the server:
> 1. Seize FSMO roles to the existing DC in the domain. For the detailed
> steps, you can refer to the following Microsoft Knowledge Base article:
> Using Ntdsutil.exe to transfer or seize FSMO roles to a domain controller
> http://support.microsoft.com/?id=255504
> 2. Remove remnant entries of the corrupted DC from AD database.
> How to remove data in Active Directory after an unsuccessful domain
> controller demotion
> http://support.microsoft.com/?id=216498
> --
> I hope that the information above helps you.
> Have a Nice day.
>
> Jorge Silva
> MCSE, MVP Directory Services
> "low40p" <low40p@discussions.microsoft.com> wrote in message
> news:0D236337-5209-4114-A9A1-3703853821B5@microsoft.com...
> > Hi :) I have this event id NTDS ISAM 467 database corruption error
> > nightmare
> > about some days ago and I can not get rid of it :(
> >
> > All DC's are Windows Server 2003 SP2, I have 3 sites. In every site I have
> > 2
> > DC's. All DC's are GC. At site A is located FSMO DC and Exchange 2003 std.
> > server. On DC's at this site A I do not have 467 errors. These error is
> > showing up only on DC's at sites B and C. But detailes of descrpition of
> > these error I did not see after searching.
> >
> > Here is the error:
> > Event Type: Error
> > Event Source: NTDS ISAM
> > Event Category: Database Corruption
> > Event ID: 467
> > Description:
> > NTDS (416) NTDSA: Index LCL_ABVIEW_index0000081A of table datatable is
> > corrupted (0).
> >
> > After that 467 error I received also these errors:
> >
> > Event Type: Warning
> > Event Source: NTDS General
> > Event Category: Internal Processing
> > Event ID: 1173
> > User: NT AUTHORITY\ANONYMOUS LOGON
> > Description:
> > Internal event: Active Directory has encountered the following exception
> > and
> > associated parameters.
> >
> > Exception:
> > e0010004
> > Parameter:
> > 0
> >
> > Additional Data
> > Error value:
> > -1414
> > Internal ID:
> > 2080490
> >
> > and error 2041 also internal processing. After that error I have:
> >
> > Event Type: Error
> > Event Source: NTDS Replication
> > Event Category: Replication
> > Event ID: 1084
> > User: NT AUTHORITY\ANONYMOUS LOGON
> > Description:
> > Internal event: Active Directory could not update the following object
> > with
> > changes received from the following source domain controller. This is
> > because
> > an error occurred during the application of the changes to Active
> > Directory
> > on the domain controller.
> >
> > Object:
> > CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
> > Objects,DC=domain,DC=sld,DC=tld
> > Object GUID:
> > dfdd5116-b09d-4f6b-9721-59e93a04379c
> > Source domain controller:
> > 2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld
> >
> > Synchronization of the local domain controller with the source domain
> > controller is blocked until this update problem is corrected.
> >
> > This operation will be tried again at the next scheduled replication.
> >
> > User Action
> > Restart the local domain controller if this condition appears to be
> > related
> > to low system resources (for example, low physical or virtual memory).
> >
> > Additional Data
> > Error value:
> > 8451 The replication operation encountered a database error.
> >
> > Event Type: Error
> > Event Source: NTDS Replication
> > Event Category: Replication
> > Event ID: 2108
> > User: NT AUTHORITY\ANONYMOUS LOGON
> > Description:
> > This event contains REPAIR PROCEDURES for the 1084 event which has
> > previously been logged. This message indicates a specific issue with the
> > consistency of the Active Directory database on this replication
> > destination.
> > A database error occurred while applying replicated changes to the
> > following
> > object. The database had unexpected contents, preventing the change from
> > being made.
> >
> > Object:
> > CN=username\0ADEL:dfdd5116-b09d-4f6b-9721-59e93a04379c,CN=Deleted
> > Objects,DC=domain,DC=sld,DC=tld
> > Object GUID:
> > dfdd5116-b09d-4f6b-9721-59e93a04379c
> > Source domain controller:
> > 2dda290a-fa20-465b-9546-2bb42feaae8a._msdcs.domain.sld.tld
> >
> > User Action
> >
> > ... (deleted because is well known ;)
> >
> > Additional Data
> > Primary Error value:
> > 8451 The replication operation encountered a database error.
> > Secondary Error value:
> > -1414 JET_errSecondaryIndexCorrupted, Secondary index is corrupt. The
> > database must be defragmented
> >
> > and also errors NTDS KCC 1566, 1311, 1865 about replication and topology.
> >
> > Only thing I tryed is defragmentation of AD database on PDC Root domain
> > controler in DSRM mode at site A with ntdsutil and it pass ok without
> > errors.
> > I did not try semantic analysis etc.
> >
> > Does anyone have suggestion what to do next ? How can I find faulty DC ?
> > Is
> > it at site A or on other sites. Is it only solution to build whole domain
> > and
> > network again ?
> >
> > So far all network activities are ok, users can log in to network but
> > replication is blocked.
>
>
>

[low40p]

"Jorge Silva" wrote:

> check if no errors on the Disks or any other hardware, also have a look at

Disk Managment snap-in does not show any errors. Hard drives on all DC's are
mirrored (win 2003 server software mirror)

> your Antiirus configuration and the possibility of server infection.

NOD32 is regularly updated every hour on all DC's, there is no sign of
infection.

> Perform restore operation to resolve the issue with your lastest Systate
> BACKUPS.

Do I have to perform restore on one, first root DC (FSMO) in site A which
does not have / show up errors in event log or to do restore on all other
DC's in other sites B and C whch have errors in event log ?

> In case this is not the only DC in the domain, you can simply rebuild it.
> At the same time, you will need to perform the steps below before
> re-promoting the server:

> 2. Remove remnant entries of the corrupted DC from AD database.

Demote which DC's, only affected with errors in sites B and C or whole bunch
of 6 DC's, or only 2 in site A which does not have errors ?

Sorry but I must post here again, 3 sites, 6 DC's - 2 at one site. Site A
does not have errors, only DC's at site B and C. Can you tell me what do you
think, which DC's causing trouble ? I do not understand completly error
message. Does it mean that DC's at site B and C can not receive updates
because they have error or DC's from site A can not send updates to sites B
and C ?

What do you suggest me, which order to use, where to start ? For example,
shutdown DC's at site B and C, install new one DC and let replicate from site
DC's at site A ?

> I hope that the information above helps you.

10x anyway what ever should be result, your help is precious. :)

> > Here is the error:
> > Event Type: Error
> > Event Source: NTDS ISAM
> > Event Category: Database Corruption
> > Event ID: 467
> > Description:
> > NTDS (416) NTDSA: Index LCL_ABVIEW_index0000081A of table datatable is
> > corrupted (0).

Also I must ask again if anyone knows. Is this index file part of ntds.dit
or something else, for example Exchange ? I did not find these type of index
file in any of KB articles.

ABVIEW seems to me like address book, 81A is code for Serbian langauge.

Before offline defragmentation on root DC I tried these:

create new OU, move problematic user account from other OU into these new
one, and then delete account and new one OU. The actions are replicated onto
other DC's and I don't have on them anymore deleted account now.

[low40p]

"Brad Rutkowski" wrote:

> -1414 is JET_errSecondaryIndexCorrupted, doing an offline defrag via DSRM
> will rebuild the indexes and perhaps resolve the issue.

OK, but where to do offline defrag ? Onto all affected DC's, which showing
up error in event log or onto DC's at site A which does not show up any
errors ?

> attaching a database last modified on an older operating system and a

All DC's are Windows 2003 Server SP2

Anyway thanks for help and tips. Best regards.

[Jorge Silva]

did you read this?
http://support.microsoft.com/kb/902396

--
I hope that the information above helps you.
Have a Nice day.

Jorge Silva
MCSE, MVP Directory Services
"low40p" <low40p@discussions.microsoft.com> wrote in message
news:C23A6DD1-FC13-4067-9258-FBBE29B45FD7@microsoft.com...
> "Jorge Silva" wrote:
>
>> check if no errors on the Disks or any other hardware, also have a look
>> at
>
> Disk Managment snap-in does not show any errors. Hard drives on all DC's
> are
> mirrored (win 2003 server software mirror)
>
>> your Antiirus configuration and the possibility of server infection.
>
> NOD32 is regularly updated every hour on all DC's, there is no sign of
> infection.
>
>> Perform restore operation to resolve the issue with your lastest Systate
>> BACKUPS.
>
> Do I have to perform restore on one, first root DC (FSMO) in site A which
> does not have / show up errors in event log or to do restore on all other
> DC's in other sites B and C whch have errors in event log ?
>
>> In case this is not the only DC in the domain, you can simply rebuild it.
>> At the same time, you will need to perform the steps below before
>> re-promoting the server:
>
>> 2. Remove remnant entries of the corrupted DC from AD database.
>
> Demote which DC's, only affected with errors in sites B and C or whole
> bunch
> of 6 DC's, or only 2 in site A which does not have errors ?
>
> Sorry but I must post here again, 3 sites, 6 DC's - 2 at one site. Site A
> does not have errors, only DC's at site B and C. Can you tell me what do
> you
> think, which DC's causing trouble ? I do not understand completly error
> message. Does it mean that DC's at site B and C can not receive updates
> because they have error or DC's from site A can not send updates to sites
> B
> and C ?
>
> What do you suggest me, which order to use, where to start ? For example,
> shutdown DC's at site B and C, install new one DC and let replicate from
> site
> DC's at site A ?
>
>> I hope that the information above helps you.
>
> 10x anyway what ever should be result, your help is precious. :)
>
>> > Here is the error:
>> > Event Type: Error
>> > Event Source: NTDS ISAM
>> > Event Category: Database Corruption
>> > Event ID: 467
>> > Description:
>> > NTDS (416) NTDSA: Index LCL_ABVIEW_index0000081A of table datatable is
>> > corrupted (0).
>
> Also I must ask again if anyone knows. Is this index file part of ntds.dit
> or something else, for example Exchange ? I did not find these type of
> index
> file in any of KB articles.
>
> ABVIEW seems to me like address book, 81A is code for Serbian langauge.
>
> Before offline defragmentation on root DC I tried these:
>
> create new OU, move problematic user account from other OU into these new
> one, and then delete account and new one OU. The actions are replicated
> onto
> other DC's and I don't have on them anymore deleted account now.

[low40p]

"Jorge Silva" wrote:

> did you read this? http://support.microsoft.com/kb/902396

Yes I've read that KB article. That's why I'm posted here that all my DC's
are under Windows Server 2003 SP2.

Did I understood wrong that hotfix is for server version which does not have
applied SP1 ?

[Jorge Silva]

In that case you may want to try to repair the Dabase or recover it from
Backup.

--
I hope that the information above helps you.
Have a Nice day.

Jorge Silva
MCSE, MVP Directory Services
"low40p" <low40p@discussions.microsoft.com> wrote in message
news:FBD8AF49-480D-4CA6-A268-7C07990B73C1@microsoft.com...
> "Jorge Silva" wrote:
>
>> did you read this? http://support.microsoft.com/kb/902396
>
> Yes I've read that KB article. That's why I'm posted here that all my DC's
> are under Windows Server 2003 SP2.
>
> Did I understood wrong that hotfix is for server version which does not
> have
> applied SP1 ?

[low40p]

Hi :) First thanks to you and Brad, now I have functional DC's again :)

On DC's with errors I've started DSRM mode. Used NTDSUtil to check integrity
- result database corruption, to perform semantic analisys without fixup - no
errors and to make offline defragmentation - no errors. I've renamed old log
files, rename old ntds.dit and copied a new one defragmented ntds.dit file in
place. After that I run again ntdsutil files integrity check - no errors and
reboot DC.

Now my AD database is 5MB smaller then previous.

Also I must reply to myself: lc_abview index file is part od ntds.dit ad
database.