Ok, I’m sorry, I’ve been extremely busy with work and family. The following article is one I’ve been working on since September of this year! So here it is….
Over the past nine months I have been battling an issue in Exchange 2010 that reported “All Global Catalog Servers in forest DC=ScottFeltmann,DC=com are not responding. I feel that I can finally say the issue has been resolved!
Some background:
This was a large Exchange deployment, two active directory sites, two Hub/CAS and two mailbox Servers in the primary site running DAG. Secondary site had two Hub/CAS and one Mailbox Server. The focus of the issue seemed to take place in the primary site. The issue we were seeing initially is that users would hang, literally hang, both in Outlook or OWA. It seemed that the Client Access Server (one of the two) was hanging, as if waiting for a response. Rebooting the server always cleared up the issue but this was by no means a solution.
Now, what made this problem difficult is that it occurred randomly and about every three to four weeks. There was no way I could recreate the problem and could only trouble shoot the issue when the problem was occurring. After my initial troubleshooting and seeing the issue occur randomly. The primary Errors in the Event Log were Event ID: 2102, Event ID: 2103, and Event ID: 2114.
Event 2102 Info:
Log Name: Application
Source: MSExchange ADAccess
Date: 3/2/2011 6:03:04 AM
Event ID: 2102
Task Category: Topology
Level: Error
Keywords: Classic
User: N/A
Computer: CAS1.scottfeltmann.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1364). All Domain Controller Servers in use are not responding:
DCAuth1.scottfeltmann.com
DCMAIN3.scottfeltmann.com
DCMain3008.scottfeltmann.com
DCMain2.scottfeltmann.com
DCSC1.scottfeltmann.com
DCSC2.scottfeltmann.com
DCMain1.scottfeltmann.com
Event 2103 Info:
Log Name: Application
Source: MSExchange ADAccess
Date: 3/11/2011 10:46:21 AM
Event ID: 2103
Task Category: Topology
Level: Error
Keywords: Classic
User: N/A
Computer: CAS1.scottfeltmann.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1524). All Global Catalog Servers in forest DC=ScottFeltmann,DC=com are not responding:
DCSC1.scottfeltmann.com
DCSC2.scottfeltmann.com
DCMain1.scottfeltmann.com
DCMain2.scottfeltmann.com
DCAuth1.scottfeltmann.com
DCMAIN3.scottfeltmann.com
DCMain3008.scottfeltmann.com
Event 2114 Info:
Error:
Log Name: Application
Source: MSExchange ADAccess
Date: 4/14/2011 10:48:30 AM
Event ID: 2114
Task Category: Topology
Level: Error
Keywords: Classic
User: N/A
Computer: CAS2.scottfeltmann.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1480). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, “Microsoft LDAP Error Codes.” Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.
I decided to call Microsoft support as I had never seen this type of behavior in any of my deployments. To be honest it came down to about 5 calls into MS Support, rare case which happens once every few weeks = difficult to trouble shoot, even with Microsoft. I spoke to both Exchange support and Active Directory support engineers. I could go into great detail about the trouble shooting steps I took with MS support but none of these resolved the issue so what’s the point. To give you an idea we tried to reset the Kerbos ticket, we ran DCDiag, checked Domain controller Replication, checked DNS, nothing. Long story short after four or five calls with MS Support I finally was able to get escalated to level 3 or 4, either way, Rick (the MS Support Engineer) figured out what the issue was.
What Rick noticed was a Warning in the System Event Log, Event ID 40961 from the source LsaSrv. This Error was never reviewed with other MS Support people as I’m not sure they thought it wasn’t a bad issue or not. To be honest, a warning isn’t something I really turn my attention to when a bad situation is occurring (learned my lesson there). Anyway, here is the event info
Event ID 40961 Info:
Log Name: System
Source: LsaSrv
Date: 4/14/2011 10:53:30 AM
Event ID: 40961
Task Category: None
Level: Warning
Keywords:
User: SYSTEM
Computer: CAS2.scottfeltmann.com
Description:
The Security System could not establish a secured connection with the server ldap/ DCMain3008.scottfeltmann.com/scottfeltmann.com@SCOTTFELTMANN.COM. No authentication protocol was available.
Rick mentioned he sees this error when systems are having issues with Remote Desktop. I couldn’t help to think but how does RDP tie into my problem with Exchange. The way it ties in is because the secure channel between the source and the target is lost and causes the systems not to be able to authenticate. Basically the CAS was trying to communicate with the DC while the user was waiting. The DC and CAS didn’t trust each other and bam, CAS HUNG!
What is the fix? After about 9 months of issues and calls with MS at random times, there is a hotfix, drum roll….. KB939820! Which is located here: http://support.microsoft.com/kb/939820
Once this hotfix was applied the problem stopped occurring, the Event Logs no longer showed a 2102, 2103, or 2114 error and life was good again!
While researching the issue I also found a posting on spiceworks where users were having the same problem. Here is the URL for that: http://community.spiceworks.com/topic/134941-exchange-2010-ad-topology-failures-all-domain-controllers-unavailable?page=4#entry-847654
Talk about Crazy! I’m just glad it is over.
Hope this helps out there!
Comments? Questions? Please Share!
I want to make sure I’m following this post.
You had 2008 and 2003 DCs, where you seeing the 40961 on the 2008 and 2003 DCs?
The hotfix looks like its only for 2003 servers.
I’m having a very similar issue, but it happens in the evening when I’m not RDP’d into the exchange or DCs.
Thanks for the post and any additional info.
JasonB, that is correct. The hot fix was for the 2003 Domain Controller which was the source of the problem. Seems to be fixing the issue for the others users in the post as well: http://community.spiceworks.com/topic/134941-exchange-2010-ad-topology-failures-all-domain-controllers-unavailable?page=5
Hello Scott,
I am very pleased i started that post, as i was pulling my hair out. Then i found our previously an authoratative restore had taken place on our domain, hence the need for the patch.
Thanks so much, Scott. Thats a tricky one to catch.
I only have windows server 2008 and 2008 r2 domain controllers no windows server 2003 domain controllers. The domain functional level is still windows server 2003.
Has anyone had a problem like this with only windows server 2008 domain controllers?
I’m not sure if there have been any authoritative restores at this company.
One of our clients is having a similar issue. The Exchange server loses connection to the Global Catalog servers in the domain and all email flow seizes. However, the same EVENTIDs listed here are logged on the Exchange server, not the DC’s. It’s a small environment with 1 Exchange 2010 server, 1 Server 2008 DC, and 1 Server 2003 DC. I wanted to make sure I’m understanding this post correctly. Everyone here had those corresponding errors logged on their DCs?
Hi Joe, Sorry for the late reply, I’ve been caught up in a few things.
I’m not certain regarding all 2008 domain controllers but by the looks of it that hotfix can be applied to a Windows 2008 Server.
Hi Jon,
No, those errors were logged on the Exchange Server. The Patch on the DCs resolved the communication issue between Exchange and DCs.
Scott,
Thanks!