Log in



Tags » ‘Active Directory’

Exchange 2010 Servers lose connectivity with Global Catalog Servers in Domain. Event IDs logged 2102, 2103, and 2114.

November 9th, 2011 by

Ok, I’m sorry, I’ve been extremely busy with work and family.  The following article is one I’ve been working on since September of this year!  So here it is….

Over the past nine months I have been battling an issue in Exchange 2010 that reported “All Global Catalog Servers in forest DC=ScottFeltmann,DC=com are not responding.  I feel that I can finally say the issue has been resolved!

Some background:

This was a large Exchange deployment, two active directory sites, two Hub/CAS and two mailbox Servers in the primary site running DAG.  Secondary site had two Hub/CAS and one Mailbox Server.  The focus of the issue seemed to take place in the primary site.    The issue we were seeing initially is that users would hang, literally hang, both in Outlook or OWA.  It seemed that the Client Access Server (one of the two) was hanging, as if waiting for a response.  Rebooting the server always cleared up the issue but this was by no means a solution.

Now, what made this problem difficult is that it occurred randomly and about every three to four weeks.  There was no way I could recreate the problem and could only trouble shoot the issue when the problem was occurring.  After my initial troubleshooting and seeing the issue occur randomly.  The primary Errors in the Event Log were Event ID: 2102, Event ID: 2103, and Event ID: 2114.

Event 2102 Info:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          3/2/2011 6:03:04 AM
Event ID:      2102
Task Category: Topology
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CAS1.scottfeltmann.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1364). All Domain Controller Servers in use are not responding:
DCAuth1.scottfeltmann.com
DCMAIN3.scottfeltmann.com
DCMain3008.scottfeltmann.com
DCMain2.scottfeltmann.com
DCSC1.scottfeltmann.com
DCSC2.scottfeltmann.com
DCMain1.scottfeltmann.com

Event 2103 Info:

Log Name:      Application
Source:        MSExchange ADAccess
Date:          3/11/2011 10:46:21 AM
Event ID:      2103
Task Category: Topology
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CAS1.scottfeltmann.com
Description:
Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1524). All Global Catalog Servers in forest DC=ScottFeltmann,DC=com are not responding:

DCSC1.scottfeltmann.com
DCSC2.scottfeltmann.com
DCMain1.scottfeltmann.com
DCMain2.scottfeltmann.com
DCAuth1.scottfeltmann.com
DCMAIN3.scottfeltmann.com
DCMain3008.scottfeltmann.com

Event 2114 Info:

Error:
Log Name:      Application
Source:        MSExchange ADAccess
Date:          4/14/2011 10:48:30 AM
Event ID:      2114
Task Category: Topology
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      CAS2.scottfeltmann.com

Description:

Process MSEXCHANGEADTOPOLOGYSERVICE.EXE (PID=1480). Topology discovery failed, error 0×80040952 (LDAP_LOCAL_ERROR (Client-side internal error or bad LDAP message)). Look up the Lightweight Directory Access Protocol (LDAP) error code specified in the event description. To do this, use Microsoft Knowledge Base article 218185, “Microsoft LDAP Error Codes.” Use the information in that article to learn more about the cause and resolution to this error. Use the Ping or PathPing command-line tools to test network connectivity to local domain controllers.


I decided to call Microsoft support as I had never seen this type of behavior in any of my deployments.  To be honest it came down to about 5 calls into MS Support, rare case which happens once every few weeks = difficult to trouble shoot, even with Microsoft.  I spoke to both Exchange support and Active Directory support engineers.  I could go into great detail about the trouble shooting steps I took with MS support but none of these resolved the issue so what’s the point.  To give you an idea we tried to reset the Kerbos ticket, we ran DCDiag, checked Domain controller Replication, checked DNS, nothing.  Long story short after four or five calls with MS Support I finally was able to get escalated to level 3 or 4, either way, Rick (the MS Support Engineer) figured out what the issue was.

What Rick noticed was a Warning in the System Event Log, Event ID 40961 from the source LsaSrv.  This Error was never reviewed with other MS Support people as I’m not sure they thought it wasn’t a bad issue or not.  To be honest, a warning isn’t something I really turn my attention to when a bad situation is occurring (learned my lesson there).  Anyway, here is the event info

Event ID 40961 Info:

Log Name:      System
Source:        LsaSrv
Date:          4/14/2011 10:53:30 AM
Event ID:      40961
Task Category: None
Level:         Warning
Keywords:
User:          SYSTEM
Computer:      CAS2.scottfeltmann.com
Description:

The Security System could not establish a secured connection with the server ldap/ DCMain3008.scottfeltmann.com/scottfeltmann.com@SCOTTFELTMANN.COM. No authentication protocol was available.


Rick mentioned he sees this error when systems are having issues with Remote Desktop.  I couldn’t help to think but how does RDP tie into my problem with Exchange.  The way it ties in is because the secure channel between the source and the target is lost and causes the systems not to be able to authenticate.  Basically the CAS was trying to communicate with the DC while the user was waiting.  The DC and CAS didn’t trust each other and bam, CAS HUNG!

What is the fix?  After about 9 months of issues and calls with MS at random times, there is a hotfix, drum roll….. KB939820!  Which is located here:  http://support.microsoft.com/kb/939820

Once this hotfix was applied the problem stopped occurring, the Event Logs no longer showed a 2102, 2103, or 2114 error and life was good again!

While researching the issue I also found a posting on spiceworks where users were having the same problem.  Here is the URL for that: http://community.spiceworks.com/topic/134941-exchange-2010-ad-topology-failures-all-domain-controllers-unavailable?page=4#entry-847654

Talk about Crazy!   I’m just glad it is over.

Hope this helps out there!

Comments? Questions?  Please Share!

 

 

 

 

Bug identified when Deploying Exchange 2010 in a domain with a ‘.’ in the NETBIOS name! (KB981033)

April 22nd, 2010 by

I was starting to deploy Exchange 2010 for a client today that contained a ‘.’ in their NETBIOS name when I came across a bug.   The majority of organizations will deploy an Active Directory environment using a single name for their NETBIOS domain name.  For example, ‘scottfeltmann.com’ would have a NETBIOS name of ‘scottfeltmann’ or ‘contoso.com’ would be ‘contoso’.  

I started out by running the Exchange 2010 Pre-Deployment Analyzer Tool.  No issues discovered.  I then deployed the Client Access Server role. No errors were reported during the installation of the CAS.  Upon the completion of the installation of the role I opened up the Exchange Management Console (EMC) clicked on the ‘Microsoft Exchange On-Premises’ option and got the following error:

Initialization failed

 “The following error occurred when getting user information from ‘domain.com\username’: 

The operation couldn’t be performed because the object ‘domain.com\username’ couldn’t be found on domaincontroller.domain.com.  It was running the command ‘Get-LogonUser’.

How odd, this isn’t normal.  I haven’t seen this in any other environment, but then again, I haven’t come across a netbios name with a ‘.’ in it.  At first I checked AD for any issues, thinking maybe there was a replication problem.  I did find one small problems, resolved it, tried again, same result. 

I then turned to my friend google. 

Doing some research I came across this article with the same problem: http://www.eggheadcafe.com/software/aspnet/35384695/exchange-2010-period-in-n.aspx  a ‘.’ in the domain NETBIOS name.  Doing further digging I came across this article on StuartP’s blog which describes the issue: http://blogs.technet.com/stuartp/archive/2010/03/31/exchange-2010-and-netbios-domains-with-dotted-notations.aspx

Yup, looks like an identified bug in Exchange 2010 and it will not be fixed until Rollup 4.  I did install Rollup 3 (which was released April 9th) to see if that would help the situation but it didn’t.  I guess at this point it is a matter of waiting for Rollup 4 to be released…..Bugger!

—**UPDATE**—

Working with MS support I was informed that there is an unpublished KB article pertaining to this issue.  If you do have this problem you will need to contact MS Support and refer to KB 981033.  It is my understanding that there is a fix for Exchange 2010 RU 1 and Exchange 2010 RU 3,  I received the update for RU 3 and will be applying it to the server later today.  The size of the update was approx. 35megs. 

I need to point out that when installing this update you will need to remove it prior to deploying RU 4, which will have the fix in the RU 4 release.

Thanks MS Support for getting this guy out here!

OCS/LCS and Windows 2008 R2 Active Directory

April 6th, 2010 by

If you’re thinking about upgrading your existing Active Directory Forest and Domain to Windows 2008 R2 you need to consider what applications you currently have running.  There have been a number of issues relating to deploying Windows 2008 R2 Domains Controllers. 

I was looking over on MS Support and noticed that OCS 2007, OCS 2007 R2 and LCS are now supported on Windows 2008 R2 Domains.  The trick of it though is if you already have OCS deployed you will need to rerun the forest prep utility.  For some reason when deploying the R2 schema and promoting to a DC the OCS schema information gets removed.

How odd but at least the fix is simple.  So in the event you have OCS or LCS deployed and decide to move to Windows 2008 R2 AD keep in mind that you will need to rerun the the forestprep for OCS or use the Lcscmd command “Lcscmd.exe /forest /action:ForestPrep”  which will work for LCS or OCS.

For more information see the following articles: 

Office Communications Server 2007 R2, OCS 2007 or LCS 2005 does not work correctly after you upgrade to Windows Server 2008 R2
and
Supportability is available for Office Communications Server 2007 R2 member server role on a Windows Server 2008 R2 operating system

 

Move Root Certificate Authority from Windows Server 2003 to Windows Server 2008

March 2nd, 2010 by

Well, I’ve been trying to write this article for about a month now and finally had some time to sit down and type it out.  I was inspired by this article when I had a client request to move their Root Certificate Authority on a Windows 2003 Domain Controller to a new Windows 2008 Domain Controller.  To be honest, there really isn’t anything to it but the information I found out on the net wasn’t that great so I thought I would provide the world with some info on how to perform this process. 

The Client setup involved a Windows 2003 domain controller that was acting up.  On this DC was their Root Certificate Authority for their entire Active Directory environment.  The client is small and does not have any special requirements for an Enterprise CA and wanted to move their CA to Windows 2008 Active Directory Certificate Services. 

The key principles here are that we need to move the private key associated with the Root Certificate Authority and also the Certificate Authority Database.  When moving a certificate Authority we need to preserve the CA name in the environment, otherwise nothing will work!  The clients will not be able to locate the CA nor will the Root certificate match up with the certificates.  Things just won’t be trusted.

To get started I reviewed the Support Article on How to move a certification authority to another server to backup the existing Windows 2003 Root CA Info.  I first used the Certificate Authority snap-in to backup the CA database and private key.  To perform the backup follow these steps:

  • In the Certification Authority snap-in, right-click the CA name, click All Tasks, and then click Back up CA to start the Certification Authority Backup Wizard.
  • Click Next, and then click Private key and CA certificate.
  • Click Certificate database and certificate database log.
  • Use an empty folder as the backup location. Make sure that the backup folder can be accessed by the new server.
  • Click Next. If the specified backup folder does not exist, the Certification Authority Backup Wizard creates it.
  • Type and then confirm a password for the CA private key backup file.
  • Click Next, and then verify the backup settings. The following settings should be displayed:
  • Private Key and CA Certificate
  • Issued Log and Pending Requests
  • Click Finish.

Next we have to save the registry settings.  To save the registry settings perform the following:

  • Click Start, and then Run.  In the Run field type regedit and click Ok
  • Locate and then right-click the following registry subkey, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration (While you are here, why not take a look at the settings, take a screen shot, make sure they match up in the end)
  • Click Export
  • Save the Registry file in the CA Backup folder that was defined above

Now that we have the database, certificate and registry backed up the next step was to remove Certificate Services from the old computer.  This process is pretty straight forward.  Go into the Control Panel, Add/Remove Programs, Windows Components and remove the Tick from Certificate Authority.  Note Be sure to remove the Certificate Authority from the old computer prior to deploying Certificate Services on the new machine.  If you deploy AD CS first the target CA will become unusable. 

Finally, rename the old server or permanently disconnect it from the network. 

In the step above I took the existing Domain Controller, removed the Certificate Services from it and then performed a DCPromo to remove Active Directory from the computer.  Once the computer was no longer a domain controller I renamed the old server.  I wanted to keep the server online for a fail back just in case, which wasn’t necessary since the move went over successfully!

Now, looking at where we stand right now I had the database, the Private Key and the certificate authority database backed up.  The data I backed up above should be copied to the new server that will be used for Active Directory Certificate Services.  This will need to be imported below. Now, the next step is to deploy Active Directory Certificate Services on the Windows 2008 domain controller.  BTW I should point out that when deploying Active Directory Certificate Services that you should use Windows 2008 Enterprise edition.  W2K8 Enterprise gives you more functionality of your Certificate Services.  For a list of features in Windows 2008 Standard vs Windows 2008 take a look at this link: Active Directory Certificate Services Step-by-Step Guide.  If you scroll down a bit you will see a comparison chart which will note which features are available with which version of Windows you use. 

Now, let’s move on to the part where we deploy and restore the Certificate Services.   Log on with local or enterprise administrator permissions to the CA computer and perform the followign:

  • Launch the Service Manager for Windows 2008. 
  • In the console tree, click Roles.
  • On the Action menu, click Add Roles.
  • If the Before you Begin wizard appears, click Next.
  • In the list of available server roles, select the Active Directory Certificate Services check box, and click Next twice.
  • Make sure that Certification Authority is selected, and click Next. (Note: If you are going to use Web Enrollment make sure to check this box.  You can always add it later but Why not add it now?  All the required roles will also be installed when you check this box since you will get a list of Add role service required)
  • Select Enterprise and click Next.  (We are doing this because this is an Enterprise Root CA that will integrate with Active Directory.  Just like the one I decommissioned.  Best practice is to have a Standalone Root CA but given the size of this organization they are not too concerned with having a Standalone Root CA.)
  • Specify Root  and click Next.  (If the CA you’re moving from was a Subordinate CA then we would want to tick the Subordinate CA option.  But since in my example this is a Root CA we are sticking with root.  Keep in main that if you’re coming from a Root CA or a Subordinate CA this option must match with what you’re coming from.)
  • At this stage, you have a choice between creating a new private key or using an existing private key.  For a migration, on the Set Up Private Key page, select Use existing private key and choose Select a certificate and use its associated private key.

You should have something that looks like this:

ADCSPic

Click Next and continue the steps below:

  • If the CA certificate we backed up above has been installed on the computer, it will be listed in the Certificates box. Otherwise, click Import to import a certificate from the .pfx file created by exporting the CA certificate and private key from the source CA.
  • Click Browse, and locate and select the file containing the certificate and private key exported from the source CA.
  • Enter the password you selected when exporting the CA certificate and key from the source CA, and click OK.  Select the Certificate that was just imported and click Next
  • When choosing your path you can either use defaults or browse to new ones.  Once done click Next
  • Complete the installation of the AD CS
  • Click Yes to accept the warning to overwrite AD DS. (This appears only if you are installing an enterprise CA.)

Congratulations, you’re almost there!  We have deployed Active Directory Certificate Services on Windows 2008.  There are still two more steps that must be completed.  This is the process of restoring the Certificate Authority Database that was backed up in the first section and restoring the registry component. 

To restore the registry simply locate the registry value that was saved above, right click the file and select merge.  This will import the Registry settings to the W2K8 server.  Next we have to restore the database.   You can check to make sure the settings were imported correctly by going to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration and verify your settings are there.  (Remember that screen shot?)

To restore the database and log files perform the following:

  • Open Server Manager on the Windows 2008 Server.
  • Expand Roles and then Expand Active Directory Certificate Services.
  • Locate the name of the CA you just deployed.
  • Right Click the CA name and select Restore CA…
  • You will get a warning message that the AD CS cannot be running to perform this action.  Simply click Ok to stop AD CS.  AD CS will begin to stop
  • On the Wizard click Next
  • On the Items to Restore screen check the box Certificate database and certificate database log only.  Click Browse to locate the database that was copied over above.  (Note: I need to point out here that you select the folder you backed up to.  i.e. if you backed up the database and logs to C:\Temp\CABackup then this will be the folder you will restore from.  The backup process will create a subdirectory that it will look for during Restore, if you go one folder too deep the restore will fail.)  Once you have located your backup click Next.
  • On the completion screen click Finish and the restore will begin. 
  • Once the restore is complete you will receive a action box that asks if you would like to restart the AD CS.  Simply click Yes.  (We shouldn’t have any incremental backups since we are doing a migration.)
  • Once the AD CS service is restarted we are good to go!

Well, what do you guys think?  Worth the effort?  Migrating to W2K8 AD CS will help your CA remain alive much longer.  During this process I also had to renew the CA Certificate which was pretty much easy. 

 I hope this article will help someone out there, I know I was able to get through it but had to go to a couple of different sources to get the exact process down.

 Enjoy!

My Client Visit Yesterday Part 1

February 11th, 2010 by

So, I was out at a client site yesterday to review the work they have completed so far on their Exchange 2007 deployment.  There were two issues that we came across that were unique that I thought I would mention to the population out there.  Hopefully this information will be able to help someone.

The first problem I noticed was the error message that stated Exchange was not part of an active directory site.  This caused me to think, why would an error come up like this.  My first instinct was to do a Gpresult /r (Windows 2008) which listed out all the information about the computer and user.  The Computer said it belonged to the site Default-First-Site-Name.  Ok, so the server was recognizing that it was a member of the AD Site, but why was exchange balking at the issue?  Well, I asked the IT Admin to open up Active Directory Sites and Services and took a look in there.  Looking over AD Sites and Services I noted the client had only one site configured, Default-First-Site-Name.  Thinking a little bit more about the situation I asked to see what subnets were configured for the site.  Well, upon review the site only had two subnets assigned to it.  Neither one of these Subnets included the subnet Exchange 2007 was in.   Talking to the Admin about this I learned that they had the same issue on the Hub Transport Servers and had to manually configure AD using ADSIEdit to insert the proper site name for the server to use!  Eww, not sure how this will impact their environment in the long run but when installing Exchange this should all be done automatically.  So, I had the admin add the Subnet to the AD Sites and Services and rebooted the mailbox servers, error gone, problem solved! 

The moral of the story above?  Make sure that you have your AD Sites and Services properly configured prior to deploying Exchange.  Oh yea, don’t forget, you need to have a domain Controller in the same AD Site as Exchange.  How the client ever got Exchange working is beyond me.  The workstation was seeing the AD Site but Exchange was not, hence the error. 

Not able to create new databases in Exchange 2010

January 4th, 2010 by

I was reading through the message forums when I came accross an interesting post regarding unable to create a Exchange 2010 database. 

The error posted is:

Active Directory operation failed on “server”. This error is not retriable. Additional information: The name reference is invalid.

This may be caused by replication latency between Active Directory domain controllers.

 

This issue may be a result of two things:

  

First it could be related to latency in Active Directory replication.  This may be a result where you have a slow site link or your replication interval may be too high. 

  

Another possibility is that the Configuration Domain Controller and the Exchange 2010 preferred Global Catalog are different.  What occurs is that the new database object is not replicated quickly enough and AD cannot verify the new database creation.

  

In order to work around scenario one you can take a look at Troubleshooting Active Directory Replication Problems

http://technet.microsoft.com/en-us/library/cc738415(WS.10).aspx

  

However, before doing that I would recommend running to following command to set the Configuration Domain Controller and the Preferr3ed Global Catalog Server:

  

Set-AdserverSettings -ConfigurationDomainController “Servername” -PreferredGlobalCatalog “ServerName” -SetPreferredDomainControllers “Servername”

  

Where the “ServerName” is would be the name of your Domain Controller, either the FQDN or the Netbios name will do.  It is recommended to setup the Configuration Domain Controllers and the Preferred Global Catalog on the same server. 

  

The Set-AdServerSettings cmdlet is used to manage the Active Directory Domain Services (AD DS) environment in the current Exchange Management Shell session.  For more information on the Set-AdServerSettings command goto http://technet.microsoft.com/en-us/library/dd298063.aspx.