Log in



Archive for March, 2010

Robocopy fails when coping to EMC Celerra

March 25th, 2010 by Scott

A colleague of mine sent me this interesting issue he ran into during a migration to EMC Celerra.  Thought I would share.

The Problem:

If a file exists on both the source and destination, by default Robocopy copies the file if the two files have different time stamps or different sizes. However, Robocopy fails to copy only modified data from an NTFS volume on a BXL server to the Celerra.

What was happening is that when he was doing a Robocopy with the /mirror or /E /purge options the Robocopy always found the source file newer then the destination file on the Cellera and copied it, even if a Robocopy was run a minute earlier. 

The reason this was happening was due to a variation in the EMC implementation of CIFS.  The NTFS date and time stamp is a 64-bit variable, which DART doesn’t deal with.  So Celerra uses the FAT timestamp format instead. 

The Resolution:

To get around this issue it is recommended to use the /FFT switch in RoboCopy.  /FFT switch stands for Fat Format Timestamps.  For more information on Robocopy switches go to http://ss64.com/nt/robocopy.html

In addition, I parameter can be set to make Celerra round up the time to the nearest second. This makes robocopy believe the files on Celerra have newer time and therefore does not copy the files.

[nasadmin@Harrison ~]$ server_param server_2 -f cifs -i nanoroundoff –v
server_2 :

name = nanoroundoff
facility_name = cifs
default_value = 0
current_value = 0
configured_value = 0
user_action = restart Service
change_effective = restart Service
range = (0,1)

description = Roundup the time information to the next second detailed_description Cifs protocol is able to set time using 64 bits value (time set in 100th of nano second since January 1, 1601 in Gegorian calendar. In Dart we do not have this granularity. This means time value read after set could be different (older). This parameter when set, is roundup the time value to the next second in order to not confused some application if the file is olderthan the time application has stored localy as the last update. Typical

application is profile storage on the DM. Setting this param improve performance at log off time, as the application will not try to flush again profile information seeing the last modification date is not older than the time info it has stored localy as the last modification date.

[nasadmin@Harrison ~]$ server_param server_2 -f cifs -m nanoroundoff -v 1
server_2 : done

Special Thanks to Keith for sharing this interesting tidbit.

Disappearing NetApp LUNs when moving to EMC?

March 10th, 2010 by Scott

I was at a client site yesterday in the process of moving them off a NetApp Storage array and onto EMC Storage.  My colleague and I ran into a very interesting problem and have no idea why it was happening.  If anyone has any ideas please, do share.

The setup was two Windows 2008 Servers with Exchange 2007 CCR split between two datacenters.  The servers were using Snap Manager for Exchange and Snap Manager.   There were four LUNs from the NetApp presented, three for databasae and one for the logs.  The LUNs were attached via iSCSI. 

Since the Exchange CCR environment was running in the HQ Datacenter we started by getting the EMC agents installon the the Server in the DR Datacenter.  The iSCSI LUNs from the NetApp were presented via Drive letters, there were four, three for the database and one for the logs.  If you’re not familar with deploying EMC apps the first app to deploy is the EMC NaviHostAgent.  When I installed the agent the server requested for a reboot.  This is where things got really funky.  When the server came back from a reboot the iSCSI disk drives from the NetApp were gone!  No idea why.  We scratched our heads and couldn’t figure out what happened.  According to the NetApp management console the LUNs were presented.  When we tried to remap the LUNs via Snap Manager it would state that the volume was already mapped to a drive on the server, but the Server could not see them!  This was odd.  We backed out the EMC NaviHostAgent and the LUNs reappeared but the drives were off line.  I brought the drives back on line and everything worked again.

We then started by reinstalling the EMC NaviHostAgent, no reboot was required this time and installed the EMCPowerPath application.  When we came back from a reboot the drives from the NetApp were off line again, I brought them back on line and had both the EMC and the NetApp storage provisioned.  Great, this is nice.  We presented the EMC LUNs as mount points and started up replication again from the HQ Exchange Server.  Once replication was caught up moved the cluster to the DR server to begin the same process on the HQ server thinking we had a fluke.

Well, the exact same thing happened!  When we installed NaviHostAgent no reboot was required.  We then installed PowerPath, Had to reboot the server and the NetApp LUNs vanished!  WTF right?  Not again!  We did the same procedure I mentioned above, removed PowerPath, removed NaviHostAgent, rebooted the Server in HQ.  When it came back the NetApp LUNs were gone!  No drive letters, no detection.  Checked the iSCSI Initator configuration, and it was perfect, it was seeing the iSCSI targets for both the NetApp and the EMC.  From the NetApps perspective the LUNs were provisioned.  What in Windows 2008 was preventing the LUNs from being seen?  I checked the Network settings via IPCONFIG and there was a “Local Area Connection 9″ that was there along with a Prod network and a iSCSI network.  Looking at the network configuration only two NICs were active for the machine.  No idea where the third NIC was coming from and by this time I was too tired to try to figure it out.  Anyone out there have thoughts on that one? 

Anyway, after troubleshooting for about 5 hours between the two servers I proposed a Plan B.  Since we had the NetApp storage (Thank God) and the EMC storage on the DR server I could create new Storage Groups and Databases on the DR server to the new EMC LUNs.  Create new EMC LUNs on the HQ Exchange Server to reflect what was in DR and let CCR take care of the rest.  Once the Storage Group and Databases were created I enabled replication only for the EMC Storage Groups (since the NetApp was off line on the HQ Server I couldn’t re-enable CCR) and started moving mailboxes.  This worked fine, disaster averted. 

I was glad Plan B worked, but more complexing is why did the NetApp LUNs disappear?  Anyone have an idea what may have happened with it?  Again this was a Windows 2008 Server with Exchange 2007.  The iSCSI network was correct, the NetApp was presenting the LUNs, when going into snap manager the LUNs said they were mapped.  I tried to do a rescan of disks, nothing.  Tried to take away the LUN and remap the LUN via NetApp management console and nothing worked.  The Windows Firewall was turned off, disabled AV Software, nothing worked.  Nothing would bring those LUNs back to the Windows 2008 server.  Has anyone seen anything like this?  Anyone have an idea on what happened?  Please do share!

Thanks,

Update Rollup 2 for Exchange Server 2010

March 5th, 2010 by Scott

The Microsoft Exchange Team has announced the release of Update Rollup 2 for Exchange Server 2010.  For the announcement go here: http://msexchangeteam.com/archive/2010/03/05/454155.aspx

Some key fixes are: 

  • KB 977633 This fixes IMAP4 clients ability to log on to their mailboxes if the mailboxes are located on Exchange 2003 backend servers and if the clients are connecting via Exchange 2010 CAS servers.
  • KB 979480 IMAPid was not working correctly after moving a lot of users from one Exchange 2010 server to another*. IMAP4 users complained about the inbox not being updated any more. Old messages were still visible, but messages which were received after the mailbox move were not visible. The problem affected different IMAP Clients. The problem did not affect MAPI clients and OWA. Now it is fixed up. *(Specifically this occurred in the situation with same DAG, now local storage instead of iSCSI storage, all servers are Exchange 2010 with Update Rollup 1 installed on Windows Server 2008 R2).
  • KB 979431 When user migrated from Exchange Server 2003 to Exchange Server 2010, and that user connected via POP3, the POP3 service crashed. This was fixed up so it will not crash.
  • KB 979563 Push Notifications didn’t work because Exchange Server 2010 was not sending SOAPAction header in the notify callback. This caused Exchange to receive a HTTP 500 response from the notification client and the webservice failed. Push notifications should now properly send that SOAP header.
  • KB 980261 We fixed passive page patching when diagnostic tracing code was needed for forensic analysis that was generating a -1022 error case.
  • KB 980262 Source side log copier errors are more gracefully handled when the log has a bad block and the read fails.
  • KB 979566 Activesync proxy was failing for linked mailboxes in a CAS to CAS proxy scenario where the users token is serialized and sent in the request. When attempting to create the client security context from the SID, a AuthZException was thrown because we did not have access to the token information of the linked account, so now for this it no longer throws exceptions.
  • For more information on the Hotfix you can go to the page at http://support.microsoft.com/?kbid=979611.

    Enjoy!

    Looking for a Good Skate Sharpener?

    March 2nd, 2010 by Scott

    If anyone out there is like me and getting skates sharpened weekly or twice a week I would encourage you to take a look at the Wissota Skate Sharpener.  I recently purchased one a few weeks ago and have enjoyed it ever since.  My son who plays hockey is getting his skates sharpened at leased once if not two times a week and the run to the skate shop not to mention the $5 a visit was getting costly. According to the Wissota Web site it costs about 17 cents, yes $0.17 to sharpen one skate.  Can you say savings?  Well, insert the Wissota Skate Sharpener.  The machine is pretty easy to use.  I think the hardest part for me was to locate the crown.  That’s the point where the wheel on the sharpener will hollow out a good radius.  Yes, I’ve been reading up on Skate Sharpening.  Once there I am now able to sharpen my skates and my sons skates any day and time and it takes only a few minutes.  Much better then dropping them off at the skate shop and waiting hours, if not a day…..

    Anyway, If anyone out there is looking for a good skate Sharpener I recommend the Wissota Skate Sharpener.  I was able to to go to the company and pick up mine but they do ship.  While I was there I got a great demonstration on how the machine worked and the Sales guy even gave me more instruction then I could possibly ask for. There is an instructional DVD included with the Machine along with a lot of extra pieces.  I didn’t have to buy any thing more to get the machine to work.

    So, if you are looking for a skate sharpener take a look at their web site, Wissota Manufacturing Company.  And no, I’m not getting paid for this or getting something for free.  I am happy with their product and wanted to recommend it to anyone out there Looking for a good Skate Sharpening Machine.

    Move Root Certificate Authority from Windows Server 2003 to Windows Server 2008

    March 2nd, 2010 by Scott

    Well, I’ve been trying to write this article for about a month now and finally had some time to sit down and type it out.  I was inspired by this article when I had a client request to move their Root Certificate Authority on a Windows 2003 Domain Controller to a new Windows 2008 Domain Controller.  To be honest, there really isn’t anything to it but the information I found out on the net wasn’t that great so I thought I would provide the world with some info on how to perform this process. 

    The Client setup involved a Windows 2003 domain controller that was acting up.  On this DC was their Root Certificate Authority for their entire Active Directory environment.  The client is small and does not have any special requirements for an Enterprise CA and wanted to move their CA to Windows 2008 Active Directory Certificate Services. 

    The key principles here are that we need to move the private key associated with the Root Certificate Authority and also the Certificate Authority Database.  When moving a certificate Authority we need to preserve the CA name in the environment, otherwise nothing will work!  The clients will not be able to locate the CA nor will the Root certificate match up with the certificates.  Things just won’t be trusted.

    To get started I reviewed the Support Article on How to move a certification authority to another server to backup the existing Windows 2003 Root CA Info.  I first used the Certificate Authority snap-in to backup the CA database and private key.  To perform the backup follow these steps:

    • In the Certification Authority snap-in, right-click the CA name, click All Tasks, and then click Back up CA to start the Certification Authority Backup Wizard.
    • Click Next, and then click Private key and CA certificate.
    • Click Certificate database and certificate database log.
    • Use an empty folder as the backup location. Make sure that the backup folder can be accessed by the new server.
    • Click Next. If the specified backup folder does not exist, the Certification Authority Backup Wizard creates it.
    • Type and then confirm a password for the CA private key backup file.
    • Click Next, and then verify the backup settings. The following settings should be displayed:
    • Private Key and CA Certificate
    • Issued Log and Pending Requests
    • Click Finish.

    Next we have to save the registry settings.  To save the registry settings perform the following:

    • Click Start, and then Run.  In the Run field type regedit and click Ok
    • Locate and then right-click the following registry subkey, HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration (While you are here, why not take a look at the settings, take a screen shot, make sure they match up in the end)
    • Click Export
    • Save the Registry file in the CA Backup folder that was defined above

    Now that we have the database, certificate and registry backed up the next step was to remove Certificate Services from the old computer.  This process is pretty straight forward.  Go into the Control Panel, Add/Remove Programs, Windows Components and remove the Tick from Certificate Authority.  Note Be sure to remove the Certificate Authority from the old computer prior to deploying Certificate Services on the new machine.  If you deploy AD CS first the target CA will become unusable. 

    Finally, rename the old server or permanently disconnect it from the network. 

    In the step above I took the existing Domain Controller, removed the Certificate Services from it and then performed a DCPromo to remove Active Directory from the computer.  Once the computer was no longer a domain controller I renamed the old server.  I wanted to keep the server online for a fail back just in case, which wasn’t necessary since the move went over successfully!

    Now, looking at where we stand right now I had the database, the Private Key and the certificate authority database backed up.  The data I backed up above should be copied to the new server that will be used for Active Directory Certificate Services.  This will need to be imported below. Now, the next step is to deploy Active Directory Certificate Services on the Windows 2008 domain controller.  BTW I should point out that when deploying Active Directory Certificate Services that you should use Windows 2008 Enterprise edition.  W2K8 Enterprise gives you more functionality of your Certificate Services.  For a list of features in Windows 2008 Standard vs Windows 2008 take a look at this link: Active Directory Certificate Services Step-by-Step Guide.  If you scroll down a bit you will see a comparison chart which will note which features are available with which version of Windows you use. 

    Now, let’s move on to the part where we deploy and restore the Certificate Services.   Log on with local or enterprise administrator permissions to the CA computer and perform the followign:

    • Launch the Service Manager for Windows 2008. 
    • In the console tree, click Roles.
    • On the Action menu, click Add Roles.
    • If the Before you Begin wizard appears, click Next.
    • In the list of available server roles, select the Active Directory Certificate Services check box, and click Next twice.
    • Make sure that Certification Authority is selected, and click Next. (Note: If you are going to use Web Enrollment make sure to check this box.  You can always add it later but Why not add it now?  All the required roles will also be installed when you check this box since you will get a list of Add role service required)
    • Select Enterprise and click Next.  (We are doing this because this is an Enterprise Root CA that will integrate with Active Directory.  Just like the one I decommissioned.  Best practice is to have a Standalone Root CA but given the size of this organization they are not too concerned with having a Standalone Root CA.)
    • Specify Root  and click Next.  (If the CA you’re moving from was a Subordinate CA then we would want to tick the Subordinate CA option.  But since in my example this is a Root CA we are sticking with root.  Keep in main that if you’re coming from a Root CA or a Subordinate CA this option must match with what you’re coming from.)
    • At this stage, you have a choice between creating a new private key or using an existing private key.  For a migration, on the Set Up Private Key page, select Use existing private key and choose Select a certificate and use its associated private key.

    You should have something that looks like this:

    ADCSPic

    Click Next and continue the steps below:

    • If the CA certificate we backed up above has been installed on the computer, it will be listed in the Certificates box. Otherwise, click Import to import a certificate from the .pfx file created by exporting the CA certificate and private key from the source CA.
    • Click Browse, and locate and select the file containing the certificate and private key exported from the source CA.
    • Enter the password you selected when exporting the CA certificate and key from the source CA, and click OK.  Select the Certificate that was just imported and click Next
    • When choosing your path you can either use defaults or browse to new ones.  Once done click Next
    • Complete the installation of the AD CS
    • Click Yes to accept the warning to overwrite AD DS. (This appears only if you are installing an enterprise CA.)

    Congratulations, you’re almost there!  We have deployed Active Directory Certificate Services on Windows 2008.  There are still two more steps that must be completed.  This is the process of restoring the Certificate Authority Database that was backed up in the first section and restoring the registry component. 

    To restore the registry simply locate the registry value that was saved above, right click the file and select merge.  This will import the Registry settings to the W2K8 server.  Next we have to restore the database.   You can check to make sure the settings were imported correctly by going to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CertSvc\Configuration and verify your settings are there.  (Remember that screen shot?)

    To restore the database and log files perform the following:

    • Open Server Manager on the Windows 2008 Server.
    • Expand Roles and then Expand Active Directory Certificate Services.
    • Locate the name of the CA you just deployed.
    • Right Click the CA name and select Restore CA…
    • You will get a warning message that the AD CS cannot be running to perform this action.  Simply click Ok to stop AD CS.  AD CS will begin to stop
    • On the Wizard click Next
    • On the Items to Restore screen check the box Certificate database and certificate database log only.  Click Browse to locate the database that was copied over above.  (Note: I need to point out here that you select the folder you backed up to.  i.e. if you backed up the database and logs to C:\Temp\CABackup then this will be the folder you will restore from.  The backup process will create a subdirectory that it will look for during Restore, if you go one folder too deep the restore will fail.)  Once you have located your backup click Next.
    • On the completion screen click Finish and the restore will begin. 
    • Once the restore is complete you will receive a action box that asks if you would like to restart the AD CS.  Simply click Yes.  (We shouldn’t have any incremental backups since we are doing a migration.)
    • Once the AD CS service is restarted we are good to go!

    Well, what do you guys think?  Worth the effort?  Migrating to W2K8 AD CS will help your CA remain alive much longer.  During this process I also had to renew the CA Certificate which was pretty much easy. 

     I hope this article will help someone out there, I know I was able to get through it but had to go to a couple of different sources to get the exact process down.

     Enjoy!