VSS Question

I was asked about Exchange backups today and after looking at some documentation I remembered that with Exchange 2003 there is a VSS backup option. For the life of me I can't think of anyone who has used it with their Exchange backups. Is there anyone out there who has attempted to use, or is using VSS with their Exchange TDP? I have never trusted it enough to rely on it for my Exchange backups. It has enough problems with the system objects, but since MS has had some time to work out the kinks has anyone been successful with it? Is it worth the hassle?

Windows VM Restore Issue

I recently ran into a problem where a number of Windows 2003 VM's would not restore. The VM host machine crashed and the Admins brought it back up on an alternate server. They recreated the VM's but when they would attempt to do a Point-In-Time restore they would fail with a File Not Found error. There was no ANS message. I think it had something to do with Active Directory or DNS, because when I used the VIRTUALNODENAME setting the restore worked fine. Anyone seen this or have insight into it?

Restoring Windows Shares

This is a repost from June 2005

Well, it inevitably happens that a large fileserver is being replaced, goes down, or goes bad and you have to restore it. Well if you are restoring to the same type of hardware then life is beautiful and NT/2000/2003 has no problems and acts like the good little boy it should be, but what happens when you have to restore to new hardware, or are refreshing the server to a newer more powerful server? This is when NT can be worse than that bratty little kid I wanted to strangle in the movie Problem Child. Personally John Ritter should have shot him and buried him in the back yard, but I digress. So how can you restore Shares or any other piece of the registry with TSM when NT doesn't like having the registry of another machine of different hardware restored? Well that's were a little ingenuity and some preemptive strategy comes in handy. The easiest way is to setup a script using the command line registry tool REG.EXE that runs a REG EXPORT KEYNAME each day and stores it as a file. Here is the string needed to backup the Shares key. (The following is all one line)

REG EXPORT HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares C:\BACKUP\SHARESEXP.TXT

The command backs up the Shares key to a folder I created on the C: drive called Backup, but you can put it wherever you like. A simple batch file will suffice to execute the export and you can either schedule it through Windows daily or through TSM. Yes, you can do that and TSM will gladly back it up. This allows you to then restore the export file and reimport it into the server of choice. Just make sure that with any schedule where you are executing a command/batch file you put the full path of the file or else TSM can possibly fail to execute it, even if its in the TSM BACLIENT directory.

Ok, not everyone has that process in place and needs to restore the Share key now! Well have no fear TSM can help you accomplish it. Let's look at the process for this situation and I'll provide some links. First off you can restore the registry without activating it. The command to do so is

RESTORE REGISTRY -ACTIVATE=NO (for Win2K/Win2003)

RESTORE REGISTRY ENTIRE -ACTIVATE=NO (for WinNT)

Now for each process you can restore specific keys but for ease of use the entire restore will work. TSM will restore the registery to the c:\adsm.sys\computername\ directory. From this point you can load the hive in RegEdit containing the share key and export it to a file. Once exported you should be able to import it into the current registry thereby restoring the shares. Here is a Microsoft document that covers loading the hive and processes to use. There is also a good document on backing up and restoring the registry here. If you have any questions feal free to comment and I'll respond as soon as possible.

Intro and VSS information

Hello to all you readers of TSMEXPERT out there. My name is Jared and I've been ask to join by my good friend Chad (who is leaving the job he helped me get!) and post some of the Windows stuff that I've been doing with TSM lately. I put together a PPT presentation that talks about VSS and TSM. It might not be that helpful but it was useful to our team here. I have spent many hours trying to fix various TSM and VSS issues over the past couple years and this is a result of that work. I will warn you that some of this is "editorialized" and those parts should be rather obvious.

Without further ado, click here to download it.

System State & Services Backup Issue

I had a problem recently with McAfee and TSM not playing nice together. Jared Annes (co-worker) figured out the problem and this is his finding. What happened was that changes to the default scanning and file protection procedures were made to McAfee to “lock down” the servers from viruses. Unfortunately the SA’s inadvertently caused TSM to fail every system state and system services backup.  The culprit was one file, TFTP.EXE in the System32 folder due to McAfee locking the file from being read. What I didn’t expect was that a failure of one file would cause TSM to fail on the whole system state and system services backup. I forgot the system state and services are looked at as one object instead of multiple files. The fix was to disable McAfee, then delete TFTP.EXE from the hidden dllcache folder, also from the System32 folder, and then re-enable McAfee. Since then I have had no more troubles with the system state and services backups.

NOTE: If you don’t delete the file from the dllcache folder Windows will copy it back over to the System32 folder recreating the problem you were trying to fix.

All Hail The Java GUI!

Interesting resolution to a restore I just had thought I would pass it on... On one of our Win2K3 servers all admin accounts were crippled due to some corruption in the registry (my vote was on Gremlins!). So they wanted to restore the system state to a week ago.  I tried and the client core dumped due to our permissions being crippled.  So I tried one other thing and that is the Java GUI.  I figured since it runs under the System Account it might work.  Well it did and fixed their problems....I had not used the Java GUI in ages but here is one situation where it came in quite handy.

TSM Express

I am awaiting the release of TSM Express and was wondering why it took so long to make a SMB version of TSM?  This segment is huge and a version that plays into the common Full + Incremental has been needed for some time. I have numerous sites where I could use Express and it will allow me to standardize on TSM as the backup tool. For those not aware TSM Express is a new TSM version due out in the 2nd quarter of 2006 that is very similar to Arcserve and BackupExec. It is a Windows only product that uses a log and DB although the DB has a 20GB limit. It follows the full plus incremental process most tools use in this segment and is suppose to be the new small business release. I would say it was a medium size business solution, but with a 20GB DB size limit that would be hard to support, especially in a Windows environment. I found this presentation through Google that discusses TSM Express along with TSM HSM. It’s worth a look.

System State Backup Issue

We recently had to rebuild a box (thank goodness it was not production) and found that the system state and objects had not backed up.  Stranger still there was no error stating they failed. In fact the only error was that the LVSA directory was not set and it would not be used during the backup.  Why would a failure to completely setup the LVSA affect the system objects and state? TSM doesn't require the LVSA to back them up.  We checked multiple systems and found this happening on more than one.  The client level we are on is 5.2.4 and higher.  Has anyone else experienced this?  BTW - The backup schedules were showing in the event log as COMPLETED not Failed.

King Of All Backups! (AKA LAN-Free to Disk)

About a year ago we were tasked to setup a large multi-clustered Exchange server and provide the best possible backup and restore performance.  After much debate and research we decided on using LAN-Free to disk.  The system was a 7 node Windows 2003 cluster connected to an SAN disk array (I can’t remember if it was EMC or Dell). The first 5 nodes were Exchange servers, the 6th was the failover node and the 7th was turned into a TSM server.  The TSM server instance had 5 500GB secondary disks assigned to it for the backup of the five Exchange servers.  These five disks would be mapped one to each Exchange server allowing for the backup to occur across the SAN to the disks owned by the TSM server. To utilize the LAN-Free to disk capability we had to install Tivoli’s SANergy product.  SANergy is no longer a separate product but is now part of a TDP/Agent type install package for TSM.  We actually installed and configured SANergy first, which was easier than it seemed in the directions, then mapped the drives.  When configured with SANergy the mapped drives become accessible across the SAN as long as the clients are on the same disk SAN fabric.  So we now had mapped SAN-accessible drives and could backup the Exchange servers to disk using the FILE device class.  The FILE device class is the device class used since TSM does not support LAN-Free backups to diskpools at this time.  The FILE device class works like a virtual tape and it was configured to migrate the data a few hours before the next backup would occur, or when the storage pool reached a specific usage threshold.  The reason for this was to allow almost a 24 hr. timeframe for a restore and along with the new Exchange 2003 restore capabilities internally; it provided a high performance backup/restore solution.  We tested the backups against a 360GB DB and backed it up in 90 minutes.  People were impressed, but they wanted to see how it performed on restore. We then restored the same amount, 360GB, in 91 minutes. WOW!  It was amazing to see those numbers (68MB/s).  We even tested it with the failover node by mapping all 5 SANergy defined drives to the failover node and still saw the same numbers. We had everything ready to go when the account decided they wanted to go in another direction.  Weeks spent configuring and implementing the solution all for not!  At least I have the experience and know it works.  So if anyone is looking to do LAN-Free to disk it works, it’s fast, it takes a lot of admin work, and it will be a good solution for anyone looking for a high performance backup/restore environment.

Restoring An Image Backup To A Larger Disk

Well just the other day my group was asked to help the NT System Admins with the combination of two large disks into one. Both of the disks have been performing image backups so we decided that the largest image would be restored first then the second drive would be restored to the new disk normally. To combine the disks the SA’s made one huge partition (400+ GB) and an SA on my team started the image restore. The following warning was issued when the image was initiated:

***************************** WARNING ********************************
The destination volume is larger than the source volume.  This will reduce the file system '\\machinename\x$' size to '    xxx.xx MB'.
Do you wish to continue? (Yes (Y)/No (N))

This was due to the target disk being larger than the image. We figured the image would not affect the space unused but we were wrong. After the image was restored explorer saw the disk as the size of the image backup but the Windows Disk Management tool saw it as the 400+ GB it was. So the problem was how to recover the disk.  Here is what MS support instructed and the problem was resolved.

When doing an image restore:

If the destination volume is larger than the source, after the restore operation you will lose the difference between the sizes. If the destination volume is located on a dynamic disk the lost space can be recovered by increasing the size of the volume. This will also increase the size of the restored volume.

We also have the following knowledge base doc:

When a Tivoli Storage Manager (TSM) image restore is performed to a target volume that is larger in size than the original volume (from which the image backup was taken), TSM will concatenate the target volume down to the original volume size.  However, within Windows Disk Management (within the Administrative Tools -> Computer Management utility), the disk will appear to still exist as the original larger volume size, even though Windows Explorer will show that the real volume size is now smaller.  In essence, this means that space has been lost, since it is not accessible on the drive and not available for use.  To prevent the loss of this space, the same volume size should be used on the target volume as on the original volume.

As far as I know this fix only works if the disk is dynamic (logical) and not a fixed (physical) disk. So the fix was to increase the disk size again to help it see the space again. I don’t think you need to increase it much but once done it resolves the issue. So if you happen to restore an image backup to a larger partition than it originated from just be aware of what you’ll need to do to reclaim your unused space after the restore finishes.

Question On Image Backups

The image backup feature is a wonderful piece that really saves you when restore time is of the essence, but what do you do when SA’s decide to combine two drives into one and want to use TSM to do it? Well we decided to use the image from the larger drive then do a standard restore of the other drive afterwards. Since I have never done this and don’t restore images frequently something I was alerted to was the message from TSM that states that file system would be changed (the drive was now bigger for the combining of the data). Will the image resize the partition? I don’t recall TSM doing that but then again I can’t remember. Anyone who has done a lot of image backups (particularly to larger drives than what the image came from) input would be appreciated.

CHKDSK Utility Flaw Explained

I recently was asked about the file system issue with Windows and thought I would be a little more in depth. The file system problem is resolved with the following patches:

MS831375 and MS873437 for Windows 2000

Here is a good description of the problem (3rd paragraph down) and Microsoft’s page.

http://www.windowsitpro.com/Article/ArticleID/41569/41569.html

http://support.microsoft.com/?kbid=831375

Basically when a large volume has over 4,194,303 files a flaw in the chkdsk utility run in fix mode or repair mode can strip the permissions from the files. Patch MS873437 is related. Here is the link.

http://support.microsoft.com/default.aspx?scid=kb;en-us;873437

MS831374 is for Windows 2003 and is the same issue with the chkdsk utility.

http://support.microsoft.com/default.aspx?scid=kb;en-us;831374

According to our administrators we had systems on other patch reboots go into the chkdsk utility and automatically start running in fix or repair mode. So in our case it was not something that was user initiated. Hope this helps and have fun patching!

Restoring NT Shares

Well, it inevitably happens that a large fileserver is being replaced, goes down, or goes bad and you have to restore it. Well if you are restoring to the same type of hardware then life is beautiful and NT/2000/2003 has no problems and acts like the good little boy it should be, but what happens when you have to restore to new hardware, or are refreshing the server to a newer more powerful server? This is when NT can be worse than that bratty little kid I wanted to strangle in the movie Problem Child. Personally John Ritter should have shot him and buried him in the back yard, but I digress. So how can you restore Shares or any other piece of the registry with TSM when NT doesn't like having the registry of another machine of different hardware restored? Well that's were a little ingenuity and some preemptive strategy comes in handy. The easiest way is to setup a script using the command line registry tool REG.EXE that runs a REG EXPORT KEYNAME each day and stores it as a file. Here is the string needed to backup the Shares key. (The following is all one line)

REG EXPORT HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares C:\BACKUP\SHARESEXP.TXT

The command backs up the Shares key to a folder I created on the C: drive called Backup, but you can put it wherever you like. A simple batch file will suffice to execute the export and you can either schedule it through Windows daily or through TSM. Yes, you can do that and TSM will gladly back it up. This allows you to then restore the export file and reimport it into the server of choice. Just make sure that with any schedule where you are executing a command/batch file you put the full path of the file or else TSM can possibly fail to execute it, even if its in the TSM BACLIENT directory.

Ok, not everyone has that process in place and needs to restore the Share key now! Well have no fear TSM can help you accomplish it. Let's look at the process for this situation and I'll provide some links. First off you can restore the registry without activating it. The command to do so is

RESTORE REGISTRY -ACTIVATE=NO (for Win2K/Win2003)

RESTORE REGISTRY ENTIRE -ACTIVATE=NO (for WinNT)

Now for each process you can restore specific keys but for ease of use the entire restore will work. TSM will restore the registery to the c:\adsm.sys\computername\ directory. From this point you can load the hive in RegEdit containing the share key and export it to a file. Once exported you should be able to import it into the current registry thereby restoring the shares. Here is a Microsoft document that covers loading the hive and processes to use. There is also a good document on backing up and restoring the registry here. If you have any questions feal free to comment and I'll respond as soon as possible.

Crititcal Windows Filesystem Issues

Well this is a copy from my other less TSM centric blog Storage Admin Blues but it needs to be known. Here is the post from February -

So I have been working a major data issue on a server with a certain Redmond operating system for almost a week now. As it turns out the afore mentioned OS has a known issue with volumes that contain over 4 million files. What happens is that the system will reboot after a patch or update is applied and on the startup begin a check-disk. That wouldn't be so bad but the check-disk strips permissions from almost all files. The response from Redmond was that we would have to restore the data if we want to fix the permissions. OK! Great! Restore 6 million files when the server is used 24/7. The volume in question is over 570GB space used and it has a gig-Ethernet connection. I swear if there isn't a conspiracy against storage administer when it comes to restore SLA's. I got called in to fix the problem and have had almost no sleep for a week, and in conjunction with that I have a virus that is causing me to cough incessantly and make it hard to breathe. Thank goodness for telecommuting or I'd be in the hospital by now. Lets just hope the people in charge listen this time (it has happened two times before this) when we warn about volume size/file management. If it wasn't for Arrested Development and Scrubs I would go nuts.

The resolution for this problem was to restore the directory structures and then have the system admins apply a script that cascaded the permissions to the files within the directory structure since they all inherited their permissions from the parent folder. We also decided to change the environment to backup all directory structures to disk and retain them there as long as possible before migrating to tape. So we had to use the migration delay and migration continue feature on the disk pool. Trust me restoring directories off of tape is no picnic...very slow. If you have run into something like this let me know how you resolved it. Sharing info is how we learn more and although the my e-mail might say TSM Expert, I'm don't know everything.

TSM and Windows Volume Shadow Copy

First off let me just say that the name TSMExpert was more of a joke. I was setting up aliases on Notes Mail and for fun put TSMExpert and viola I got it. Didn't think it would go through but hey as long as I have it I might as well use it. I can be reached at tsmexpert@us.ibm.com and will answer questions as much as possible. Remember TSM is a fickle beast but when setup correctly it is the best enterprise backup/archive tool on the market.

I have been working with many issues on Windows 2003 and the TSM client. We upgraded to 5.2.4 on many of our clients and some to 5.3 due to the shadow copy issues. TSM seems to have serious issues with VSC and gives RC12 on schedules when its implemented. I am not quite sure how VSC is suppose to help when it resides on the same disk as the data its supposed to correct, but I get its more like the snapshot feature on NetApps. I would recommend you upgrade all 2003 servers if you are having this problem to 5.3 since it is supposed to resolve this issue. Also be aware there are problems with older client versions stability and reliability on 2003 servers. We have experienced numerous crashes and System State backup failures with the older clients. This is another reason for upgrading.