When Data Isn't There, But Is Really There!

OK so the title seems odd, but I had this occur recently when our UNIX Admins decided to convert a subdirectory to a file system. So they created the file system and wiped out the subdirectory expecting us to restore the data. When you would query for the backups nothing would show as having been backed up. (Let me clarify that the Admins did the change and then a week later alerted us and asked for the restore.) So the project manager asked for the restore but did not clarify the directory had gone from /opt/ar/ to being file system /opt/ar/. So any

dsmc res /opt/ar/* /opt/ar/ -pick -inactive -subdir=yes 

returned no data. What I had to do was

dsmc res /opt/* /opt/ar/ -pick -inactive -subdir=yes -pitd=

and then the data would show and I could select it manually for restore. If there is one thing I have learned with TSM is always double, triple, and quadruple check for "missing" data using various methods, chances are it is probably there. Remember the user is clueless....don't get me started on symbolic links and how users don't always know the actual location of their data.

Active Data Pool - What's The Point?

To go along with the ADP story I have moved this older post up for easy reference.

With the release of TSM 5.4 Tivoli has added the ability to create an active data storage pool to allow for faster restore times. We have been looking into using them at work when I stumbled upon this interesting factoid in the description of the active data pools limitations:

1. The server will not attempt to retrieve client files from an active-data pool during a point-in-time restore. Point-in-time restores require both active and inactive file versions. Active-data pools contain only active file versions. For optimal efficiency during point-in-time restores and to avoid switching between active-data pools and primary or copy storage pools, the server retrieves both active and inactive versions from the same storage pool and volumes.

So my question is why don't they allow the client to restore all the active data first then restore the inactive, or why didn't they implement a multi-session restore process when they added active data pools to the product, thereby speeding up the restore process? With the amount of P-I-T restores I do, this issue makes the whole active data pool useless not just for me but I'm sure for many of you.

TSM DB Dump Load Audit

There's nothing quite like the smell of TSM database structure issues in the morning.Only run through this proceedure if you are sure you have DB structure issues.

Here are the instructions on running a dump/load/audit.
.
1. Make a copy of the following files:
- dsmserv.opt
- dsmserv.dsk
- volhist (volume history file)
- devconfig (device configuration file)
.
2. Set the following options in your dsmserv.opt file:
EXPINTERVAL 0
DISABLESCHEDS YES
NOMIGRRECL
.
3. Define a file devclass un... click here to read the full article

Bare Metal Restore

All the best for 2008
Bare Meal Restore - Put simply

For W2K the TSM dsm.opt DOMAIN ALL-LOCAL is the default, this will include
SYSTEMOBJECT backup for incrementals.

Backup Commands:
dsmc incr

TSM Windows 2000 restore steps:
1. Repartition, format harddrive
2. Install W2K, set computername to original
3. Install appropriate W2K Service Pack
4. Install tsm ba client
5. dsmc restore c:\ -replace=all -subdir=yes
6. dsmc restore systemobject
7. Reboot

Cheers
Ryan Partington

DR Tests & System Restores

I know I have covered this before, but we have been doing DR tests and when restoring UNIX servers I find it easier and faster to use the Web/Java GUI to select all file systems required to restore the data than following the IBM guideline to build a small script that runs them serially. The thing I don't get is why they would push command line restore. Doing the restore file system by file system will require the same tapes be mounted multiple times where the GUI will restore all the data available on the tape for the selected backup. They do state that you can execute more than one restore simultaneously, but in every DR exercise I have run that would just cause tape contention since I do not collocate by file space. Unless I am wrong in how TSM handles a GUI based multiple file system restore wouldn't the GUI be the better solution when in a DR or full UNIX system restore scenario?

Of course this is all dependent on whether you have Java 1.4.1 or higher installed.

Recovery

It's not about backup, it's about recovery. Let's take a look at a few articles that can help you protect your TSM environment. You're backup solution is no good if you can't recovery from a disaster. These articles are based on Windows.

http://ryanpartington.com/article/tsmdaily/
Working through your daily tasks should look something like this, ensuring you have an offsite copy of your primary pools.

Once you're happy with your daily tasks, schedule a test server recovery exercise every quarter. This will allow you to optimise your processes and resolve any potential stumbling blocks. The following two articles look at recovering the server.
http://ryanpartington.com/article/tsmdisaster/ - Step by step text
http://ryanpartington.com/article/tsmrecovery/ - Step by step videos (posted earlier this week)

Any comments welcome
Thanks

Restoring Windows Shares

This is a repost from June 2005

Well, it inevitably happens that a large fileserver is being replaced, goes down, or goes bad and you have to restore it. Well if you are restoring to the same type of hardware then life is beautiful and NT/2000/2003 has no problems and acts like the good little boy it should be, but what happens when you have to restore to new hardware, or are refreshing the server to a newer more powerful server? This is when NT can be worse than that bratty little kid I wanted to strangle in the movie Problem Child. Personally John Ritter should have shot him and buried him in the back yard, but I digress. So how can you restore Shares or any other piece of the registry with TSM when NT doesn't like having the registry of another machine of different hardware restored? Well that's were a little ingenuity and some preemptive strategy comes in handy. The easiest way is to setup a script using the command line registry tool REG.EXE that runs a REG EXPORT KEYNAME each day and stores it as a file. Here is the string needed to backup the Shares key. (The following is all one line)

REG EXPORT HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares C:\BACKUP\SHARESEXP.TXT

The command backs up the Shares key to a folder I created on the C: drive called Backup, but you can put it wherever you like. A simple batch file will suffice to execute the export and you can either schedule it through Windows daily or through TSM. Yes, you can do that and TSM will gladly back it up. This allows you to then restore the export file and reimport it into the server of choice. Just make sure that with any schedule where you are executing a command/batch file you put the full path of the file or else TSM can possibly fail to execute it, even if its in the TSM BACLIENT directory.

Ok, not everyone has that process in place and needs to restore the Share key now! Well have no fear TSM can help you accomplish it. Let's look at the process for this situation and I'll provide some links. First off you can restore the registry without activating it. The command to do so is

RESTORE REGISTRY -ACTIVATE=NO (for Win2K/Win2003)

RESTORE REGISTRY ENTIRE -ACTIVATE=NO (for WinNT)

Now for each process you can restore specific keys but for ease of use the entire restore will work. TSM will restore the registery to the c:\adsm.sys\computername\ directory. From this point you can load the hive in RegEdit containing the share key and export it to a file. Once exported you should be able to import it into the current registry thereby restoring the shares. Here is a Microsoft document that covers loading the hive and processes to use. There is also a good document on backing up and restoring the registry here. If you have any questions feal free to comment and I'll respond as soon as possible.

TSM Client Restore/ActLog Issue

So one of my colleagues kicked off a restore and upon doing an ActLog query for an unrelated issue it was noticed that every file restored was reporting in the TSM ActLog. Does anyone know why when I run a restore the ActLog is listing every file restored for a Windows client? I have never seen this happen before and not sure what setting is allowing it. Is this new in one of the 5.3.x releases or am I missing something? This can have a huge impact on ActLog size so I am a little concerned. I am used to seeing this in the DSMSCHED.LOG not the TSM ActLog.

ANR9999D Error Restoring From Backupsets

We have a site that uses backupsets for extended retention (which I hate, but that's beside the point). I had a number of restores (and "QUERY BACKUPSETCONTENTS" commands) that were failing with the error "Command failed – internal server error detected." Further research into the problem revealed only that 9999D means "thread terminated", with no other specific information.

After a call to Tivoli support, and some exchanging of some logs, it was determined that tape containing the backupset was not in the library. When the "request" timer would expire, the server would terminate the thread, but no "message" would return. This includes running commands from the server and client command line interfaces. You can replicate the error by running the command in one window, then canceling the request on the server in another. The Tivoli rep said that he would send it to the developers, so this may be "fixed" to include a proper error message in a future version.

All Hail The Java GUI!

Interesting resolution to a restore I just had thought I would pass it on... On one of our Win2K3 servers all admin accounts were crippled due to some corruption in the registry (my vote was on Gremlins!). So they wanted to restore the system state to a week ago.  I tried and the client core dumped due to our permissions being crippled.  So I tried one other thing and that is the Java GUI.  I figured since it runs under the System Account it might work.  Well it did and fixed their problems....I had not used the Java GUI in ages but here is one situation where it came in quite handy.

UNIX Permission Issues

Awhile back I ran into an issue with how TSM handles permissions on UNIX files and wanted to get some feedback from you readers out there on how you would handle it.  What happened was a user somehow was given root and he chown’ed the /home dir recursively.  It was made worse by the fact that he did that on Friday and didn’t alert anyone until the following Monday, and by the time it got to us another day had passed.  The customer of course wanted us to restore the directory and file permissions, but the kicker was that TSM does not back a UNIX file up again when the permissions change.  It just updates the database to reflect the permission changes (I got that directly from support and was floored; I had no idea it handled UNIX that way).  So here was our dilemma, if the file was the only version in backup I would not have any way of resetting its permissions.  Is the gravity of the situation hitting home?  Because it doesn’t backup the file again or track permissions I could not successfully restore to a point-in-time.  Sure I might get a good portion of the files fixed but there would still have been a large portion that we would be unable to get the permissions corrected.  The customer wasn’t happy and our only out was that the customer should not have been doing chown‘s as root.  I thought I once saw someone post a undocumented option you can set in the options file that will backup a file if it changes in any way, permissions included, but I can’t find it.  I thought I saw it on the new ADSM.org but am unable to locate it.  Anyone know the option or have an idea on how to approach this?  I brought it up with some Tivoli people who asked me what I thought should be added or changed in TSM, but so far I haven’t seen any change in their processing.          

Restoring An Image Backup To A Larger Disk

Well just the other day my group was asked to help the NT System Admins with the combination of two large disks into one. Both of the disks have been performing image backups so we decided that the largest image would be restored first then the second drive would be restored to the new disk normally. To combine the disks the SA’s made one huge partition (400+ GB) and an SA on my team started the image restore. The following warning was issued when the image was initiated:

***************************** WARNING ********************************
The destination volume is larger than the source volume.  This will reduce the file system '\\machinename\x$' size to '    xxx.xx MB'.
Do you wish to continue? (Yes (Y)/No (N))

This was due to the target disk being larger than the image. We figured the image would not affect the space unused but we were wrong. After the image was restored explorer saw the disk as the size of the image backup but the Windows Disk Management tool saw it as the 400+ GB it was. So the problem was how to recover the disk.  Here is what MS support instructed and the problem was resolved.

When doing an image restore:

If the destination volume is larger than the source, after the restore operation you will lose the difference between the sizes. If the destination volume is located on a dynamic disk the lost space can be recovered by increasing the size of the volume. This will also increase the size of the restored volume.

We also have the following knowledge base doc:

When a Tivoli Storage Manager (TSM) image restore is performed to a target volume that is larger in size than the original volume (from which the image backup was taken), TSM will concatenate the target volume down to the original volume size.  However, within Windows Disk Management (within the Administrative Tools -> Computer Management utility), the disk will appear to still exist as the original larger volume size, even though Windows Explorer will show that the real volume size is now smaller.  In essence, this means that space has been lost, since it is not accessible on the drive and not available for use.  To prevent the loss of this space, the same volume size should be used on the target volume as on the original volume.

As far as I know this fix only works if the disk is dynamic (logical) and not a fixed (physical) disk. So the fix was to increase the disk size again to help it see the space again. I don’t think you need to increase it much but once done it resolves the issue. So if you happen to restore an image backup to a larger partition than it originated from just be aware of what you’ll need to do to reclaim your unused space after the restore finishes.

Question On Image Backups

The image backup feature is a wonderful piece that really saves you when restore time is of the essence, but what do you do when SA’s decide to combine two drives into one and want to use TSM to do it? Well we decided to use the image from the larger drive then do a standard restore of the other drive afterwards. Since I have never done this and don’t restore images frequently something I was alerted to was the message from TSM that states that file system would be changed (the drive was now bigger for the combining of the data). Will the image resize the partition? I don’t recall TSM doing that but then again I can’t remember. Anyone who has done a lot of image backups (particularly to larger drives than what the image came from) input would be appreciated.

NetApp Filer TOC Issue

I thought I would pass along a notice that we have been informed of a problem with TOC file corruption on NetApp filers that have files with dates before Jan. 1, 1970, and after Jan. 19, 2038. Somehow it causes the TOC backup to abend. It also looks like the TOC can be corrupted when the bitfile in which the TOC file is stored is damaged. If this is the case and the corruption is recent it is possible that the TOC is undamaged on the copypool and could possibly be used for the restore. If the TOCs are unavailable then file level restore will need the absolute path to succeed.    

CHKDSK Utility Flaw Explained

I recently was asked about the file system issue with Windows and thought I would be a little more in depth. The file system problem is resolved with the following patches:

MS831375 and MS873437 for Windows 2000

Here is a good description of the problem (3rd paragraph down) and Microsoft’s page.

http://www.windowsitpro.com/Article/ArticleID/41569/41569.html

http://support.microsoft.com/?kbid=831375

Basically when a large volume has over 4,194,303 files a flaw in the chkdsk utility run in fix mode or repair mode can strip the permissions from the files. Patch MS873437 is related. Here is the link.

http://support.microsoft.com/default.aspx?scid=kb;en-us;873437

MS831374 is for Windows 2003 and is the same issue with the chkdsk utility.

http://support.microsoft.com/default.aspx?scid=kb;en-us;831374

According to our administrators we had systems on other patch reboots go into the chkdsk utility and automatically start running in fix or repair mode. So in our case it was not something that was user initiated. Hope this helps and have fun patching!

ETA Please!

Well I just had to perform a restore for a small server and the speed at which the restore ran was atrocious. I mean it was slower than rush hour in LA. Let's discuss, and THIS TIME I WANT AND REQUEST FEEDBACK! It turned out a disk went bad on a webserver and the system admins requested a number of filesystem restores. The combined amount was about 22-25GB. OK! No problem! Well that probably would have been the case if the restore request had been during the day but the request came in at night and the restore was competing with the nightly backups. Over a gig-ether (Fiber Gig-ether) connection I was able to get 1.3MBs and an aggregate rate of 668Kps. So do the math and it took a long period of time. The other thing that didn't help was it was a web server with TONS of little objects. It's livable but there were a lot of small files. The problem was eveyone and their brother wanted an ETA. "How long? It's Small! It should only take a couple hours max!" and so on. Well people now want some solution to this situation but of course the problem will be keeping it somewhat cheap. Even though everyone asks if we can halt the backups while we perform the restores we all know that's not really a viable option, so I came up with this idea, tell me what you think. Since major restores are few and far between I am proposing we create a new VLAN and run a single cable to each row of servers in the server room with enough slack to stretch to any server in the row. If a restore is required we simply plug in the "restore" connection, set an IP and away it rips. When finished we put the system back on the backup network it is assigned and rollup the excess ethernet cord and place it in the rack of the server in the middle of the row. I am only thinking of this for major restores and since I am not requesting that we buy more NIC's I think it's doable. Let me know what restore process you have in place when the network is saturated. I'd love suggestions!

Restoring NT Shares

Well, it inevitably happens that a large fileserver is being replaced, goes down, or goes bad and you have to restore it. Well if you are restoring to the same type of hardware then life is beautiful and NT/2000/2003 has no problems and acts like the good little boy it should be, but what happens when you have to restore to new hardware, or are refreshing the server to a newer more powerful server? This is when NT can be worse than that bratty little kid I wanted to strangle in the movie Problem Child. Personally John Ritter should have shot him and buried him in the back yard, but I digress. So how can you restore Shares or any other piece of the registry with TSM when NT doesn't like having the registry of another machine of different hardware restored? Well that's were a little ingenuity and some preemptive strategy comes in handy. The easiest way is to setup a script using the command line registry tool REG.EXE that runs a REG EXPORT KEYNAME each day and stores it as a file. Here is the string needed to backup the Shares key. (The following is all one line)

REG EXPORT HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Shares C:\BACKUP\SHARESEXP.TXT

The command backs up the Shares key to a folder I created on the C: drive called Backup, but you can put it wherever you like. A simple batch file will suffice to execute the export and you can either schedule it through Windows daily or through TSM. Yes, you can do that and TSM will gladly back it up. This allows you to then restore the export file and reimport it into the server of choice. Just make sure that with any schedule where you are executing a command/batch file you put the full path of the file or else TSM can possibly fail to execute it, even if its in the TSM BACLIENT directory.

Ok, not everyone has that process in place and needs to restore the Share key now! Well have no fear TSM can help you accomplish it. Let's look at the process for this situation and I'll provide some links. First off you can restore the registry without activating it. The command to do so is

RESTORE REGISTRY -ACTIVATE=NO (for Win2K/Win2003)

RESTORE REGISTRY ENTIRE -ACTIVATE=NO (for WinNT)

Now for each process you can restore specific keys but for ease of use the entire restore will work. TSM will restore the registery to the c:\adsm.sys\computername\ directory. From this point you can load the hive in RegEdit containing the share key and export it to a file. Once exported you should be able to import it into the current registry thereby restoring the shares. Here is a Microsoft document that covers loading the hive and processes to use. There is also a good document on backing up and restoring the registry here. If you have any questions feal free to comment and I'll respond as soon as possible.

Crititcal Windows Filesystem Issues

Well this is a copy from my other less TSM centric blog Storage Admin Blues but it needs to be known. Here is the post from February -

So I have been working a major data issue on a server with a certain Redmond operating system for almost a week now. As it turns out the afore mentioned OS has a known issue with volumes that contain over 4 million files. What happens is that the system will reboot after a patch or update is applied and on the startup begin a check-disk. That wouldn't be so bad but the check-disk strips permissions from almost all files. The response from Redmond was that we would have to restore the data if we want to fix the permissions. OK! Great! Restore 6 million files when the server is used 24/7. The volume in question is over 570GB space used and it has a gig-Ethernet connection. I swear if there isn't a conspiracy against storage administer when it comes to restore SLA's. I got called in to fix the problem and have had almost no sleep for a week, and in conjunction with that I have a virus that is causing me to cough incessantly and make it hard to breathe. Thank goodness for telecommuting or I'd be in the hospital by now. Lets just hope the people in charge listen this time (it has happened two times before this) when we warn about volume size/file management. If it wasn't for Arrested Development and Scrubs I would go nuts.

The resolution for this problem was to restore the directory structures and then have the system admins apply a script that cascaded the permissions to the files within the directory structure since they all inherited their permissions from the parent folder. We also decided to change the environment to backup all directory structures to disk and retain them there as long as possible before migrating to tape. So we had to use the migration delay and migration continue feature on the disk pool. Trust me restoring directories off of tape is no picnic...very slow. If you have run into something like this let me know how you resolved it. Sharing info is how we learn more and although the my e-mail might say TSM Expert, I'm don't know everything.