Data Loss Top 10

Ontrack, which handles data recovery, compiled a list of the strangest reasons data was lost. You can read their list here. I remember when I was teaching TSM some students had some of the craziest stories for data loss (many were disgruntled employee stories).

One story I heard from a student occurred when a manager was stupid enough to fire an employee, but require him to work the rest of the week if he wanted his pay. So the employee worked the rest of the week and on his way out (unescorted) he went into the server room and proceeded to urinate into the mainframe. Supposedly he shorted out the mainframe and caused all kind of havoc. If you have a story you'd like to share feel free to leave it in the comments section.

ACSLS Checkout Script Update

I Modified the ACSLS checkout script due to an Admin with BYU pointing out it doesn't work when the library clients reside on seperate servers. It was changed due to how TSM processes the CMDFILE= option in my MOVE DRM macro, plus if your TSM server names vary the while loop is not as easy to setup. So here is the new script and macro. I switched it to use an array for the servernames for easier processing when names don't follow a similar convention. Also you might need to modify the grep if your volumes don't start with L0 thru 4. If your volser is a different range just modify the grep so it will work for you.

<---------------------MOVE DRM MACRO------------------------>

move drm * so=dbb wherestate=mo tostate=vault remove=no WAIT=YES
move drm * so=dbb wherestate=vaultr tostate=onsiter

<------------------------Cut Below-------------------------->

#!/bin/ksh

cd /usr/tivoli/tsm/client/ba/bin

ADSMID=`cat /usr/local/scripts/ADSMID`
ADSMPASS=`cat /usr/local/scripts/ADSMPASS`

OFFSITE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/offsite.txt
DRVOLS=/usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List
RETRIEVE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/retrieve.txt

set -A SERV TSM-1 TSM-2 TSM-3 TSM-4

cp $OFFSITE "$OFFSITE.BAK"

mv /usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List /usr/tivoli/tsm/client/ba/bin/OFFSITE/Vol_List.bak

printf "Use this list to determine tapes that are to go offsite report any discrepancies to the Recovery Services Team.\n\n" > $OFFSITE
printf " \n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE
printf " The following tapes will checkout from the library\n\n" >> $OFFSITE
printf " and should be sent offsite\n\n" >> $OFFSITE
printf " Current as of: `date`\n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE

printf "Use this list to determine tapes that are to come back onsite from Iron Mountain for reuse. Report any discrepancies to the Recovery Service Team.\n\n" > $RETRIEVE
printf " \n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE
printf " Tapes to be brought back onsite from Iron Mountain\n" >> $RETRIEVE
printf " and placed back into TSM library for scratch.\n\n" >> $RETRIEVE
printf " Current as of: `date`\n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE

i=0
while [ $i -lt 4 ]
do
dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} -dataonly=yes "select volume_name from drmedia where state='MOUNTABLE' and volume_name not in (select volume_name from drives where volume_name is not NULL) " grep L[0-4] >> $DRVOLS

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} -dataonly=yes "select volume_name from drmedia where state='VAULTRETRIEVE' " grep L[0-4] >> $RETRIEVE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=${SERV[$i]} 'macro /usr/tivoli/tsm/client/ba/bin/move_drmedia.mac'

sleep 120

i=$(( $i + 1 ))
done

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername= CHECKIN LIBVOL search=yes stat=private checklabel=barcode vollist=FILE:$DRVOLS

sleep 180

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername= CHECKOUT LIBVOL checkl=no remove=bulk vollist=FILE:$DRVOLS

cat $DRVOLS sort >> $OFFSITE

mail -s "Daily Tape Rotation - Tapes to be sent to Iron Mountain" tape_rpt < $OFFSITE mail -s "Daily Tape Return" tape_rpt < $RETRIEVE

TSM DB Dump Load Audit

There's nothing quite like the smell of TSM database structure issues in the morning.Only run through this proceedure if you are sure you have DB structure issues.

Here are the instructions on running a dump/load/audit.
.
1. Make a copy of the following files:
- dsmserv.opt
- dsmserv.dsk
- volhist (volume history file)
- devconfig (device configuration file)
.
2. Set the following options in your dsmserv.opt file:
EXPINTERVAL 0
DISABLESCHEDS YES
NOMIGRRECL
.
3. Define a file devclass un... click here to read the full article

REPOST: ACSLS Shared Library Checkout Script

For those with a shared ACSLS TSM library looking for a checkout script, I thought I would repost this so it would be easier to find.

Here is a basic rundown of what the script does.

• On each library client DRM checks the tapes out with a REMOVE=NO option and creates a file with a list of the tapes.
• On the library manager the tapes are checked back into the library.
• The Library manager then checks every library clients DRM tapes out.

I know it sounds complex but it's not. It also is not perfect so you will need to keep on top of it, but don't worry it is not so bad that it's a headache. Some explaination is required also; In my environment I have 7 TSM instances and the firsts is just a library controller. This is why the following script starts at TSMSERV2 and stops the loop after TSMSERV7. Also note that the pipe sysmbol will not show up for some reason so you need to add it before the grep statements. Before you use the script you will also notice that it uses a TSM macro. Create the file by putting the following command into a macro called move_drmedia.mac.

move drm * so=dbb wherestate=mo tostate=vault remove=no CMD=&VOL CMDFILE=/usr/tivoli/tsm/client/ba/bin/Vol_List APPEND=YES WAIT=YES
move drm * so=dbb wherestate=vaultr tostate=onsiter


=-=-=-=-= Below Starts Checkout Script =-=-=-=-=

#!/bin/ksh

cd /usr/tivoli/tsm/client/ba/bin

OFFSITE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/offsite.txt
RETRIEVE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/retrieve.txt
ADSMID=`cat /usr/local/scripts/ADSMID`
ADSMPASS=`cat /usr/local/scripts/ADSMPASS`
I=2

cp $OFFSITE "$OFFSITE.BAK"
cp $RETRIEVE "$RETRIEVE.BAK"
cp /usr/tivoli/tsm/client/ba/bin/Vol_List /usr/tivoli/tsm/client/ba/bin/Vol_List.bak

cat /dev/null > /usr/tivoli/tsm/client/ba/bin/Vol_List

printf "Use this list to determine tapes that are to go offsite report any discrepancies to the Recovery Services Team.\n\n" > $OFFSITE
printf " \n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE
printf " Tapes to be sent offsite\n\n" >> $OFFSITE
printf " Current as of: `date`\n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE

printf "Use this list to determine tapes that are to come back onsite from Iron Mountain for reuse. Report any discrepancies to the Recovery Service Team.\n\n" > $RETRIEVE
printf " \n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE
printf " Tapes to be brought back onsite from Iron Mountain\n" >> $RETRIEVE
printf " and placed back into TSM library for scratch.\n\n" >> $RETRIEVE
printf " Current as of: `date`\n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE

while [ $I -lt 8 ]
do
S=TSMSERV-$I

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S -dataonly=yes "select volume_name from drmedia where state='MOUNTABLE' " grep L[0-3] >> $OFFSITE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S -dataonly=yes "select volume_name from drmedia where state='VAULTRETRIEVE' " grep L[0-3] >> $RETRIEVE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S 'macro move_drmedia.mac'

sleep 120

I=$(( $I + 1 ))
done

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=TSMSERV-1 'CHECKIN LIBVOL TSMLIB search=yes stat=private checklabel=no vollist=FILE:/usr/tivoli/tsm/client/ba/bin/Vol_List'

sleep 180

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=TSMSERV-1 'CHECKOUT LIBVOL TSMLIB vollist=FILE:/usr/tivoli/tsm/client/ba/bin/Vol_List'

mail -s "TDC Daily Tape Checkout" $HN tape_rpt < $OFFSITE
mail -s "TDC Daily Tape Return" $HN tape_rpt < $RETRIEVE

Data DeDuplication - Been There Done That!

I just got off a pretty good NetApp webcast covering their VTL and FAS solutions. One of the items they discussed was the data deduplication feature with their NAS product. When the IBM rep spoke up they discussed TSM's progressive backup terminology and I find it interesting to contrast TSM's process with the growing segment of disk based storage that is the deduplication feature. The feature really helps save TONS of space with the competing backup tools since they usually follow the FULL+INC model causing them to backup files even when they haven't changed. Here deduplication saves them room by removing the duplicate unchanged files, but this shows how superior TSM is, in that it doesn't require this kind of wasted processing. What would be interesting is to see how much space is saved in redundant OS files, but that is still minor compared to the weekly full process that wastes so much space.

This brings us to the next item, disk based backup. This is definitely going to grow over time, but costs are going to have to come down for it to fully replace tape. The two issues I see with disk only based backups is in DRM/portability and capacity/cost. If you cannot afford to have duplicate sites with the data mirrored then you are left having to use a tape solution for offsite storage. Also with portability disk can be an issue. For example we are migrating some servers from one data center to another and we used the export/import feature. We have also moved TSM tapes from one site to another and rebuilt the TSM environment. To do this with disk is a little more time consuming, you would need the same disk solution and the network capacity to mirror the data (time consuming on slow connection) or have to move the whole hardware solution. Tape in this scenario is a lot easier to deal with. Now when it comes to capacity vs. cost there is a definite difference that will keep many on tape for years to come. Many customers want long term retention of their data, say 30+ days for inactive files and TDP backups (sometimes longer with e-mail and SARBOX data). So what is the cost comparison for that type of disk retention (into the PB) compared to tape. Currently it's no contest and tape wins in the cost vs. capacity realm, but hopefully that can someday change. So if any of you have disk based solutions or VTL solutions chime in I'd like to hear what you have to say and how it's worked for you.

ACSLS Not Really Ready For Library Sharing!

So we have a large ACSLS library at one of our data centers and we decided to use library sharing with 5.3. Tivoli stated that in 5.3 ACSLS is now supported in a library sharing environment. So we setup library sharing and proceeded to also setup DRM to handle our offsite tape rotation. Now I have used IBM equipment almost exclusively when I was with IBM so this ACSLS stuff is new to me, so when MOVE DRMEDIA did not seem to be working I immediately thought it was an ACSLS issue. The problem was that when we ran a MOVE DRMEDIA the library would only checkout one tape at a time. We have two 40 tape I/O doors so why it would only check out one tape at a time was puzzling(it would wait until the tape was removed from the I/O door before checking out the next tape). So I called Sun/STK and was told it was a software issue. So when I called Tivoli it took days before someone found the issue was in fact TSM. The APAR is IC45537 which states that TSM library clients do not currently support the concurrent checkout of multiple volumes. There was a local work around which entailed using MOVE DRMEDIA REMOVE=NO from the library client, checking them back in on the library controller, and then checking them out on the library controller. It's a mess of a script but it works. This APAR was released in April of 2005 yet I am still having to use a local work around. It just drives me nuts! Especially because the problem was not one that even Tivoli support easily found. Let's hope there is an actual fix in 5.4 but I'm not betting on it.

MOVE DRMEDIA from shared ACSLS library

OK, so I my work has an ACSLS library shared by 7 instances of TSM and my move drmedia commands fail. I have tried it with and without the cap= option and also have used remove=bulk and remove=yes. The volumes fail to checkout and the ACSLS documentation I have found so far does not cover shared ACSLS checkout. Any help is appreciated.

NIM vs. SysBack for DRP

We recently performed a DR test and one business group decided to use a NIM server to rebuild their systems for the test. This meant taking the local NIM server and having to restore it then restore the individual clients, meanwhile us TSM admins sat around twiddling our thumbs waiting for them to finish (for some reason the system images and NIM are handle by the System Administrators group and not the Backup/Recovery group). I would have preferred SysBacks (since we are an AIX shop), but the SA had this brilliant idea to use NIM. This added another layer of complexity and actually due to disk issues it took forever to get the NIM server running. I would like feedback on what others have done for their DR tests, and if anyone has experience with the TSM integrated  SysBack feature I would love to hear how it works for you.

The Case For Raw Volumes!

If you are serious about TSM server rebuild times and want the quickest way to get up and running then I suggest you look into raw logical volumes for all your TSM DB, Log, and storage needs. Of course if you are running on NT I can't say I know of any way TSM can use raw, but in our AIX shop we live by raw volumes. The creation time is quick and with a little script I can have my volumes created and ready for the DB restore in no time. I have been down the road of DSMFMT and know how long large volumes can take to create and since TSM does not like more than 16 volumes some older Unix servers can take time to format. The other nice thing about raw volumes is if the server crashes its rare, except for disk failure, for volume corruption to occur. I have had too many dirty super blocks to deal with in my time, and I don't miss them. Remember, all TSM is really doing with DSMFMT is creating a file and in a way converting it back into raw. So why do the extra steps, save yourself some time if you ever are in a true DR situation.