Update: Weird Volhist Records

Upon further investigation I found that my other shared library environment, that has multiple NAS devices defined to the library manager, is also showing the NAS device class for all LTO-3 volumes used by the library clients. If anyone has a NAS attached to a shared library environment please compare your volhist devclass to those of your library clients and tell me if you see the same situation. CRAZY THING IS, it doesn't seem to affect the library clients from using their tapes or acquiring scratch. What my concern is, is that I am slowing losing use of tapes that should be marked back as scratch, but due to the volhistory settings is unable to be released. I am wondering if this is a bug in TSM that could be affecting others.

Weird Volhist Records

I am reviewing my Volume History file on my ACSLS library controller and it shows volumes as REMOTE and owned by the library clients but the weird thing is it shows the DEVCLASS as a device class that is not even defined on the library clients. Why would TSM do this? So I have a device class called IBM-LTO-3 on all my TSM instances (8 total including the library manager) and on the library manager there is a device class called NAS-DEV-CL. I show tons of tape volumes using the NAS-DEV-CL devclass on the library manager's volume history but not on the other instances (since they don't even have the NAS-DEV-CL devclass defined). Why or how would this occur? I tried an audit library on one of my library clients but it is not completing (I've had issues with audit library commands from library clients before). Any ideas?

Here is an example of what I see:

tsm: PD-703-S-AITSM-1>select * from volhistory where volume_name='L40202'

DATE_TIME: 2009-08-19 03:33:07.000000
UNIQUE: 0
TYPE: REMOTE
BACKUP_SERIES:
BACKUP_OPERATION:
VOLUME_SEQ:
DEVCLASS: NAS-DEV-CL
VOLUME_NAME: L40202
LOCATION: PD703-UAX007
COMMAND:

When I query the volume on the library client I see:

Volume Name: L40202
Storage Pool Name: NP-STD-TAPE
Device Class Name: IBM-LTO-3
Estimated Capacity: 1.6 T
Scaled Capacity Applied:
Pct Util: 0.4
Volume Status: Filling
Access: Read-Only
Pct. Reclaimable Space: 26.1
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 14
Write Pass Number: 1
Approx. Date Last Written: 08/26/09 11:01:12
Approx. Date Last Read: 09/22/09 11:19:00
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
Last Update Date/Time: 09/14/09 13:14:21
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: None

So the question is how did this volume history record on the library manager end up with the devclass that doesn't exist on the library client? It doesn't seem to affect the volumes or processing daily tasks, but I am worried its not freeing up the tapes to return to a scratch status when they are reclaimed.

Migrating To A Library Controller Instance

For those of you who use a shared library environment, but do not have a dedicated TSM library controller instance, now is the time to make the change. I know many of you figured it was to hard or risky to try but that is no longer the case. IBM has just release a Redpaper that will help you move your library controller to a dedicated instance. The new Redpaper is called:

IBM Tivoli Storage Manager: How to Migrate the Library Manager Function

I made the switch years ago and it was the best thing I ever did. Have to control the library with a production instance was a headache. If the servers came down the time it would take to come back up due to log cleanup made it frustrating and problematic for other instances that could come up more quickly. Now with a dedicated library controller instance I don't have to worry about production servers waiting for the library to come back up since the library controller is small and fast when restarting. Check out the paper it even provides scripts to help ease some of the headache involved.

ACSLS Shared Library DRM Checkout

I was recently asked by a reader if I would show them how I do the checkout of my DRM tapes in an shared ACSLS library. If you are not aware, ACSLS does not really support true DRM checkout within a shared library environment. Unfortunately, what occurs when you run the checkout is that ACSLS tries to check the tapes out one at a time; prompting for a request response before checking out the next tape. I found this out when we implemented our STK L5510 library and the checkout process was broken. So I went searching for some kind of answer to the situation and found APAR IC45537 which states the problem has been known since March of 2005. IBM has not resolved the issue within TSM but has relied on a local fix to resolve this issue. So I decided to post the script I created here to help those people out who have an ACSLS library and are looking to share it between multiple TSM instances and don't want to have to deal with Gresham's software.

Here is a basic rundown of what the script does.

• On each library client DRM checks the tapes out with a REMOVE=NO option and creates a file with a list of the tapes.
• On the library manager the tapes are checked back into the library.
• The Library manager then checks every library clients DRM tapes out.

I know it sounds complex but it's not. It also is not perfect so you will need to keep on top of it, but don't worry it is not so bad that it's a headache. Some explaination is required also; In my environment I have 7 TSM instances and the firsts is just a library controller. This is why the following script starts at TSMSERV2 and stops the loop after TSMSERV7. Also note that the pipe sysmbol will not show up for some reason so you need to add it before the grep statements. Before you use the script you will also notice that it uses a TSM macro. Create the file by putting the following command into a macro called move_drmedia.mac.

move drm * so=dbb wherestate=mo tostate=vault remove=no CMD=&VOL CMDFILE=/usr/tivoli/tsm/client/ba/bin/Vol_List APPEND=YES WAIT=YES
move drm * so=dbb wherestate=vaultr tostate=onsiter


=-=-=-=-= Below Starts Checkout Script =-=-=-=-=

#!/bin/ksh

cd /usr/tivoli/tsm/client/ba/bin

OFFSITE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/offsite.txt
RETRIEVE=/usr/tivoli/tsm/client/ba/bin/OFFSITE/retrieve.txt
ADSMID=`cat /usr/local/scripts/ADSMID`
ADSMPASS=`cat /usr/local/scripts/ADSMPASS`
I=2

cp $OFFSITE "$OFFSITE.BAK"
cp $RETRIEVE "$RETRIEVE.BAK"
cp /usr/tivoli/tsm/client/ba/bin/Vol_List /usr/tivoli/tsm/client/ba/bin/Vol_List.bak

cat /dev/null > /usr/tivoli/tsm/client/ba/bin/Vol_List

printf "Use this list to determine tapes that are to go offsite report any discrepancies to the Recovery Services Team.\n\n" > $OFFSITE
printf " \n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE
printf " Tapes to be sent offsite\n\n" >> $OFFSITE
printf " Current as of: `date`\n\n" >> $OFFSITE
printf "=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-\n\n" >> $OFFSITE

printf "Use this list to determine tapes that are to come back onsite from Iron Mountain for reuse. Report any discrepancies to the Recovery Service Team.\n\n" > $RETRIEVE
printf " \n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE
printf " Tapes to be brought back onsite from Iron Mountain\n" >> $RETRIEVE
printf " and placed back into TSM library for scratch.\n\n" >> $RETRIEVE
printf " Current as of: `date`\n\n" >> $RETRIEVE
printf "********************************************************\n\n" >> $RETRIEVE

while [ $I -lt 8 ]
do
S=TSMSERV-$I

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S -dataonly=yes "select volume_name from drmedia where state='MOUNTABLE' " grep L[0-3] >> $OFFSITE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S -dataonly=yes "select volume_name from drmedia where state='VAULTRETRIEVE' " grep L[0-3] >> $RETRIEVE

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=$S 'macro move_drmedia.mac'

sleep 120

I=$(( $I + 1 ))
done

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=TSMSERV-1 'CHECKIN LIBVOL TSMLIB search=yes stat=private checklabel=no vollist=FILE:/usr/tivoli/tsm/client/ba/bin/Vol_List'

sleep 180

dsmadmc -id=$ADSMID -password=$ADSMPASS -servername=TSMSERV-1 'CHECKOUT LIBVOL TSMLIB vollist=FILE:/usr/tivoli/tsm/client/ba/bin/Vol_List'

mail -s "TDC Daily Tape Checkout" $HN tape_rpt < $OFFSITE
mail -s "TDC Daily Tape Return" $HN tape_rpt < $RETRIEVE

Shared Library Problem Solved?

Well after trying numerous things to get the problematic shared library working I finally got support on the line to fix this problem. The other day the problem reoccurred after about a week. I thought I had fixed the problem by changing the IP/VLAN the servers communicated over, and I thought it was until it all came crashing down. So I upgraded the problem instance to the same release level as the controller (controller has to be at or higher then the client servers). That didn’t fix it! I turned RESETDRIVES to NO and that didn’t do it either (Thanks for the advice Scott). So we got Tivoli on the line and Andy Ruhl with Tivoli support worked with us to identify the problem. First thing we did was check the ATAPE level and it turned out the controller was at a higher version. So we updated it to the same level as the controller. Then Andy found this APAR IC49066 which interestingly enough describes my problem. Turns out there was a known issue with this type of behavior and although it was due to be fixed in 5.3.4 it was added to the 5.3.3.2 update. Supposedly the explanation states it occurs when the client is accessing two different controllers, but Andy stated it was a misprint and it applies to single controller instances as well. So we updated all 5 TSM servers to the fixed level and hope we don’t experience any more issues. So far so good!

Help! Problem With Shared Library Client!

Ok, here is my problem.  I have a large shared library that supports 5 TSM servers. All of the TSM servers, save one, run without any problems. The one with problems is pathed to 32 of the 5 drives (they are designated just for its use) and periodically its mount requests go into a RESERVED state but never mount.  I have tried FORCESYNC'ing both sides to see if it was communication but that didn't work.  I have tried setting paths offline then back online and that didn't work either. The only thing that works is to cycle the TSM server then it comes back up and starts mounting tapes like nothing was wrong.  This happens on average every 24-48 hours.  This is a large DB client TSM instance so the reboots are hurting critical business apps backups. I don't see any glaring errors in the activity log or in the AIX errpt. Any suggestions would be gladly received.

LAN-Free In A Shared Library Environment

I had to setup a LAN-Free client recently and ran into some trouble with the tape environment.  As you all know I use a shared library environment where multiple TSM servers share a single large library through a library controller TSM instance.  This works great since it allows for a shared scratch pool and TSM can be bounced faster since the DB is less than 2GB.  The problem arises when trying to setup a LAN-Free agent in this complex environment.  If you have ever read the directions for LAN-Free you’ll notice they can be quite confusing since they switch the client port to 1502 to accommodate the LAN-Free agent’s default of 1500.  Now understand 1500 makes sense because the LAN-Free agent is really just a stripped down TSM server. Since the TSM server uses 1500 by default so does the agent.  I decided to use port 1502 for the agent but forgot to change it in the dsmsta.opt file. This kept me from being able to connect to the agent from the client since it was looking at port 1502 and the agent was still using port 1500.

Another issue the directions don’t fully explain is where to point the LAN-Free agent when setting it up.  I will explain how here and hopefully make it easy to understand.

1. Setup the dsm.sys client file with the following lines

ENABLELANFREE           YES
LANFREECOMMMethod       TCPIP
LANFREETCPServeraddress <loopback, DNS name, or IP>
LANFREETCPPort          1502

2. Define the agent to the node’s TSM server and the library controller as a server

define server storagent serverpassword=passw0rd hladdress=<loopback, DNS name, or IP> lladdress=1502

Also don’t forget to register the node if not registered already.

register node <nodename> <password> domain=<domain name>

3.  On the TSM library controller instance path the drives seen by the node/LAN-Free agent machine.  I am assuming you have connected the client node to the SAN environment and zoned a number of drives to the node.  When seen in the OS you can define the path. Make sure you map the paths correctly. In AIX this can be done by running the lscfg –vp | more command and noting the drive serials and noting them to their corresponding drive in TSM.

DEFine PATH STORAGENT <Drive Name> SRCType=SERVER AUTODetect=YES DESTType=DRIVE LIBRary=<Library Name> DEVIce=/dev/rmtX ONLine=YES

Note:   For ACSLS libraries more configuration parameters need to be set. I am showing an IBM hardware example since I would never use anything else!

4. On the client node go to the storage agent’s folder and update the dsmsta.opt file with the following

     tcpport 1502

5. Now from that storage agent directory run the following command

dsmsta setstorageserver myname=storagent mypassword=passw0rd myhladdress=<loopback,DNS name, or IP> servername=<node’s tsm server IP or DNS name> lladdress=1500 <or port used by node’s tsm server>

Note:  Notice I am pointing the agent to the node’s TSM server. Even though it is a library client and does not control the drives, by defining the agent to both the client TSM server and the controller instance a handoff from the library client to the library controller will occur automatically.  If you do not define the LAN-Free agent to the library controller the agent will fail.

You should now be able to test the configuration by starting the dsmsta in the foreground and then from another telnet window start a TSM command line client and run a backup(if using Windows open a dsmc client). If setup correctly you’ll see the agent contact the controller for a drive and mount a tape and commence a LAN-Free backup.

How To Setup A Library Manager Instance (Revised)

I had to revise this as I left out the checkin of the tapes before the audit.  I had to do this process this weekend as we moved to a new library and library manager and I realized I left out the tape checkin.

I was recently asked how to setup a dedicated TSM Library Manager instance to resolve issues with using a production backup instance as the manager. There are definite benefits to using a dedicated instance like performance and manageability. Performance wise TSM has DB issues and when large tasks are running all around performance is impacted. This impact affects library usage and so having the dedicated library manager allows for a smaller DB and no heavy tasks to bog down all the mount requests. Manageability is increased in that all you need to worry about are the drives and their paths.  That’s right, no backups to worry about (except the DB) and even if you lost the systems DB backup and had to start from scratch the other servers sharing this instance could easily reconcile their inventory and you would not lose any server data (so it’s also a lot safer to use than relying on a production TSM instance). If this instance is on a dedicated server also, do not backup the server to the library manager, back it up to one of the other instances. NO BACKUPS OR ARCHIVES SHOULD BE DONE TO THIS INSTANCE. The key to the dedicated instance is keeping the DB as small as possible and this is easily done if nothing else in run on the instance. When creating our library manager instance I was unsure how large to make the DB so I made it 2GB, and it currently is 4.6% utilized. The instance starts fast and cycling it for updates or when problems occur keeps downtime to a minimum. Ok so here are the steps to setup the instance, just remember there will be an outage needed on the current TSM library manager as you remove drives and paths.

Step 1: Create the dedicated instance

You can create the instance fairly quickly by creating a subdirectory on the TSM server like so:

mkdir /usr/tivoli/tsm/libserv/bin

Copy the dsmserv.opt and if planning to use the old web interface the dsmserv.idl file to this new directory.  Then define a DB and Log volume either using raw logical volumes or us dsmfmt to format one. Depending on your library size I would make it 500MB to be safe.  At that size its space should not be too much of a premium.  The log can be half that and still function fine.  If you want to be extra careful you can be like me and make it 1-2GB but again my DB is barely used at that size. Once you have created the volumes then you need to run dsmserv format to initialize the instance. If the instance is going to be on a server with other TSM instances you’ll need to export TSM variables so they point to the new /usr/tivoli/tsm/libserv/bin directory.  Export the following variables before running the dsmserv format command:

export DSMSERV_CONFIG =/usr/tivoli/tsm/libserv/bin/dsmserv.opt
export DSMSERV_DIR=/usr/tivoli/tsm/server/bin
export DSMSERV_ACCOUNTING_DIR=/usr/tivoli/tsm/libserv/bin

Now you can run dsmserv format and the instance will be initialized and ready to be used. In the ../libserv/bin directory you should now have a dsmserv.dsk file that lists this instance’s DB and Log files. You can at this point either use the instance as is or load the old web interface definitions so that they are available to those who prefer it over the ISC. Once ready to use you can now start the instance and setup its settings for Server-to-Server communication. Remember the other TSM instances have to be able to talk to this instance. Make sure you set the HL and LL address, set the server password, set the server name, the URL, and don’t forget to set crossdefine to ON.

Note: The next step can be done ahead of time if you can allow for a longer library outage.

Once you have set the parameters you feel are necessary, whether for security or event and actlog retention, you can then move over to the current TSM library manager and begin removing the paths and then the drives (in that order). TSM does not like conflicts so don’t try to define the drives on the new Library Manager ahead of time as you could cause serious problems with the library. Delete all paths and drives then delete the library path and finally delete the library itself. You can no longer use this library definition since it is an automated library and when sharing the library the non-library manager TSM instances use a Shared library definition.

Step 2: Define server to server communication for all servers sharing the library

This is quite easy and I wont cover this much as this is fairly straightforward. On all servers that will be sharing the library make sure you have set their server name, server password, HL and LL address, URL, and set crossdefine to ON. Then (using the old web interface) on the servers that will be library clients select Object View -> Server -> Other Servers. Here you can see the servers defined and define new servers. Select Define Server from the drop down menu and then put in the new library managements info into the corresponding fields. Make sure you crossdefine and then your ready to setup the library.

Step 3: Define drives and paths on new library manager

Back on the new library manager instance you can now define the library and the drives and paths. Make sure when defining the new automated library you select the shared option to allow the library to be shared by other instances otherwise it will be accessible only to this instance.

DEFINE LIBRARY ATL01 LIBTYPE=SCSI SHARED=YES AUTOLABEL=YES

Now that the library is defined you need to define the library path and all the drives and their respective paths, this includes the path definitions for every other TSM server that will be using this library. Remember no drive or path definitions are made on the library clients. The only definitions for the library clients are paths on the library manager. Once the drives and paths are defined you will need to checkin the tapes and audit the library. The library manager should see the tapes in the library when you decide to check them in and now you can progress to the next step. If you have tapes that are moving from another library check them in at this time. You have to make a decision at this point to either checkin the tapes now or wait until the other servers have completed Step 4. When you check them in you should check them in as private then after each TSM server reacquires its tapes you can update the private tapes not owned by any TSM server to scratch.  This can be seen when you do a Q LIBVOL and the easiest way to do this is to use a select call to create a list then run the list against a shell script to update each tape.

Here is the select command:

SELECT VOLUME_NAME FROM LIBVOLUMES WHERE OWNER IS NULL

Here is the shell script I use:

echo PROCESSING.........

while read LINE
do
  dsmadmc -id=<admin id> -pa=<admin password> upd libvol <library name> $LINE status=scratch
  echo $LINE updated
done < $1

Step 4: Define shared library and update device classes

Now that the library clients can communicate with the new library manager you will need to define a new library definition. Know that you can reuse the name from the old library settings you deleted if you like. Here we will be setting up a shared library definition (again from the old web admin interface); select Object View -> Server Storage -> Libraries and Drives -> Shared Libraries. From the drop down menu select Define Shared Library and fill out the needed information of Library Name and Primary Library Manager. The primary library manager info should point to the server name you defined for the new library manager instance. Since the server name you defined has all the TCP/IP and port settings it is important you use the same name as the name given to the new library manager.  Once the library definition is created you will need to update the device classes on each library client to use the new library. You can use the old web admin interface selecting Object View -> Server Storage -> Device Classes then select the device class type that was in use (i.e. LTO device class) and update each device class by selecting each one then selecting the drop down menu and choose Update Device Class. Update the library name field and if needed set the mount limit if you don’t want this server to be able to use all the drives in the library. Setting the mount limit is an easy way to pseudo-partition the library when its shared. I highly recommend using the mount limit option to allow each system some dedicated access to drives without having to designate specific drives using set path definitions for each library client.

Step 5: Run AUDIT LIBRARY on each TSM library client

Now that the library clients can see the library manager they need to reconcile the library inventory so they can use their tapes. This can be done by running the audit library command AUDIT LIBRARY <SHARED LIBRARY> CHECKLABEL=YES. This will allow each library client to contact the TSM Library Manager and tell it to reassign its tapes back. This audit should be done periodically to make sure each TSM server sees correct volhistory information. If once the volumes have been reconciled for all TSM library clients and you do not see any scratch, you can run a check-in to allow the scratch volumes to be recognized. One thing to be aware of, on the old TSM library manager you will need to delete all volumes listed as remote. This will free up tapes to go to scratch otherwise you’ll have “phantom” tapes that have no data, are not used in any storage pool for any servers, but will show assigned to the old library manager instance. You can run the following command to delete them from your volume history, unfortunately there is no blanket command that seems to work to remove them all at once:

DELete VOLHistory TODate=TODAY Type=REMOTE VOLume=NT1904 FORCE=Yes

What you will want to do is generate a list of volumes from the old library manager that are in REMOTE status and then run that list through this command to delete the volumes from its volume history file allowing them to go back to scratch status when needed.

I hope you have all found this helpful and if I missed a step please feel free to add your input.
    

TSM Library Sharing

TSM is truly an Enterprise Backup tool and one of the key selling points is how well it allows you to consolidate and centralize the backup environment. One feature I have used extensively is the library sharing option. This feature allows multiple TSM servers to use the same library without conflict. It can be easier than partitioning the library, allows for dynamic allocation of tape drives, and allows for a single shared scratch pool. The key to using a shared library is creating a separate TSM instance as a dedicated library manager (whether on the same server as another TSM instance or on its own hardware). The library manager instance will not perform any backups and will not be a big instance. From my experience it will have very little impact on the physical server. When creating our library manager instance I was unsure how large to make the DB so I made it 2GB, and it currently is 4.6% utilized. The library it manages is a 9 frame 42 drive 3584. The library supports 3 TSM backup instances (2 on one server and 1 on a separate physical server) with a total of 951 clients. We put the library manager instance on the server with 2 instances, thereby creating a third TSM instance. By creating the third instance dedicated to managing the library you will have increased uptime and when you need to do maintenance or restart the instance the library manager is back up very quickly. This will not be the case if you make a production backup TSM instance the library manager due to a production TSM servers need to run expiration and usually having a larger log size making restart time take that much longer.

The other nice feature of library sharing is the ability to dynamically allocate drives to specific TSM servers. I have seen some administrators take twenty drives and path ten to one TSM instance and ten to another but this defeats the purpose. If you setup the library manager and TSM backup instances correctly then you’ll allow every instance connected to the library manager to see all the drives. This coupled with the device class mount limit setting can allow you to change drive usage on the fly. So say you have two TSM servers TSM-A and TSM-B using a twenty drive 3584 managed by a library manager instance called TSM-C. Each TSM instance sees all twenty drives but you have their device class mount limits set to ten. Now each TSM instance can only use ten drives out of the twenty. Lets say on a specific day TSM-A server is in need of more drives and TSM-B is currently only using three drives. You could update the TSM device class on each to allow TSM-A fifteen drives and TSM-B five drives.

Recently we purchased some newer hardware to replace our 9 frame LTO-1 library and are looking at using two small AIX servers using HACMP as the library controller since the new library will need to be available at all times and will be supporting at least six TSM servers. Unfortunately I have had to partition this new library due to application demands. The problem we are experiencing with this new library is that the logical portioning requires us to use the Virtual I/O option in the 3584. This option allows the library to automatically insert volumes when placed in the I/O door. This would not be a problem if the library did not then require us to assign the volumes to a specific logical library. This is done through the library and not through TSM, which adds another process to our tape management. I would have preferred to not have partitioned the library and allowed a TSM library manager instance to handle allocation of tape drives, but alas I am not able to at this time (still awaiting the HACMP instances).