Simple TSM 6.2 Server Restore

I just completed a DR test and we had to restore one of our TSM servers from a Data Domain replicated copy. This was our first time restoring a TSM server from a replicated DD copy and after importing the replicated volumes and defining our initiators we set about restoring the TSM database. Our AIX server had been restored from an image (SysBack) and we had a current volhist and devconfig file so we began our restore. If you think that the restore from a Data Domain is not relevant to your environment because you use tape, think again. The Data Domain mimics an STK library with IBM drives and so we had to follow the same directions as anyone using tape backup.

To restore the TSM 6.x DB from tape you must have your volhist and devconfig files. You will need to modify the devconfig so that the only lines are those defining the devclass, server name, and server password; all other lines should be deleted. Then you need lines defining a manual library, a tape drive, and a line defining a path to the drive (which for us was an LTO3 drive).


DEFINE LIBRARY MANLIB LIBTYPE=MANUAL
DEFINE DRIVE MANLIB DRIVE1 ONLINE=YES 
DEFINE PATH TSMSERV1 DRIVE1 SRCT=SERVER DESTT=DRIVE LIBR=MANLIB DEVICE=/dev/rmt1 ONLINE=YES

Note: Do not define an element address or serial with the drive, TSM will detect these when you run the DSMSERV RESTORE DB command.

When running the DSMSERV RESTORE DB command TSM will start up and query the devconfig file to retrieve the information on the devclass, drive, library type, server name, and password. Once TSM has successfully queried the tape drive it will query the volhist file for the most current DB backup volume depending on whether you are restoring to the most current date or to a specific point in time. When TSM has identified the volume to use it will prompt you to mount the tape. When I saw the mount I went into the Data Domain web based GUI and moved the DB backup volume from its "virtual slot" to the drive that is /dev/rmt1. Once the tape was mounted, TSM was able to recognize the tape had been loaded and began restoring the DB. If more than one tape is required to complete the restore TSM will prompt you for each tape. With the library web GUI available you can move the tapes as needed and accomplish the restore. Once the restore completes you can bring TSM back up and audit/fix anything that could be out of sync. With the switch to DB2 I was expecting a little more work to get TSM back up and running, but surprisingly it was quite simple.

Now if you don't have a SysBack of your TSM server the rebuild can take a lot longer and requires you to recreate some of the DB2 dependent files. I might have to do a BRM restore without an image in the near future and if I do I'll post a step by step process for everyone.  If anyone has already done this and would like to post the process on TSMAdmin let me know.

Update: Weird Volhist Records (Part Two)

IBM did get back to me about my library manager/library client volume history devclass issue and I found it interesting how development handled it. Initially IBM support said the following:

Our server development has confirmed with our design, it is possible that the remote volhist entry can be different. Development believe this is their design. However, the existing document did not document this clearly. They agreed to open a Doc. APAR to properly document this for command query volhist.

In other words they felt that no action was needed other than to document that this possible occurrence within the TSM Admin and Reference guides. So I asked it be escalated since it is definitely a flaw and finally heard back from support and was told the following:

They agree what you observed or reported in this case is incorrect even though it does not cause any function lose. They have agreed to take the APAR IC65048 as a defect ( instead of Document ). However, this APAR would take a big code change to "fix" this issue and after first evaluate, they will not be able to deliver a fix in the service stream. They request to open a DCR ( design change record ) so development can make this change on a release boundary so that there is sufficient testing for a code change.

So it looks like IBM will eventually fix this. Thanks to IBM support for helping me get development to at least go beyond the Doc. APAR.

Update: Weird Volhist Records

Upon further investigation I found that my other shared library environment, that has multiple NAS devices defined to the library manager, is also showing the NAS device class for all LTO-3 volumes used by the library clients. If anyone has a NAS attached to a shared library environment please compare your volhist devclass to those of your library clients and tell me if you see the same situation. CRAZY THING IS, it doesn't seem to affect the library clients from using their tapes or acquiring scratch. What my concern is, is that I am slowing losing use of tapes that should be marked back as scratch, but due to the volhistory settings is unable to be released. I am wondering if this is a bug in TSM that could be affecting others.

Weird Volhist Records

I am reviewing my Volume History file on my ACSLS library controller and it shows volumes as REMOTE and owned by the library clients but the weird thing is it shows the DEVCLASS as a device class that is not even defined on the library clients. Why would TSM do this? So I have a device class called IBM-LTO-3 on all my TSM instances (8 total including the library manager) and on the library manager there is a device class called NAS-DEV-CL. I show tons of tape volumes using the NAS-DEV-CL devclass on the library manager's volume history but not on the other instances (since they don't even have the NAS-DEV-CL devclass defined). Why or how would this occur? I tried an audit library on one of my library clients but it is not completing (I've had issues with audit library commands from library clients before). Any ideas?

Here is an example of what I see:

tsm: PD-703-S-AITSM-1>select * from volhistory where volume_name='L40202'

DATE_TIME: 2009-08-19 03:33:07.000000
UNIQUE: 0
TYPE: REMOTE
BACKUP_SERIES:
BACKUP_OPERATION:
VOLUME_SEQ:
DEVCLASS: NAS-DEV-CL
VOLUME_NAME: L40202
LOCATION: PD703-UAX007
COMMAND:

When I query the volume on the library client I see:

Volume Name: L40202
Storage Pool Name: NP-STD-TAPE
Device Class Name: IBM-LTO-3
Estimated Capacity: 1.6 T
Scaled Capacity Applied:
Pct Util: 0.4
Volume Status: Filling
Access: Read-Only
Pct. Reclaimable Space: 26.1
Scratch Volume?: Yes
In Error State?: No
Number of Writable Sides: 1
Number of Times Mounted: 14
Write Pass Number: 1
Approx. Date Last Written: 08/26/09 11:01:12
Approx. Date Last Read: 09/22/09 11:19:00
Date Became Pending:
Number of Write Errors: 0
Number of Read Errors: 0
Volume Location:
Volume is MVS Lanfree Capable : No
Last Update by (administrator): CSMALL
Last Update Date/Time: 09/14/09 13:14:21
Begin Reclaim Period:
End Reclaim Period:
Drive Encryption Key Manager: None

So the question is how did this volume history record on the library manager end up with the devclass that doesn't exist on the library client? It doesn't seem to affect the volumes or processing daily tasks, but I am worried its not freeing up the tapes to return to a scratch status when they are reclaimed.

Volume History Issues

I have a very large shared library environment and currently have 3 TSM instances connecting to a 9 frame 3584. When I first configured this library we had one of the 3 instances running as the library controller (2 of the instances are on the same server the third is another system). The problems we ran into were numerous when we needed to do library maintenance since we adversely affected the server running as the library manager that was also a production TSM server for backups. So after a suggestion from Tivoli that we at least create a library manager instance we did so. It has helped a lot with handling tape issues, but one issue we were unaware of and had difficulty in resolving was with the volume history on the old library manager not releasing volumes listed as REMOTE. The new library manager could not force the old manager to "let go" of the tapes so they never were freed back into the scratch pool. We tried deleting all volumes of TYPE=REMOTE but it said it needed more parameters. Here is an example:

DELete VOLHistory TODate=TODAY Type=REMOTE FORCE=Yes
ANR2022E DELETE VOLHISTORY: One or more parameters are missing.

So no luck on that working to free up all the REMOTE tapes. I looked through the documentation on deleting volume history information and found nothing on REMOTE volumes. Also you'll notice there is nothing mentioned in the documentation that states individual volumes can be deleted. So we were in a serious jam. So I looked up the issue on ADSM.org and found the following command posted by a contributor.

DELete VOLHistory TODate=TODAY Type=REMOTE VOLume=NT1904 FORCE=Yes

This command allowed us to delete the individual volume and freed the tape to return to a scratch status. Thank goodness for search.ADSM.org or I'd never find the answer to half my problems. This command worked as advertised and the tapes were deleted from the old library managers volume history and they went back to a scratch status.