Unload/Load of DB pitfall

I am writing this article to save others from the situation we had at one of our customers. In the beginning a request came for replacing the RAID array which hosts the primary storage pools. I taked that as an excellent opportunity for reorganizing the DB which was 66GB with cca 50% utilization (after a massive deletion in last month or so). There was a planned downtime needed for copying the data from one RAID to another so I thought I had a plenty of time for unload/load.
The "dsmserv unloaddb" utility went smoothly - taking about 4hrs, creating cca 20GB dump (which was expected as the ESTIMATE DBREORGSTAT revealed we can save cca 7GB). Then I proceed with "dsmserv loadformat" and "dsmserv loaddb" (taking approx. 5hrs). So far everything was seemingly OK. I was able to start the server, reduce the DB and run some tests. The problem appeared when I tried to create a new stgpool


12/13/05 12:42:21 ANR2017I Administrator HARRY issued command: DEFINE STGPOOL archivedisk disk (SESSION: 3884)
12/13/05 12:42:21 ANR0102E sspool.c(1648): Error 1 inserting row in table "SS.Pool.Ids". (SESSION: 3884)
12/13/05 12:42:21 ANR2032E DEFINE STGPOOL: Command failed - internal server error detected. (SESSION: 3884)


Google revealed that this error is a known one and is fixed in 5.3.2.1 (we was on 5.3.2.0 - upgraded to this level just a few days before the fix appeared .. bad luck). Basically - there is an error in LOADFORMAT/LOADDB corrupting some DB values


http://www-1.ibm.com/support/docview.wss?uid=swg1IC47516


I did not want to go for loaddb again (there were changes already made to the DB, some migrations were run (luckily for me they were from stgpool with caching set on) .. etc.
So I tried to run the "dsmserv auditdb inventory fix=yes" - IBM says it can help if you do it after a loaddb - long story short - after 8hrs of audit (with message that some vaules were corrected) the problem was still there ...
So the only option was to apply the patch and do use loaddb again - so another 4hrs of waiting - and now it seems to work (still running tests). So watch for this problem and check your TSM level before reorganizing your DB.

System State Backup Issue

We recently had to rebuild a box (thank goodness it was not production) and found that the system state and objects had not backed up.  Stranger still there was no error stating they failed. In fact the only error was that the LVSA directory was not set and it would not be used during the backup.  Why would a failure to completely setup the LVSA affect the system objects and state? TSM doesn't require the LVSA to back them up.  We checked multiple systems and found this happening on more than one.  The client level we are on is 5.2.4 and higher.  Has anyone else experienced this?  BTW - The backup schedules were showing in the event log as COMPLETED not Failed.

Got HACMP?

Well our group got stung again this last week. While working a HACMP implementation we were asked to install the Oracle TDP and TSM client. We installed the TSM client and all went well and we then installed the TDP and the DBA’s could not get the password file to generate. Each time they tried the utility would core dump. When trying to run the utility multiple ways it always core dumped. After calling support we were informed that the current 64-bit Oracle TDP (And from what I have found through Google it could be any 64-bit TDP). It seems they rebuilt HACMP version 5.3 and the TDP is not compatible with the new way it was compiled. There is a patch projected to be released as of end of the month, but we cannot wait that long so our options are to use temporary disk and mirror the DB, then break and backup the mirrored copy when we want to backup, or we can roll back to HACMP 5.2. The problem with the latter is that we basically have to rebuild the boxes. So be warned there are issues with the latest and greatest version of HACMP at this time.

Setting Up A Secondary TSM Instance

Someone requested a post on how to setup a secondary instance of TSM on a UNIX server so here is the skinny on how that is setup:

First create a directory where the config files for the new server will be stored.

mkdir /usr/tivoli/tsm/serverb
mkdir /usr/tivoli/tsm/serverb/bin

then copy the dsmserv.opt over and modify the needed settings in it like devconfig and volhist to save in the new dir.  Then create DB and Log volumes that this instance will use. Once those are created you need to export the following environmental variables:

export DSMSERV_CONFIG=/usr/tivoli/tsm/serverb/bin/dsmserv.opt
export DSMSERV_DIR=/usr/tivoli/tsm/serverb/bin
export DSMSERV_ACCOUNTING_DIR=/usr/tivoli/tsm/serverb/bin

Now you can run the dsmserv format command to initialize the DB and Log volumes and it will create the dsmserv.dsk in the serverb directory.  Make sure you run the dsmserv runfile commands to load the scripts and webimages (even if TSM 5.3).  The final step is to create the startup script so that TSM initializes correctly.  Here is our script:

#!/bin/ksh

exec > /tmp/libserv.out 2>&1  #optional -> sends output to out file

ulimit -d unlimited

export DSMSERV_CONFIG=/usr/tivoli/tsm/serverb/bin/dsmserv.opt
export DSMSERV_DIR=/usr/tivoli/tsm/serverb/bin
export DSMSERV_ACCOUNTING_DIR=/usr/tivoli/tsm/serverb/bin

print "$(date '+%D %T') Starting Tivoli Storage Manager Server"
cd /usr/tivoli/tsm/serverb/bin

dsmserv

We use the following command to start TSM so we don’t have to deal with nohup:

echo “/usr/tivoli/tsm/serverb/bin/rc.adsmserv” | at now

This uses the at command to run the script immediately.  You can then edit the inittab and place a line in to start this instance on boot or put a script in the run level startup folders, your choice.  You should now be ready to run the second instance.  One note to those thinking of doing this and sharing a library, Tivoli recommends that you create a TSM instance to be just a library manager, no clients, no real work other than handling the library and tape mounts.  I agree with this and it has been a lot easier to manage and handle library issues.  Not knowing how large the DB could get I gave it 2GB and it is currently 3.8% utilized, and it has been in place for over a year and a half. Swapping a library manager from one system to another is not as hard as it would seem so consider it and if anyone wants docs on how to do the switch let me know I’ll post it.    

UNIX Permission Issues

Awhile back I ran into an issue with how TSM handles permissions on UNIX files and wanted to get some feedback from you readers out there on how you would handle it.  What happened was a user somehow was given root and he chown’ed the /home dir recursively.  It was made worse by the fact that he did that on Friday and didn’t alert anyone until the following Monday, and by the time it got to us another day had passed.  The customer of course wanted us to restore the directory and file permissions, but the kicker was that TSM does not back a UNIX file up again when the permissions change.  It just updates the database to reflect the permission changes (I got that directly from support and was floored; I had no idea it handled UNIX that way).  So here was our dilemma, if the file was the only version in backup I would not have any way of resetting its permissions.  Is the gravity of the situation hitting home?  Because it doesn’t backup the file again or track permissions I could not successfully restore to a point-in-time.  Sure I might get a good portion of the files fixed but there would still have been a large portion that we would be unable to get the permissions corrected.  The customer wasn’t happy and our only out was that the customer should not have been doing chown‘s as root.  I thought I once saw someone post a undocumented option you can set in the options file that will backup a file if it changes in any way, permissions included, but I can’t find it.  I thought I saw it on the new ADSM.org but am unable to locate it.  Anyone know the option or have an idea on how to approach this?  I brought it up with some Tivoli people who asked me what I thought should be added or changed in TSM, but so far I haven’t seen any change in their processing.          

Poll Results

I closed the poll covering things we would like to see in future releases of TSM. I expected the removal of the ISC interface to win hands down but actually it was spread pretty even between 5 of the selections. Although Get rid of ISC interface and Return of the old web interface were the two higher vote getters I was suprised how evenly spread the voting was. I personally voted for conversion to DB2 database but I have been whining about that for years...ask the Tivoli folks I am sure they are familiar with my whining (I'm hoping for a cheese basket this christmas from the developers...oh and please no stinky cheese the wife is pregnant and foul odors make her sick).

ISC+TSM AC

Many people have been complaining about the ISC+TSM AC and I have been one of them.  The concerns have been plentiful, “We have to learn a new interface”, “There’s no DRM functions”, and “It requires a server just to support it!”  All these are valid protests and the one that bothers me the most is the fact that in a DR situation you would have to rebuild the ISC system/instance along with the TSM server to have web accessibility.  This adds time to an already urgent situation.  So what good does the ISC+TSM AC provide?  For starters a single interface for accessing all your servers, a single login, more functions when it comes to hardware management, and in the event of a disaster it forces you to learn the command line.  I know the last item might frustrate a lot of people but the truth is you need to know the command line to be a proficient TSM administrator.  I love the web interface and I recommend TSMManager, but when in a DR situation you have to be able to handle the command line if you want to get back up and running. Granted you have no other choice than command line until the system comes back up and is running again, but afterwards you’ll need to do typical admin work and the ability to do it through the command line will increase you rebuild speed helping you meet SLA time frames.  I don’t think Tivoli had this in mind when they went to the ISC+TSM AC but in my opinion too many people rely on the web and don’t learn the commands needed to be truly proficient.  I find it amazing how many don’t even know how to use the HELP command.  So I could complain about the change in interface but change happens and although we don’t always like it (I don’t care for this one) we have to be able to change and adapt with it if we want to last in this work force.