TSM 6.2.3 and Lower DB Reorg Issue

One of our TSM servers started to experience large numbers of "ANR0530W - internal server error detected messages."  With further investigation we identified that these were related to ANR0162W which are DB deadlock or timeout problems. These errors were causing our DB TDP backups to fail and I eventually called support. I provided our db2diag.log file and a dump of our actlog for the last 24 hrs. and they found the issue to be the DB2 reorg process was locking records and tables and creating the deadlock situation. The problem was compounded in that with TSM versions 6.2.3 or lower the DB2 reorg process cannot be schedule so it can kick off during backups or processes that can create these deadlocks. So to resolve the issue I was told I needed to upgrade our TSM server to 6.2.4 or higher. With TSM 6.2.4 and higher you can schedule the reorg process using the REORGBEGINTIME and REORGDURATION parameters to schedule the reorg within a window. You can see the details of the APAR here.

DBMEMPERCENT...Where'd That Come From?

I was having performance issues with a couple TSM 6.2 servers and could not find anything that pointed to the issue. I'm not one to call support unless I'm totally stumped and cannot find help through the web, but this time I finally relented and made the call. The issue was problems with backups failing repeatedly and when researched we were getting internal server errors along with DB table errors. IBM support asked for some DB2 log files and within 30 or so minutes had identified the problem.

TSM has a server option I have never used or heard of that had somehow been set that adversely affected all backups. Somehow the option DBMEMPERCENT was set in the dsmserv.opt file. This option tells TSM what percentage of the overall server's memory it can allocate for use. The default is AUTO and would have been fine, but somehow DBMEMPERCENT was set to 10 in the dsmserv.opt. Which means out of 16GB of RAM I was only using 1.6GB?!?!? How'd that happen? I didn't set it, none of my coworkers remember setting it, so where did it come from? IBM support stated the default was AUTO so the option was manually set. Since I had never used this option and its 6.x specific, I never would have looked for it. Good thing I called support.

db2adutl Error

I recently had an issue with a client and storage agent upgrade that resulted in problems with the db2adutl utility being unable to return any data. Here's the errors:

Retrieving

Error: Begin query image failed with TSM return code 136

Error: Get next image failed with TSM return code 2041

I pretty much knew what caused the error, the problem was how to fix it. The cause was due to an upgrade of the TSM client on the DB2 server that (after further investigation) could not support the more current TSM Storage Agent. An OS patch would have to be applied, however, that could not be done without an outage. Our only option was to roll the client back to a supported TSM client / storage agent level. The problem was that while attempting to figure out a better solution than rolling back the client the DB2 database had run a backup.  When the client API was rolled back it could not "interpret" the new API's backup causing the db2adutl errors.

Support suggested renaming the node or the file space (file space is better since you don't have to stop and start db2 to reset the password as you would with the new node name). I didn't want to have to do either. The backups taken since the rollback were good, but db2adutl couldn't return the list of backups as long as the objects done with the newer API were still present. Luckily I have been dealing with Oracle admins long enough to have a solid grasp on manually deleting objects on the TSM server. When Oracle DBA's neglect their RMAN duties, I pulled out my trusty delete object command and I was able to remove the backup objects from the period of time that the new API had been used. Once completed db2adutl was able to immediately see it's backups and return a list of what was available.

TSM Server 5.5, 6.1, & 6.2 Bug

I figured I best share with you all a bug I have experienced with TSM Server version 6.2.2 that has resulted in the TSM instance core dumping. The problem stems from TSM being unable to mount a scratch tape with, in our case, a TSM storage agent. When the mount fails it seems TSM lists an internal server error then core dumps a few seconds later.  The fix is in 6.2.3 and higher. It also affects 5.5 and 6.1 versions, check this technote for further details.

Warning: This update has a bug also concerning Data Domain that causes errors if the defined VTL has more than 4400 slots. So when we upgraded the server to fix the LAN-Free bug we inadvertently encountered this bug.

Update: Here is the APAR listing for the Data Domain VTL slot bug in TSM 6.2.x

Memory Efficient Backup

I have come across a few servers that have memory issues when backing up large drives / filesystems and using MEMORYEFFICIENTBACKUP was not resolving the backup failures. The error code I would see was

ANS1030E The operating system refused a TSM request for memory allocation.

The problem was one drive that has millions of small files all under one directory. It's one of those poorly devised custom programs, and I've seen it a hundred times.  I could journal the drive, but we seemed to get better results with an alternative method.  A coworker reminded me of the INCLUDE.FS option using DISKCACHE and specifying the cache location.

INCLUDE.FS G: MEMORYEFFICIENTBACKUP=DISKCACHEMETHOD DISKCACHELOCATION=L:\TSM_cache

With the include.fs TSM will only use the memoryefficientbackup diskcache method for that particular drive, allowing the rest of the backup to perform normally. So if you have a drive or filesystem with millions of files this is an alternative to journaling.

TSM 6.2 GUID Error

So when I and a co-worker were finishing up a TSM 6.2 install we ran into a problem running the tsmapipw command to set the $$_TSMDBMGR_$$ password. After investigating the error and search the web we realized the GUID was the problem and not our IP as was suggested in the following APAR.  The errors we received were:

ANR2987W Session ended because of machine GUID or local host IP
 address mismatch.
ANR0430W Session XXXX for node $$_TSMDBMGR_$$ (AIX) refused
 - node name is locked.

We tried various things and nothing seemed to correct the GUID issue. The TSM server started without any GUID related error (in fact it stated the GUID initialized successfully). So upon further searching we found this APAR that gives the solution.  What they state is that by adding the following line to the dsmserv.opt and restarting the server fixes the problem.

DBMTRUSTEDGUIDIGNORE YES

Once we added the line and restarted the TSM server the tsmapipw command ran successfully.

TSM 6.2 GUID Error

So when I and a co-worker were finishing up a TSM 6.2 install we ran into a problem running the tsmapipw command to set the $$_TSMDBMGR_$$ password. After investigating the error and search the web we realized the GUID was the problem and not our IP as was suggested in the following APAR.  The errors we received were:

ANR2987W Session ended because of machine GUID or local host IP
 address mismatch.
ANR0430W Session XXXX for node $$_TSMDBMGR_$$ (AIX) refused
 - node name is locked.

We tried various things and nothing seemed to correct the GUID issue. The TSM server started without any GUID related error (in fact it stated the GUID initialized successfully). So upon further searching we found this APAR that gives the solution.  What they state is that by adding the following line to the dsmserv.opt and restarting the server fixes the problem.

DBMTRUSTEDGUIDIGNORE YES

Once we added the line and restarted the TSM server the tsmapipw command ran successfully.

Weird Query Results From 6.1.4

So we have a couple servers we upgraded from 5.3 to 6.1.4.2 and since the upgrade the following select statement does not work. I posted it on ADSM.org and others report their 6.x servers run it without issue.  Here is the statement and my results:

select server_name, node_name, PLATFORM_NAME || ' ' || cast(CLIENT_OS_LEVEL as char(20)) as OS_Version, cast(client_version as char) || '.' || cast(client_release as char) || '.' || cast(client_level as char) || '.' || cast(client_sublevel as char(2)) as TSM_Version from nodes, status


here is the return code I get:


ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.

I run this through a while loop and append the data to an comma-delimited output file so I can pull it into a spreadsheet and organize clients by version. I can then more easily prioritize my client upgrades.  I really need this to work so I can sort and identify the clients location by server_name. Any ideas?

Weird Query Results From 6.1.4

So we have a couple servers we upgraded from 5.3 to 6.1.4.2 and since the upgrade the following select statement does not work. I posted it on ADSM.org and others report their 6.x servers run it without issue.  Here is the statement and my results:

select server_name, node_name, PLATFORM_NAME || ' ' || cast(CLIENT_OS_LEVEL as char(20)) as OS_Version, cast(client_version as char) || '.' || cast(client_release as char) || '.' || cast(client_level as char) || '.' || cast(client_sublevel as char(2)) as TSM_Version from nodes, status


here is the return code I get:


ANR2034E SELECT: No match found using this criteria.
ANS8001I Return code 11.

I run this through a while loop and append the data to an comma-delimited output file so I can pull it into a spreadsheet and organize clients by version. I can then more easily prioritize my client upgrades.  I really need this to work so I can sort and identify the clients location by server_name. Any ideas?

We found this bug

We found this bug in Hungary. Click for details!

Who Did What?

This weekend I had an issue with a backup failing on a client using LAN-Free. When checking the TSM actlog the error I kept seeing was:

ANR9999D (Session: 265500, Origin: OD0BG-UAX001-STA) mmsshr.c(3874): ThreadId<13> Unable to obtain model type for '/dev/rmt23', rc = 46(SESSION: 265500)

So I had a good idea that the client and TSM Storage Agent were having issues communicating. I knew this because I had seen this error before when the TSM client and TSM Storage Agent were not at the same release levels. Checking the TSM client I found it at 5.4.0.0 and the Storage Agent at 5.3.4.0. Turns out an admin upgraded the client without knowledge of the Storage Agent, causing the backup failures. This is an example of what can happen when someone tries elbow their way into another groups area without the knowledge to do it right.