DBMEMPERCENT...Where'd That Come From?

I was having performance issues with a couple TSM 6.2 servers and could not find anything that pointed to the issue. I'm not one to call support unless I'm totally stumped and cannot find help through the web, but this time I finally relented and made the call. The issue was problems with backups failing repeatedly and when researched we were getting internal server errors along with DB table errors. IBM support asked for some DB2 log files and within 30 or so minutes had identified the problem.

TSM has a server option I have never used or heard of that had somehow been set that adversely affected all backups. Somehow the option DBMEMPERCENT was set in the dsmserv.opt file. This option tells TSM what percentage of the overall server's memory it can allocate for use. The default is AUTO and would have been fine, but somehow DBMEMPERCENT was set to 10 in the dsmserv.opt. Which means out of 16GB of RAM I was only using 1.6GB?!?!? How'd that happen? I didn't set it, none of my coworkers remember setting it, so where did it come from? IBM support stated the default was AUTO so the option was manually set. Since I had never used this option and its 6.x specific, I never would have looked for it. Good thing I called support.

ANS1035S (RC406) Options file '*' could not be found.

Came across this the other day whilst installing TSM for Enterprise Resource Planning on an AIX 5L server:

Data Protection for mySAP.com(R) technology

Interface between SAPDBA Utilities and Tivoli Storage Manager

- Version 3, Release 3, Modification 10 for AIX LF 64-bit -

Build: 219 compiled on Dec 4 2003(c) Copyright IBM Corporation, 1996, 2003, All Rights Reserved.
BKI5014E: Tivoli Storage Manager Error:

ANS1035S (RC406) Options file '*' could not be found.

BKI5014E: Tivoli Storage Manager Error:ANS1035S (RC406) Options file '*' could not be found.

For a while now I have been symlinking the configuration files back to a central repository to make administration easier, so checking the symlinks is the most obvious place to start:

# ls -ltr /usr/tivoli/tsm/tdp_r3/ora64 | egrep "dsm.opt|dsm.sys"

lrwxrwxrwx 1 root system 16 Nov 20 14:34 dsm.sys -> /var/tsm/dsm.sys

lrwxrwxrwx 1 root system 20 Nov 20 14:35 dsm.opt -> /var/tsm/backint.opt

Does backint.opt exist and is it valid?

# cat /var/tsm/backint.opt

SErvername TSM_BACKINT

* TRACEFILE /tmp/trace.out

* TRACEFLAGS api api_detail verbinfo verbdetail timestamp

What about DSM.SYS?
# cat /var/tsm/dsm.sys

SErvername TSMPRD

COMMmethod TCPip

TCPPort 1500

TCPServeraddress

passwordaccess generate

resourceutilization 2

inclexcl /var/tsm/tsm.include

schedlogname /var/tsm/log/dsmsched.log

schedlogretention 14

errorlogname /var/tsm/log/dsmerror.log

errorlogretention 14

queryschedperiod 2

schedmode prompted

ManagedServices Schedule WebClient
SErvername TSM_BACKINT

COMMmethod TCPip

TCPPort 1500

TCPServeraddress

TCPNodelay yes

TCPBuffSize 32

TCPWindowSize 64

LargeCommBuffers yes

TXNByteLimit 25600

passwordaccess prompt

resourceutilization 2

* inclexcl /var/tsm/backint.include

* schedlogname /var/tsm/log/dsmsched.log

schedlogretention 31

errorlogname /oracle/DEV/sapbackup/backint_error.log

errorlogretention 14

queryschedperiod 2

schedmode prompted

So everything was looking to be configured correctly, so I attempted to turn on tracing on the TSM API but as the TDP could not find the option file it would not work.
The next logical step was to trace the client and figure out where it was dieing:

# truss brbackup -u / -c force -t offline_force -m all -p initDEV.sap -a -c force -p initDEV.sap -cds
execve("/usr/tivoli/tsm/tdp_r3/ora64/backint", 0x2FF22CCC, 0x2FF22CE4) argc: 5

kload("/usr/lib/nls/loc/en_US__64", 128, "") = 4026536384

kload("libi18n.a(shr_64.o)", 384, "/usr/ccs/lib") = 4026536536

access("/usr/lib/nls/msg/en_US/backintv3.cat", 0) = 0

open("/usr/lib/nls/msg/en_US/backintv3.cat", O_RDONLY) = 3

open("/usr/tivoli/tsm/client/api/bin64/en_US/dsmclientV3.cat", 04) = 0

access("/usr/tivoli/tsm/client/api/bin64/en_US/dsmclientV3.cat", 04) = 0 access("/usr/tivoli/tsm/client/api/bin64/en_US/dsmclientV3.cat", 04) = 0open("/usr/tivoli/tsm/client/api/bin64/dsm.sys", O_RDONLY|O_LARGEFILE) Err#2 ENOENT

Bingo - as it turns out I had forgotten to also create my symlinks to the configuration files in the "client/api/bin64" directory.