Poor Performance

Currently I work in an environment where we have a specific TSM instance for a large SAP DB (99TB currently). We just upgraded the drives in the tape library (yes we use tape! I know...I know....) from MagStar 3592 TS1130 (E06) drives to TS1140 (E07) drives. The upgrade was pushed in hopes of a jump in write/backup performance, but I was skeptical. TSM adds so much overhead you cannot use the RAW tape read/write numbers from any manufacturer. Typically IBM is somewhat reasonable with their numbers, but in this case I have seen NO performance increase what-so-ever.  Here is a query of the processes for storage pool backup.

UPDATE (04/04/2014):  Let me give you some more specs, we have the 99TB DB split between 4 TSM Storage Agents each having 4 8Gb HBA's. Each storage agent runs 4 sessions (allocates 4 drives) for their backup process. So all 4 storage agents account for 16 simultaneous sessions and it still takes over 24 hours to perform the 99TB backup. The backups are averaging around 70-78MB/sec. Is this a TSM overhead issue or do I have a tuning issue with the TDP and TSM? I'm getting less than 50% of the throughput I should see.

Here's the command that is run to execute the DB backup:

ksh -c export DB2NODE=7 ; db2 "backup db DB8   LOAD /usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a OPEN 4 SESSIONS OPTIONS /db2/DB8/dbs/tsm_config/vendor.env.7 WITH 14 BUFFERS BUFFER 1024 PARALLELISM 8 WITHOUT PROMPTING" ; echo BACKUP_RC=$?

PROCESS_NUM: 2667
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:54
   DURATION: 00 23:20:13
      BYTES: 6.0TB
 AVG_THRPUT: 75.87 MB/s

PROCESS_NUM: 2668
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 78.48 MB/s

PROCESS_NUM: 2669
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 77.99 MB/s

PROCESS_NUM: 2670
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.4TB
 AVG_THRPUT: 80.13 MB/s

I average anywhere from 75 to 80 MB/sec.  Here is the Magstar performance chart. I am using JB media, not JC so I do take a little hit in performance for that.

So with JB media I could get as high as 200MB/sec but I am not even 50% of that number.  Is there any specific tuning parameter I should look at that could be hindering the performance? 

FYI - The backup of the 99TB DB runs LAN-Free using 16 tape drives over 26 hrs.

Good TSM Oracle Backup Blog

I came across this website and it has a lot of TSM Oracle backup related information. If you perform Oracle TDP backups then check out this site.

Finally! MySQL backups with TSM

Computer Technology Review has an announcement from Zmanda stating they are releasing a TSM backup agent (Zmanda Recovery Manager for MySQL). The ZRM-TSM agent is available immediately. You can read the full article here. Those of you who have been asking for a TDP for MySQL, the wait is over.

TSM Sharepoint TDP

Has anyone used the Sharepoint TDP for TSM yet? I have a number of Sharepoint servers at work and would like to know what users thoughts are on its capabilities. Is it easy to use or a nightmare to implement? Does it backup well or is it slow? Does it adversely affect the systems performance? If anyone can give me some feedback I'd appreciate it.

Oracle TDP and HACMP 5.3 Incompatibility Resolved

Well I was have a heck of a time with this problem. It seems the 5.3 HACMP binaries are incompatible with the Oracle 64-bit API so the TDP would crash when attempting to backup a DB on a cluster. I was expecting a fix and was told it would be available by January. So I waited, and watched and never saw an update to the TDP. Frustration mounted until I was able to get ahold of a TDP developer who pointed out that it was the HACMP side that had to do the patching and now all is well. So if you are looking to use HACMP and backup Oracle DB's with the TDP make sure they have the latest patch levels installed or the 64-bit API will not work.

Got HACMP?

Well our group got stung again this last week. While working a HACMP implementation we were asked to install the Oracle TDP and TSM client. We installed the TSM client and all went well and we then installed the TDP and the DBA’s could not get the password file to generate. Each time they tried the utility would core dump. When trying to run the utility multiple ways it always core dumped. After calling support we were informed that the current 64-bit Oracle TDP (And from what I have found through Google it could be any 64-bit TDP). It seems they rebuilt HACMP version 5.3 and the TDP is not compatible with the new way it was compiled. There is a patch projected to be released as of end of the month, but we cannot wait that long so our options are to use temporary disk and mirror the DB, then break and backup the mirrored copy when we want to backup, or we can roll back to HACMP 5.2. The problem with the latter is that we basically have to rebuild the boxes. So be warned there are issues with the latest and greatest version of HACMP at this time.

Oracle RMAN Catalogue Cleanup

Why do people love Oracle? When I hear mention of Oracle I think of Luke Skywalker when he saw the Millenium Falcon, "What a piece of junk!" Like the Falcon it looks clunky, breaks down easily, and has the most tempermental behavior. When it's running, however, it screams. The problem is that the RMAN catalogue sometimes doesn't do appropriate cleanup. If you want to make sure that a particular node is performing cleanup within TSM run the following select command -

select object_id from backups where node_name= and
backup_date < '2005-07-01 00:00:00'

This can be redirected to a file then used later to delete with an undocumented delete command. I will give the delete command out to those who need it, but remember any deletion from the TSM DB is done AT YOUR OWN RISK! It's unsupported because Tivoli doesn't trust you to not screw stuff up, and although I don't think you will, it's better safe then sorry.