Poor Performance

Currently I work in an environment where we have a specific TSM instance for a large SAP DB (99TB currently). We just upgraded the drives in the tape library (yes we use tape! I know...I know....) from MagStar 3592 TS1130 (E06) drives to TS1140 (E07) drives. The upgrade was pushed in hopes of a jump in write/backup performance, but I was skeptical. TSM adds so much overhead you cannot use the RAW tape read/write numbers from any manufacturer. Typically IBM is somewhat reasonable with their numbers, but in this case I have seen NO performance increase what-so-ever.  Here is a query of the processes for storage pool backup.

UPDATE (04/04/2014):  Let me give you some more specs, we have the 99TB DB split between 4 TSM Storage Agents each having 4 8Gb HBA's. Each storage agent runs 4 sessions (allocates 4 drives) for their backup process. So all 4 storage agents account for 16 simultaneous sessions and it still takes over 24 hours to perform the 99TB backup. The backups are averaging around 70-78MB/sec. Is this a TSM overhead issue or do I have a tuning issue with the TDP and TSM? I'm getting less than 50% of the throughput I should see.

Here's the command that is run to execute the DB backup:

ksh -c export DB2NODE=7 ; db2 "backup db DB8   LOAD /usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a OPEN 4 SESSIONS OPTIONS /db2/DB8/dbs/tsm_config/vendor.env.7 WITH 14 BUFFERS BUFFER 1024 PARALLELISM 8 WITHOUT PROMPTING" ; echo BACKUP_RC=$?

PROCESS_NUM: 2667
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:54
   DURATION: 00 23:20:13
      BYTES: 6.0TB
 AVG_THRPUT: 75.87 MB/s

PROCESS_NUM: 2668
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 78.48 MB/s

PROCESS_NUM: 2669
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 77.99 MB/s

PROCESS_NUM: 2670
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.4TB
 AVG_THRPUT: 80.13 MB/s

I average anywhere from 75 to 80 MB/sec.  Here is the Magstar performance chart. I am using JB media, not JC so I do take a little hit in performance for that.

So with JB media I could get as high as 200MB/sec but I am not even 50% of that number.  Is there any specific tuning parameter I should look at that could be hindering the performance? 

FYI - The backup of the 99TB DB runs LAN-Free using 16 tape drives over 26 hrs.

IBM P7 Strange Behaviour

We have a P7 frame that has 4 LPARs that are used as TSM storage agents from which snapshots of our SAP DB's are mounted for backup. They have always had great performance until one LPAR had a bad HBA that phoned home and was replaced. After it was replaced performance for backups dramatically decreased from 800MB/s to 150MB/s and overall performance of the server would drastically drop. When the DB requiring backup is over 25TB that is a huge hit, and we could not find the root cause.  At first IBM said it was our Hitachi disk that was the problem. We eliminated that right away, so we then replaced the new HBA, checked our fiber, and then checked the GBIC and nothing seemed to fix the situation. During the first week I asked the IBM service technician if we could possibly have a bad drawer or slot and he emphatically said "No! If you did you would have errors all over the place." So we checked firmware, we moved cards within the frame (again), we double checked the fiber, now we were going into the third week. So I kept asking if something could be wrong with the drawer/slots and I kept getting the same answer. The reason I suggested it was due to previous experience. I have seen hardware go bad without totally going "out". So after exhausting everything other than the replacing the slots, IBM finally replaced the slots. Viola! Backup speeds went back to normal and system degradation during the backup disappeared.  So the slots/drawer was the issue. No errors relating to a slot/drawer hardware issue occurred but something caused the slots to degrade performance.  It took almost a month to resolve the issue, I wouldn't say that IBM support was very thorough and at times tried to push off the problem to other vendors (i.e. Hitachi). I can only suggest in the future you trust your instincts and push the CE's to follow down every avenue. My headache is over, but now the RCA begins.

Solaris LAN-Free

I recently had to configure a Solaris box for LAN-Free and had to dig up my old documentation. Here's what I did to get LAN-Free to work after loading the drivers and the devices not being found in the tape list file. These directions are for IBM LTO drives only.

http://www-01.ibm.com/support/docvie...S7002972&aid=1
Adobe Reader page 137 (actual doc page 117)

We need to make sure the native "st" driver is not loaded. Run

rem_drv st

to unload. And comment out everything in /kernel/drv/st.conf

then we need to run the following:

rm /dev/rmt/*	removes any tape drive definitions in the rmt folder. Do this only if the IBM tape drives are the only drives used on the server
/opt/IBMtape/tmd –s	Stops the Tape Monitor Daemon
/usr/sbin/rem_drv IBMtape	Removes the IBMtape driver
The commands to reload the device driver are:
/usr/sbin/add_drv -m ’* 0666 bin bin’ IBMtape	This reloads the driver but does not set the correct driver type
/usr/sbin/update_drv -av -i ’"scsiclass,01.vIBM.pULTRIUM-TD3"’ IBMtape	This will add the drive type to the /etc/driver_aliases file.
/opt/IBMtape/tmd	Reloads the IBM Tape Monitor Daemon

Then run/opt/IBMtape/tapelist –Ac to see if the drives are discovered correctly.

Solaris Driver Issue

I have a Solaris (version 10) server that I am trying to configure LTO drives on for a Storage Agent. My problem is that I ran the procedure to assign the drives to the IBMtape driver but doc state a reboot is needed. Has anyone done a successful modunload/modload of the IBMtape driver thereby avoiding a reboot? The server is a production box and the DB takes 45 minutes to take down and come back up so reboots are usually scheduled and the next one is weeks off. Any ideas or help is appreciated.

Who Did What?

This weekend I had an issue with a backup failing on a client using LAN-Free. When checking the TSM actlog the error I kept seeing was:

ANR9999D (Session: 265500, Origin: OD0BG-UAX001-STA) mmsshr.c(3874): ThreadId<13> Unable to obtain model type for '/dev/rmt23', rc = 46(SESSION: 265500)

So I had a good idea that the client and TSM Storage Agent were having issues communicating. I knew this because I had seen this error before when the TSM client and TSM Storage Agent were not at the same release levels. Checking the TSM client I found it at 5.4.0.0 and the Storage Agent at 5.3.4.0. Turns out an admin upgraded the client without knowledge of the Storage Agent, causing the backup failures. This is an example of what can happen when someone tries elbow their way into another groups area without the knowledge to do it right.

LAN-Free Unknown Feature

My boss brought this article to my attention so I thought I would pass it on. It discusses an unknown or little discussed capability of LAN-Free to act as a "pass-thru" server for other clients. In other words a normal client can connect to TSM through the LAN-Free agent that resides on another server. Basically the Storage Agent becomes a dummy TSM server. This can be helpful when you have backups that need to go to tape directly but the network connection between the client and the TSM server is hurt by inadequate bandwidth between switches, or firewall issues. You can check the article out here. I remember seeing this discussed on ADSM.org before, but have never attempted to try it.

Storage Agent Device Issue

So one of my co-workers was tasked with figuring out why a drive was shown in TSM as being assigned to a storage agent when the storage agent was not actually using it (we had even restarted the agent and the library controller still showed the agent as owner of the drive). As it turns out there is an issue with down level TSM device drivers being used between an AIX TSM server and Windows Storage agents. You can find the APAR here. Here is a brief rundown of the problem.

After a review with the Windows and AIX Device Driver development team, it was determined that one of the Windows STA's placed a persistent registration on a drive, failed the persistent reservation, and then failed to release the registration during the recovery path. The registration causes a problem for AIX but not Windows due to the differences in SCSI 2 and SCSI 3 used by Windows and AIX. This hasbeen fixed and the recommendation was to upgrade to the current Windows Device Driver - 6.1.9.3. If you are not at 6.1.9.3 or higher, this is most likely your causing your problem.

You can download the latest driver from here:

ftp://ftp.software.ibm.com/storage/devdrvr/Windows/

Here is the tsm apar I mentioned:

http://www-01.ibm.com/support/entdocview.wss?uid=swg1IC57753&NotUpdateReferer=

LAN-FREE

We face a lot of problems recently with LAN-FREE on MS Windows platform. The TSM server runs on AIX or HP-UX, but the client is always MS Windows. Sometimes we lose one or a few drives, sometimes all of them. The problem doesn't evolve immediately only after a while. For these problematic customers we use J1A, E05 or LTO3 drives with the newest firmwares and drivers, of course.

Does anyone else have the same CHALLENGE?

LAN-Free In A Shared Library Environment

I had to setup a LAN-Free client recently and ran into some trouble with the tape environment.  As you all know I use a shared library environment where multiple TSM servers share a single large library through a library controller TSM instance.  This works great since it allows for a shared scratch pool and TSM can be bounced faster since the DB is less than 2GB.  The problem arises when trying to setup a LAN-Free agent in this complex environment.  If you have ever read the directions for LAN-Free you’ll notice they can be quite confusing since they switch the client port to 1502 to accommodate the LAN-Free agent’s default of 1500.  Now understand 1500 makes sense because the LAN-Free agent is really just a stripped down TSM server. Since the TSM server uses 1500 by default so does the agent.  I decided to use port 1502 for the agent but forgot to change it in the dsmsta.opt file. This kept me from being able to connect to the agent from the client since it was looking at port 1502 and the agent was still using port 1500.

Another issue the directions don’t fully explain is where to point the LAN-Free agent when setting it up.  I will explain how here and hopefully make it easy to understand.

1. Setup the dsm.sys client file with the following lines

ENABLELANFREE           YES
LANFREECOMMMethod       TCPIP
LANFREETCPServeraddress <loopback, DNS name, or IP>
LANFREETCPPort          1502

2. Define the agent to the node’s TSM server and the library controller as a server

define server storagent serverpassword=passw0rd hladdress=<loopback, DNS name, or IP> lladdress=1502

Also don’t forget to register the node if not registered already.

register node <nodename> <password> domain=<domain name>

3.  On the TSM library controller instance path the drives seen by the node/LAN-Free agent machine.  I am assuming you have connected the client node to the SAN environment and zoned a number of drives to the node.  When seen in the OS you can define the path. Make sure you map the paths correctly. In AIX this can be done by running the lscfg –vp | more command and noting the drive serials and noting them to their corresponding drive in TSM.

DEFine PATH STORAGENT <Drive Name> SRCType=SERVER AUTODetect=YES DESTType=DRIVE LIBRary=<Library Name> DEVIce=/dev/rmtX ONLine=YES

Note:   For ACSLS libraries more configuration parameters need to be set. I am showing an IBM hardware example since I would never use anything else!

4. On the client node go to the storage agent’s folder and update the dsmsta.opt file with the following

     tcpport 1502

5. Now from that storage agent directory run the following command

dsmsta setstorageserver myname=storagent mypassword=passw0rd myhladdress=<loopback,DNS name, or IP> servername=<node’s tsm server IP or DNS name> lladdress=1500 <or port used by node’s tsm server>

Note:  Notice I am pointing the agent to the node’s TSM server. Even though it is a library client and does not control the drives, by defining the agent to both the client TSM server and the controller instance a handoff from the library client to the library controller will occur automatically.  If you do not define the LAN-Free agent to the library controller the agent will fail.

You should now be able to test the configuration by starting the dsmsta in the foreground and then from another telnet window start a TSM command line client and run a backup(if using Windows open a dsmc client). If setup correctly you’ll see the agent contact the controller for a drive and mount a tape and commence a LAN-Free backup.

King Of All Backups! (AKA LAN-Free to Disk)

About a year ago we were tasked to setup a large multi-clustered Exchange server and provide the best possible backup and restore performance.  After much debate and research we decided on using LAN-Free to disk.  The system was a 7 node Windows 2003 cluster connected to an SAN disk array (I can’t remember if it was EMC or Dell). The first 5 nodes were Exchange servers, the 6th was the failover node and the 7th was turned into a TSM server.  The TSM server instance had 5 500GB secondary disks assigned to it for the backup of the five Exchange servers.  These five disks would be mapped one to each Exchange server allowing for the backup to occur across the SAN to the disks owned by the TSM server. To utilize the LAN-Free to disk capability we had to install Tivoli’s SANergy product.  SANergy is no longer a separate product but is now part of a TDP/Agent type install package for TSM.  We actually installed and configured SANergy first, which was easier than it seemed in the directions, then mapped the drives.  When configured with SANergy the mapped drives become accessible across the SAN as long as the clients are on the same disk SAN fabric.  So we now had mapped SAN-accessible drives and could backup the Exchange servers to disk using the FILE device class.  The FILE device class is the device class used since TSM does not support LAN-Free backups to diskpools at this time.  The FILE device class works like a virtual tape and it was configured to migrate the data a few hours before the next backup would occur, or when the storage pool reached a specific usage threshold.  The reason for this was to allow almost a 24 hr. timeframe for a restore and along with the new Exchange 2003 restore capabilities internally; it provided a high performance backup/restore solution.  We tested the backups against a 360GB DB and backed it up in 90 minutes.  People were impressed, but they wanted to see how it performed on restore. We then restored the same amount, 360GB, in 91 minutes. WOW!  It was amazing to see those numbers (68MB/s).  We even tested it with the failover node by mapping all 5 SANergy defined drives to the failover node and still saw the same numbers. We had everything ready to go when the account decided they wanted to go in another direction.  Weeks spent configuring and implementing the solution all for not!  At least I have the experience and know it works.  So if anyone is looking to do LAN-Free to disk it works, it’s fast, it takes a lot of admin work, and it will be a good solution for anyone looking for a high performance backup/restore environment.