Poor Performance Followup

As a follow-up to the previous poor performance post I thought I'd post what the outcome was. As it turns out we checked performance tuning settings in TSM and AIX and no performance increase was seen. We asked the DB2 admins to review any of their settings and they could not find any tunables that had not already been implemented. We sent in servermon.pl output and although they saw performance that was sub-par, they couldn't designate what was causing it. There are no server/adapter/switch/disk/tape errors so nothing emerged as the culprit for our poor throughput performance.

So we reviewed the backup time of each TSM storage agent server used to backup this 101 TB SAP database. At the time the storage agents that perform the backup consisted of 5 LPARS, 4 of those in a single frame each with their own assigned I/O drawer. The 5th was in a separate 740 frame with its own I/O drawer. The 5th storage agent was completing the backup in a fraction of the time of the other 4 so we concluded we must be overloading the CEC on the 740. We moved one of the four storage agents out of the frame to a secondary frame and the results were awesome. See below:

You'll notice that the backup time didn't change with the update of the tape drives from E06 to E07. Hardware layout matters more than the performance of the tape drives. When a vendor tells you just updating the hardware to newer iterations will increase performance take it will a grain of salt. In our case we did testing of the new tape drives and no performance gains were seen but the go ahead was given to upgrade to the newer hardware and as you'll see we didn't gain anything until we reworked the environment. Our task now is to identify how to increase TSM internal job performance (i.e. migration and storage pool backup) which has not seen significant performance gains from the tape upgrades.

Sony Develops 185TB Tape

Sony announced they have developed a tape medium and write process that can support 185TB per tape. Whoa, That's huge! Now if we can see it hit the market before some other storage strategy catchphrase becomes the "it" thing.  Check out the link below...."To the cloud!"

http://www.extremetech.com/computing/181560-sony-develops-tech-for-185tb-tapes-3700-times-more-storage-than-a-blu-ray-disc

Poor Performance

Currently I work in an environment where we have a specific TSM instance for a large SAP DB (99TB currently). We just upgraded the drives in the tape library (yes we use tape! I know...I know....) from MagStar 3592 TS1130 (E06) drives to TS1140 (E07) drives. The upgrade was pushed in hopes of a jump in write/backup performance, but I was skeptical. TSM adds so much overhead you cannot use the RAW tape read/write numbers from any manufacturer. Typically IBM is somewhat reasonable with their numbers, but in this case I have seen NO performance increase what-so-ever.  Here is a query of the processes for storage pool backup.

UPDATE (04/04/2014):  Let me give you some more specs, we have the 99TB DB split between 4 TSM Storage Agents each having 4 8Gb HBA's. Each storage agent runs 4 sessions (allocates 4 drives) for their backup process. So all 4 storage agents account for 16 simultaneous sessions and it still takes over 24 hours to perform the 99TB backup. The backups are averaging around 70-78MB/sec. Is this a TSM overhead issue or do I have a tuning issue with the TDP and TSM? I'm getting less than 50% of the throughput I should see.

Here's the command that is run to execute the DB backup:

ksh -c export DB2NODE=7 ; db2 "backup db DB8   LOAD /usr/tivoli/tsm/tdp_r3/db264/libtdpdb264.a OPEN 4 SESSIONS OPTIONS /db2/DB8/dbs/tsm_config/vendor.env.7 WITH 14 BUFFERS BUFFER 1024 PARALLELISM 8 WITHOUT PROMPTING" ; echo BACKUP_RC=$?

PROCESS_NUM: 2667
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:54
   DURATION: 00 23:20:13
      BYTES: 6.0TB
 AVG_THRPUT: 75.87 MB/s

PROCESS_NUM: 2668
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 78.48 MB/s

PROCESS_NUM: 2669
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.2TB
 AVG_THRPUT: 77.99 MB/s

PROCESS_NUM: 2670
    PROCESS: Backup Storage Pool
 START_TIME: 03-27 23:21:55
   DURATION: 00 23:20:12
      BYTES: 6.4TB
 AVG_THRPUT: 80.13 MB/s

I average anywhere from 75 to 80 MB/sec.  Here is the Magstar performance chart. I am using JB media, not JC so I do take a little hit in performance for that.

So with JB media I could get as high as 200MB/sec but I am not even 50% of that number.  Is there any specific tuning parameter I should look at that could be hindering the performance? 

FYI - The backup of the 99TB DB runs LAN-Free using 16 tape drives over 26 hrs.

IBM P7 Strange Behaviour

We have a P7 frame that has 4 LPARs that are used as TSM storage agents from which snapshots of our SAP DB's are mounted for backup. They have always had great performance until one LPAR had a bad HBA that phoned home and was replaced. After it was replaced performance for backups dramatically decreased from 800MB/s to 150MB/s and overall performance of the server would drastically drop. When the DB requiring backup is over 25TB that is a huge hit, and we could not find the root cause.  At first IBM said it was our Hitachi disk that was the problem. We eliminated that right away, so we then replaced the new HBA, checked our fiber, and then checked the GBIC and nothing seemed to fix the situation. During the first week I asked the IBM service technician if we could possibly have a bad drawer or slot and he emphatically said "No! If you did you would have errors all over the place." So we checked firmware, we moved cards within the frame (again), we double checked the fiber, now we were going into the third week. So I kept asking if something could be wrong with the drawer/slots and I kept getting the same answer. The reason I suggested it was due to previous experience. I have seen hardware go bad without totally going "out". So after exhausting everything other than the replacing the slots, IBM finally replaced the slots. Viola! Backup speeds went back to normal and system degradation during the backup disappeared.  So the slots/drawer was the issue. No errors relating to a slot/drawer hardware issue occurred but something caused the slots to degrade performance.  It took almost a month to resolve the issue, I wouldn't say that IBM support was very thorough and at times tried to push off the problem to other vendors (i.e. Hitachi). I can only suggest in the future you trust your instincts and push the CE's to follow down every avenue. My headache is over, but now the RCA begins.

Data Domain vs. Protectier

Where I am currently employed we are looking to replace our 3592 based library with a deduplication solution. Currently the higher ups are leaning towards IBM ProtecTIER without having thoroughly investigated any other solutions. Having previously used Data Domain solutions at my previous employ I was somewhat concerned that the ProtecTIER solution would be a bad fit for our environment. I have had some run ins with people who have used IBM's ProtecTIER solution and when compared to those who have used Data Domain (including myself) you immediately see the difference in how they talk about the two products. So I was hoping to find a good write-up showing in-depth details comparing the two solutions and it took a blogger like me to provide a great comparison.  If you would like a good overview of how Data Domain and ProtecTIER stack up against one another in technology and performance check out the following link. It's very informative and solidifies why I would prefer using Data Domain.

Deduplication: Data Domain Vs. ProtecTIER Performance

One item that was not covered was the NFS capabilities of both. While I used VTL functionality with Data Domain, I was a HUGE NFS proponent. You can save a lot of money over a TSM TDP + LAN-Free solution using NFS with 10Gb Ethernet for your DB backups  (since IBM's licensing costs are still questionable). When I was first exploring ProtecTIER they did not yet have NFS capabilities, so I'd like to see a NFS performance comparison between the two products.

NFS Mount Issue

I recently had a number of AIX server backups miss due to the backup hanging when doing its initial filesystem listing. At some point within the last day the mount's source was rebooted and all the mount points went bad. The problem is that TSM sits there trying to query it even though the default action is to not backup NFS mount points. So I had to log into each server, umount -f the file system, remount it, and then TSM was able to run successfully. TSM does not allow a DOMAIN -ALL-NFS so no matter what I do TSM is going to hang on the listing of file systems. Of course doing a df on the server hangs also, so it's not just a TSM issue. Anyone else ran across this issue?

TSM Backup Issue

Anyone had an issue where their backups were extremely slow and their Interrupts were huge? I've got 400GB DB's taking 40hrs to backup over a 4 port Ether-channel connection. No errors in my AIX errpt and the network guys are telling me they don't think it's them. Any suggestions on what to look at are appreciated.  Below is an example when I run entstat.

ETHERNET STATISTICS (en8) :

Device Type: IEEE 802.3ad Link Aggregation

Hardware Address: 00:14:5e:e7:26:41

Elapsed Time: 9 days 19 hours 20 minutes 35 seconds

Transmit Statistics:                          Receive Statistics:

--------------------                          -------------------

Packets: 5470416553                           Packets: 24510516113

Bytes: 440661650021                           Bytes: 32245892708954

Interrupts: 0                                 Interrupts: 6027433898

Transmit Errors: 0                            Receive Errors: 691

Packets Dropped: 0                            Packets Dropped: 0

                                              Bad Packets: 0

Max Packets on S/W Transmit Queue: 298

S/W Transmit Queue Overflow: 0

Current S/W+H/W Transmit Queue Length: 355

Broadcast Packets: 8786                       Broadcast Packets: -1346793420

Multicast Packets: 225928                     Multicast Packets: 136913

No Carrier Sense: 0                           CRC Errors: 0

DMA Underrun: 0                               DMA Overrun: 691

Lost CTS Errors: 0                            Alignment Errors: 0

Max Collision Errors: 0                       No Resource Errors: 0

Late Collision Errors: 0                      Receive Collision Errors: 0

Deferred: 141004                              Packet Too Short Errors: 0

SQE Test: 0                                   Packet Too Long Errors: 0

Timeout Errors: 0                             Packets Discarded by Adapter: 0

Single Collision Count: 0                     Receiver Start Count: 0

Multiple Collision Count: 0

Current HW Transmit Queue Length: 355

General Statistics:

-------------------

No mbuf Errors: 0

Adapter Reset Count: 0

Adapter Data Rate: 1701737521

Driver Flags: Up Broadcast Running

        Simplex 64BitSupport ChecksumOffload

        PrivateSegment LargeSend DataRateSet

TSM Client Scheduler Issue

I just recently had an issue with a handful of TSM clients that would not run their backups. The clients all backup to a TSM 5.5.2 server and were all running Windows 2008. The clients use TSM version 6.2.3. The five clients had all been missing their backups for days and what makes the situation more interesting is that there are other Windows 2008 servers with this version of TSM installed and they are all running their schedules without issue.

When reviewing the TSM Schedule log the scheduler listed that it had received the schedule info and was waiting for the TSM server to initiate the schedule. The TSM server never made an attempt to contact the clients in question and never showed any errors other than ANR2578W stating the client missed its schedule. There were no errors in the error log and not much to go by from the TSM server activity log. Even though the TSM client backs up over the public network I switched to polling mode to see if client based initiation of the backup would work. It didn't! The TSM client scheduler would receive the schedule upon polling the TSM server but would never execute it. So now what? I added the TCPCLIENTADDRESS and TCPCLIENTPORT and switched back to SCHEDMODE PROMPTED, still the scheduler would not run backups.

Now I was getting frustrated. I removed the scheduler service and redefined it using dsmcutil and voila, the schedule ran...ONCE! After the initial schedule ran the previous problem returned. Schedules were not running and the TSM server would not show any errors saying it could not contact the client. It just would not run the schedule. Well that left me no choice but to call support. IBM support's response was to make sure the TCPCLIENTADDRESS and TCPCLIENTPORT were defined in the dsm.opt and also to define the client HLADDRESS and LLADDRESS on the TSM server? Define the HL and LL addess? TSM gets that when the client connects doesn't it? Yes and No! It appears that without the optional setting the TSM server can have issues contacting some clients. Why? No idea, but adding the HL and LL address did the trick and the backups have been running without issue since.

How many of you define the HL and LLADDRESS when registering nodes? I've never suspected it was needed until now.

TSM to DPM transition

I am currently working a contract transitioning a companies old TSM environment to Microsoft DPM and am discovering once again how Microsoft operates. Just like many of their software products when they were first released, Microsoft's product lacks certain features that others have had for years. As they revise the product they slowly reach a close parity, but it's never quite equal to older more mature products. The clincher is that it's a Microsoft tool and Windows shops don't know any better. As I learn more about DPM I'm convinced it's not going to supplant TSM or Netbackup by features, but shear lack of backup expertise from the administrators. Their MS rep will come in and tout the product with the limited information he has, show them a decent bundle price, sell them DPM, and not be heard of until the next licensing cycle. (Does The Music Man ring a bell?) The reporting capabilities alone are awful! You can't easily create your own and DPM runs on MS SQL. Go figure! And what's with Microsoft pushing their PowerShell?? You can script reports but have to know how to script within MS PowerShell. I thought the whole purpose of Windows was to eliminate the command line? (Not that I mind, I just find it ironic) So as I learn more I'll post what I find, but so far I'm shocked they thought DPM was a better tool than TSM...what they really needed was a good TSM admin not new software.

Data Loss Top 10

Ontrack, which handles data recovery, compiled a list of the strangest reasons data was lost. You can read their list here. I remember when I was teaching TSM some students had some of the craziest stories for data loss (many were disgruntled employee stories).

One story I heard from a student occurred when a manager was stupid enough to fire an employee, but require him to work the rest of the week if he wanted his pay. So the employee worked the rest of the week and on his way out (unescorted) he went into the server room and proceeded to urinate into the mainframe. Supposedly he shorted out the mainframe and caused all kind of havoc. If you have a story you'd like to share feel free to leave it in the comments section.

Personal Backups

A friend of mine lost a bunch of family pictures and video when his hard drives died (he lost both at the same time). As backup people we are all well aware that the current home computer user does not have an easy and safe way to backup all their data easily. Oh, sure they can buy a USB drive but those can also be destroyed in a fire, flood, or even stolen. There are some online options but wanted to ask those out there what their experiences are with them and any suggestions. I currently have 200+GB I need protected so what are the safest options...and no I wont be buying a tape drive. ;-)

Question About Active Data Pools

Geoffrey Huntley recently asked me what situation would an Active Data pool be useful? To be honest, I couldn't think of a good situation. Since I don't use an Active Data Pool I thought I would throw the question out to our fellow readers to get some feedback as to how you might be using ADPs. My biggest reason for not using them was the PIT restore issue. You can read my full gripe here. Basically if TSM wont utilize it when doing a PIT then what's the benefit? I'd say more than half, if not 75%, my restores are from older dates.

LAN-Free Unknown Feature

My boss brought this article to my attention so I thought I would pass it on. It discusses an unknown or little discussed capability of LAN-Free to act as a "pass-thru" server for other clients. In other words a normal client can connect to TSM through the LAN-Free agent that resides on another server. Basically the Storage Agent becomes a dummy TSM server. This can be helpful when you have backups that need to go to tape directly but the network connection between the client and the TSM server is hurt by inadequate bandwidth between switches, or firewall issues. You can check the article out here. I remember seeing this discussed on ADSM.org before, but have never attempted to try it.

Texas Government Pulls Plug on IBM

Well if you get a chance check out this article on ZDnet. It looks like IBM is losing the Texas government account due to backup failures and lost data. Having previously worked for IBM what I would like to know is what is not being said. When IBM takes over an account on an outsourcing deal they have to assume the horrible practices already in place then do their best to convert to a better software/hardware solution and processes. For example when I was with IBM we had a remote site that was using Arcserve for their backups and had two weeks worth of tapes. A couple months into the contract they lost a server and needed to restore the data and guess what...they couldn't. The process in place was for them to rotate two weeks worth of tapes continually. The tapes were four to five years old and had never been tested. They found out after the fact that the tapes had gone bad some time in the past and they were pretty much out of luck. Is this IBM's fault? They took some of the blame even though IBM was only a couple months into the contract and the smaller remote sites were secondary in the process change timeline.

The Monring News did report the following:

In a Nov. 3 letter to the governor’s office, IBM acknowledges the company overreached by assuming responsibility for existing technological conditions that are inadequate, inconsistent and not sustainable.

VSS issues on Windows

October last year I blogged about VSS issues on Windows. Here is another link which may help troubleshooting;

http://www-1.ibm.com/support/docview.wss?uid=swg21243837

Cheers

Bare Metal Restore

All the best for 2008
Bare Meal Restore - Put simply

For W2K the TSM dsm.opt DOMAIN ALL-LOCAL is the default, this will include
SYSTEMOBJECT backup for incrementals.

Backup Commands:
dsmc incr

TSM Windows 2000 restore steps:
1. Repartition, format harddrive
2. Install W2K, set computername to original
3. Install appropriate W2K Service Pack
4. Install tsm ba client
5. dsmc restore c:\ -replace=all -subdir=yes
6. dsmc restore systemobject
7. Reboot

Cheers
Ryan Partington

Tip of the day

setopt maxsessions - server side option which specifies total number of sessions, client and server, that can run at any one time

set schedmaxsessions - this number is a PERCENTAGE of 'maxsessions' and specifies what percent of the maxsessions can be used for server/client schedules, like the maintenance script or daily client backups

If # of clients trying to backup over night exceed the percent of maxsessions available, you'll get an error in dsmsched.log which looks something like this. "Could not backup, no sessions available"

Journal

The TSM journal can be your friend or foe. Which one depends on when and how you use it. IBM have released an FAQ around the journal and below I point out a couple of important areas. It will take you no longer than 60 seconds to improve/refresh you're understanding of the journal.

60 second intro

• Journal Backup is appropriate for backing up files systems with small or moderate amounts of change activity between backup cycles.
• A traditional incremental backup of the file system must be completed while the
  journal for the file system is active.
• This backup must result in the Last Backup Completion Date being updated on
  the TSM server.
• Once the full incremental backup has been completed, the change journal is marked
  as valid for the TSM client node and server the backup was performed with.

more...

Monthly Backups

Many people I have talked with have a requirement to keep a month end backup for an extended period of time. This is where backupsets can be very helpful, but what do you do when backupsets just wont cut it? Well I have a requirement to keep some month end data for extended periods but all other backups are kept for 30 days. How do I accomplish this? The answer is a shell script. Before the ehanced scheduler came about in TSM 5.3 the only way to get the backup to change to a different type at the end/beginning of the month was to have some script handle it or you could manually control it. So here is a good shell script that looks at the day of the month and if it is the 1st it runs a backup to an alternate nodename. In this example the nodename is the same as the default nodename with -MONTHLY added to the end. We also made a dsm.opt file called dsm.opt.monthly that references the stanza in the dsm.sys with the alternate name. Then schedule this script as a daily command script for the node and it will run the daily incremental to the default nodename every day except on the 1st of the month. I hope this is useful for some of you. Click Read More to see the script.

#!/usr/bin/ksh
#set -x
export PATH=$PATH:.:/etc:/usr/sbin:/bin:/usr/local/lib:
export LIBPATH=$LIBPATH:/usr/odcs/bin
export DSM_LOG=/opt/tivoli/tsm/client/ba/bin
export DSM_DIR=/opt/tivoli/tsm/client/ba/bin
BK_SCRIPT_LOGFILE=/usr/local/scripts/logs/Filesystem-incremental-backup.log
if [ -f $BK_SCRIPT_LOGFILE ]; then
mv $BK_SCRIPT_LOGFILE $BK_SCRIPT_LOGFILE.`date +%d`
printf "`date` - Starting Backup\n" >> $BK_SCRIPT_LOGFILE
fi
printf " test " >> $BK_SCRIPT_LOGFILE
# Modified to use the monthly opt file on the 1st for monthly backups that are
# kept for a extended period of time.


case `date +%d` in
01)
export DSM_CONFIG=/opt/tivoli/tsm/client/ba/bin/dsm.opt.monthly
;;
*)
export DSM_CONFIG=/opt/tivoli/tsm/client/ba/bin/dsm.opt
;;
esac
# Back up the filesystems
#
printf "`date` - Starting the backup of the filesystems.\n" >> $BK_SCRIPT_LOGFILE
# Set the success variable to zero
INCR_BACKUP_SUCCESS=0


/opt/tivoli/tsm/client/ba/bin/dsmc incr >> $BK_SCRIPT_LOGFILE
# /opt/tivoli/tsm/client/ba/bin/dsmc q fi >> $BK_SCRIPT_LOGFILE INCRCODE=$?
if [[ $INCRCODE -ne 0 && $INCRCODE -ne 4 && $INCRCODE -ne 8 ]]
then
printf "`date` - Incremental backup failed - return code: $INCRCODE\n" >> $BK_SCRIPT_LOGFILE
exit 1111
else
printf "`date` - Incremental of $FS successful - return code: $INCRCODE\n" >> $BK_SCRIPT_LOGFILE
fi
printf "`date` - Backup has completed successfully\n" >> $BK_SCRIPT_LOGFILE
echo $?

FilesX Xpress Restore & TSM

If you click the title of this post you can read the news article that explains this new tool that has been validated to work with TSM. It provides a block level disk backup of application data allowing TSM to work with the Xpress Restore repository enhancing the speed of backups...This is one tool I would like more information on so I will be looking into how it works with TSM.