Poor Performance Followup

As a follow-up to the previous poor performance post I thought I'd post what the outcome was. As it turns out we checked performance tuning settings in TSM and AIX and no performance increase was seen. We asked the DB2 admins to review any of their settings and they could not find any tunables that had not already been implemented. We sent in servermon.pl output and although they saw performance that was sub-par, they couldn't designate what was causing it. There are no server/adapter/switch/disk/tape errors so nothing emerged as the culprit for our poor throughput performance.

So we reviewed the backup time of each TSM storage agent server used to backup this 101 TB SAP database. At the time the storage agents that perform the backup consisted of 5 LPARS, 4 of those in a single frame each with their own assigned I/O drawer. The 5th was in a separate 740 frame with its own I/O drawer. The 5th storage agent was completing the backup in a fraction of the time of the other 4 so we concluded we must be overloading the CEC on the 740. We moved one of the four storage agents out of the frame to a secondary frame and the results were awesome. See below:

You'll notice that the backup time didn't change with the update of the tape drives from E06 to E07. Hardware layout matters more than the performance of the tape drives. When a vendor tells you just updating the hardware to newer iterations will increase performance take it will a grain of salt. In our case we did testing of the new tape drives and no performance gains were seen but the go ahead was given to upgrade to the newer hardware and as you'll see we didn't gain anything until we reworked the environment. Our task now is to identify how to increase TSM internal job performance (i.e. migration and storage pool backup) which has not seen significant performance gains from the tape upgrades.

Data Domain vs. Protectier

Where I am currently employed we are looking to replace our 3592 based library with a deduplication solution. Currently the higher ups are leaning towards IBM ProtecTIER without having thoroughly investigated any other solutions. Having previously used Data Domain solutions at my previous employ I was somewhat concerned that the ProtecTIER solution would be a bad fit for our environment. I have had some run ins with people who have used IBM's ProtecTIER solution and when compared to those who have used Data Domain (including myself) you immediately see the difference in how they talk about the two products. So I was hoping to find a good write-up showing in-depth details comparing the two solutions and it took a blogger like me to provide a great comparison.  If you would like a good overview of how Data Domain and ProtecTIER stack up against one another in technology and performance check out the following link. It's very informative and solidifies why I would prefer using Data Domain.

Deduplication: Data Domain Vs. ProtecTIER Performance

One item that was not covered was the NFS capabilities of both. While I used VTL functionality with Data Domain, I was a HUGE NFS proponent. You can save a lot of money over a TSM TDP + LAN-Free solution using NFS with 10Gb Ethernet for your DB backups  (since IBM's licensing costs are still questionable). When I was first exploring ProtecTIER they did not yet have NFS capabilities, so I'd like to see a NFS performance comparison between the two products.

DBMEMPERCENT...Where'd That Come From?

I was having performance issues with a couple TSM 6.2 servers and could not find anything that pointed to the issue. I'm not one to call support unless I'm totally stumped and cannot find help through the web, but this time I finally relented and made the call. The issue was problems with backups failing repeatedly and when researched we were getting internal server errors along with DB table errors. IBM support asked for some DB2 log files and within 30 or so minutes had identified the problem.

TSM has a server option I have never used or heard of that had somehow been set that adversely affected all backups. Somehow the option DBMEMPERCENT was set in the dsmserv.opt file. This option tells TSM what percentage of the overall server's memory it can allocate for use. The default is AUTO and would have been fine, but somehow DBMEMPERCENT was set to 10 in the dsmserv.opt. Which means out of 16GB of RAM I was only using 1.6GB?!?!? How'd that happen? I didn't set it, none of my coworkers remember setting it, so where did it come from? IBM support stated the default was AUTO so the option was manually set. Since I had never used this option and its 6.x specific, I never would have looked for it. Good thing I called support.

TSM Backup Issue

Anyone had an issue where their backups were extremely slow and their Interrupts were huge? I've got 400GB DB's taking 40hrs to backup over a 4 port Ether-channel connection. No errors in my AIX errpt and the network guys are telling me they don't think it's them. Any suggestions on what to look at are appreciated.  Below is an example when I run entstat.

ETHERNET STATISTICS (en8) :

Device Type: IEEE 802.3ad Link Aggregation

Hardware Address: 00:14:5e:e7:26:41

Elapsed Time: 9 days 19 hours 20 minutes 35 seconds

Transmit Statistics:                          Receive Statistics:

--------------------                          -------------------

Packets: 5470416553                           Packets: 24510516113

Bytes: 440661650021                           Bytes: 32245892708954

Interrupts: 0                                 Interrupts: 6027433898

Transmit Errors: 0                            Receive Errors: 691

Packets Dropped: 0                            Packets Dropped: 0

                                              Bad Packets: 0

Max Packets on S/W Transmit Queue: 298

S/W Transmit Queue Overflow: 0

Current S/W+H/W Transmit Queue Length: 355

Broadcast Packets: 8786                       Broadcast Packets: -1346793420

Multicast Packets: 225928                     Multicast Packets: 136913

No Carrier Sense: 0                           CRC Errors: 0

DMA Underrun: 0                               DMA Overrun: 691

Lost CTS Errors: 0                            Alignment Errors: 0

Max Collision Errors: 0                       No Resource Errors: 0

Late Collision Errors: 0                      Receive Collision Errors: 0

Deferred: 141004                              Packet Too Short Errors: 0

SQE Test: 0                                   Packet Too Long Errors: 0

Timeout Errors: 0                             Packets Discarded by Adapter: 0

Single Collision Count: 0                     Receiver Start Count: 0

Multiple Collision Count: 0

Current HW Transmit Queue Length: 355

General Statistics:

-------------------

No mbuf Errors: 0

Adapter Reset Count: 0

Adapter Data Rate: 1701737521

Driver Flags: Up Broadcast Running

        Simplex 64BitSupport ChecksumOffload

        PrivateSegment LargeSend DataRateSet

AIX Image Backup Performance Issue

I am running 3 simultaneous image backup jobs on an AIX server and the HBA only shows 20% utilized and my throughput sucks. I am using -imagetype=static and the FS's in questions are EMC clones split off and remounted to a new server. When I have done image backups before they were wicked fast...now not so much. Any ideas what I might have missed? I have an ACSLS STK 8500 with LTO-3 drives. Even over gig-ether I've seen better performance. We did check the CPU and memory usage and they are fine. Lots of I/O wait, not sure why I would see that with statc image backups.

60 seconds of tips

Good morning comrades

After running a database restore you need to run an audit on your storage pools so they are consistent. Failure to do so may results in lost space and errors in your log.
Read more

How do you know from day to day the throughput of your client backups? Here we take a glimps at a method which can help you review performance with next to no effort.
Read more

Enjoy the weekend