TSM 6.1 & 6.2 DB2 Issue

I had a TSM server crash mutliple times over the course a week and after working with Tivoli support and sending them the core files, it was determined that the following error was the cause. Interesting, in that I never thought about the connections from TSM to the DB2 DB. So to summarize, the current connection from TSM to DB2 is not a TCP based but IPC and AIX has a limitation of 1024 IPC connections to DB2 otherwise the application in question (TSM in this case) can crash. The following link has directions on how to convert TSM to DB2 connections to TCP to eliminate this issue.

TSM Server 5.5, 6.1, & 6.2 Bug

I figured I best share with you all a bug I have experienced with TSM Server version 6.2.2 that has resulted in the TSM instance core dumping. The problem stems from TSM being unable to mount a scratch tape with, in our case, a TSM storage agent. When the mount fails it seems TSM lists an internal server error then core dumps a few seconds later.  The fix is in 6.2.3 and higher. It also affects 5.5 and 6.1 versions, check this technote for further details.

Warning: This update has a bug also concerning Data Domain that causes errors if the defined VTL has more than 4400 slots. So when we upgraded the server to fix the LAN-Free bug we inadvertently encountered this bug.

Update: Here is the APAR listing for the Data Domain VTL slot bug in TSM 6.2.x

Update: Weird Volhist Records (Part Two)

IBM did get back to me about my library manager/library client volume history devclass issue and I found it interesting how development handled it. Initially IBM support said the following:

Our server development has confirmed with our design, it is possible that the remote volhist entry can be different. Development believe this is their design. However, the existing document did not document this clearly. They agreed to open a Doc. APAR to properly document this for command query volhist.

In other words they felt that no action was needed other than to document that this possible occurrence within the TSM Admin and Reference guides. So I asked it be escalated since it is definitely a flaw and finally heard back from support and was told the following:

They agree what you observed or reported in this case is incorrect even though it does not cause any function lose. They have agreed to take the APAR IC65048 as a defect ( instead of Document ). However, this APAR would take a big code change to "fix" this issue and after first evaluate, they will not be able to deliver a fix in the service stream. They request to open a DCR ( design change record ) so development can make this change on a release boundary so that there is sufficient testing for a code change.

So it looks like IBM will eventually fix this. Thanks to IBM support for helping me get development to at least go beyond the Doc. APAR.

We found this bug

We found this bug in Hungary. Click for details!

Storage Agent Device Issue

So one of my co-workers was tasked with figuring out why a drive was shown in TSM as being assigned to a storage agent when the storage agent was not actually using it (we had even restarted the agent and the library controller still showed the agent as owner of the drive). As it turns out there is an issue with down level TSM device drivers being used between an AIX TSM server and Windows Storage agents. You can find the APAR here. Here is a brief rundown of the problem.

After a review with the Windows and AIX Device Driver development team, it was determined that one of the Windows STA's placed a persistent registration on a drive, failed the persistent reservation, and then failed to release the registration during the recovery path. The registration causes a problem for AIX but not Windows due to the differences in SCSI 2 and SCSI 3 used by Windows and AIX. This hasbeen fixed and the recommendation was to upgrade to the current Windows Device Driver - 6.1.9.3. If you are not at 6.1.9.3 or higher, this is most likely your causing your problem.

You can download the latest driver from here:

ftp://ftp.software.ibm.com/storage/devdrvr/Windows/

Here is the tsm apar I mentioned:

http://www-01.ibm.com/support/entdocview.wss?uid=swg1IC57753&NotUpdateReferer=

Windows 5.5. TSM Client Crashing

I ran into a problem today with a Windows 2003 32-bit client scheduler crashing with the ANS1009W DRM FilesNotToBackup RC 13 error. It is caused by malformed values in the Windows registry key that lists the default files to exclude. I had never seen this error before but it was happening with the base 5.5 client. When I had the Win Admin apply the 5.5.0.4 patch the problem was resolved. All attempts at trying to fix the registry entries manually were unsuccessful, so I would suggest applying the patch if you experience this problem.

Tivoli Storage Manager v5.5...

REPOST: I think this article from Flex is a good one to keep on the front page since TSM 5.5 migrations are only going to continue. Check the comments for solutions to this problem.

Today afternoon I had a little time and tried the brand new v5.5 client on MS Windows platform (x32) and got this error message in the dsmerror.log:
"ANS1009W An error occurred processing the operating system include/exclude statements.
The error was detected while processing: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup\DRM. RC = 13."
I didn't want to believe what I saw. This funny message still exists.
Here is the solution from IBM: ANS1009W DRM FilesNotToBackup RC 13
/* Edit the registry manually... this is the solution in the 21st century... */

Do you have this problem? Here in Hungary, all Windows servers are affected...

I like the TSM concepts itself but I don't even want to think about what ISC 5.5 will be like... ;-)

I really miss the 64bit HSM and Journal on MS Windows (we got only Online image and Open file), where is the IBM 3494 support on HP-UX Itanium (I don't understand why this is a problem for a developer? Someone could really explain it...), on the fly db reorg function (ESTIMATE DBREORGSTAT - What a solution? ), ... but the restartable server-to-server export/import sounds good.

Cheers!

TSM Express Flaw!

I don't know how many of you readers use TSMExpress but there is a flaw that has been identified by IBM. ComputerWorld has a description of the flaw here. So if you do have TSMExpress in use make sure you look into this.

Tivoli's October "Valuable Support Info" E-mail

Are you subscribing to IBM/Tivoli's Valuable Support Information E-mail? If not take the time to subscribe because the links, notices, and information are worth the few minutes it takes. For those of you not subscribing yet here it is in its entirety.

Welcome to the IBM Tivoli Storage Manager (TSM) technical support information update. This communication is designed to help you derive maximum value from your TSM software by providing the most up-to-date technical information, answers to frequently asked questions, and links to other key information. Please take a moment to read through the materials provided below. We are sure that you will find answers to many of your questions. This month's mailing has five main sections:

1. Frequently Encountered Situations and FAQs
2. News & Technical Flashes
3. Recent & Important Downloads
4. Upcoming Events & Live Technical Trainings
5. Problem-Solving Resources on our web site

You are receiving this notification because you are one of the IBM Tivoli Storage Manager customers, who have called for technical support in the past year.

To Unsubscribe: Please reply to this email and change the Subject field to 'unsubscribe.'
To Subscribe: Please reply to this email and change the Subject field to 'subscribe.'
=========================================================================

Frequently Encountered Situations and FAQs

Title: TSM Journal Based Backup FAQ
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21155524

Title: Library fails with ANR8840E and ANR8418E
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21218357

Title: Replacing a damaged primary storage pool volume
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21219227

Title: Windows 2003 Volume Shadow Copy Service (VSS) Hotfixes for Systemstate Backup
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21242128

Title: IBM Tivoli Storage Manager Administration Center Requirements
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21195062

Title: Overview - Tivoli Storage Manager Supported Operating Systems
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21243309

Title: Upgrade Instructions - Tivoli Storage Manager (TSM) 5.2 to 5.4 or 5.2 to 5.3 Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21251665

Title: MustGather: Read First for Tivoli Storage Manager Products
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21263547

Title: Guideline For Selecting The Appropriate IBM Tape Device Driver for Windows 2000
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21268024

Title: IBM Ultrium Generation 4 (LTO-4) drive and drive encryption support
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg27009625

News & Technical Flashes

Title: URGENT Actions Required: Changes to Daylight Saving Time will affect IBM Tivoli Storage Manager Administrative Interfaces
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21254129

Title: Backupsets should not be used to store Data Protection client or API client data
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21268040

Title: Data loss may occur on TSM 5.4 servers when performing off-site reclamation of a copy storage pool and active-data pools are defined
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21268228

Title: Two security vulnerabilities exist in the IBM Tivoli Storage Manager (TSM) client
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21268775

Title: Featured Documents for Tivoli Storage Manager
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg27009872Recent & Important Downloads

Title: Tivoli APAR notification page
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21177831

Title: Recommended TSM Client and Server Fixes
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21247179

Upcoming Events & Live Technical Trainings

Tivoli's Support Technical Exchange web seminars allows you to participate in live discussions on topics such as deployment, trouble-shooting tips, common issues, problem solving resources and other Support and Services recommendations.

To see the connection information for these seminars, as well as ALL Tivoli products just use the main link below.

Link: http://www.ibm.com/software/sysmgmt/products/support/supp_tech_exch.html

Mark your calendars: Here are the current listings for the FREE seminars though December, 2007

10/18/07 TSM Backup Considerations with Nseries
10/23/07 Overview, setup, and usage of NDMP operations
10/23/07 Understanding TSM HSM for Windows

11/06/07 LVSA / Open File Support
11/15/07 Tivoli Storage Manager Update
11/15/07 Ask the Experts call in session: TSM devices for Windows
11/20/07 Installing the Integrated Solutions Console (ISC) and Administration Center
11/29/07 Tuning Disk for use with Tivoli Storage Manager (TSM)

12/06/07 Exploiting Disk Technology with Tivoli Storage Manager

Help us choose future topics for our calls!
If you have ideas for a future discussion topics for TSM or to report problems with viewing content, please send feedback to: isstte@us.ibm.com

Problem-Solving Resources Online

Visit the IBM Tivoli Storage Manager product support page
Link: http://www.ibm.com/software/sysmgmt/products/support/IBMTivoliStorageManager.html IBM Education Assistant for TSM

Short, task-based audio and visual presentations on pertinent topics.
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21236711

Free IBM Support Assistant product plug-in for TSM A desktop tool to aid in your TSM administration.
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg24009958

Self help support info for new Tivoli Storage Manager Administrators
Link: http://www-1.ibm.com/support/docview.wss?rs=663&uid=swg21212718

Complimentary TSM Concepts Poster & Complimentary Online Self-Paced Training Classes
Link: http://www.ibm.com/support/docview.wss?rs=663&uid=swg21217663

Participate in the ADSM.org and ADSM/L forums:
Link: http://www.adsm.org

TSM 5.1 Flash!

This flash was posted by IBM on Sept. 25, 2007:

Customers running IBM Tivoli Storage Manager (TSM) 5.1.6 Windows, HP, Sun, AIX and Linux and using devices other than 3570, 3590, or IBM LTO may have problems with the automatic labeling of tapes by the TSM server.

You can click on the title of this post to see the IBM page.

Vulnerabilities Found In TSM Client!!!

(APARs IC52905 and IC53616)

Wow I'm glad I added the news section to the blog I saw this article about a possible exploit with the TSM CAD on client systems. I'd try to explain it but it's best to check the article. It can leave systems open for execution of arbitrary code. Check it out here and here. This exploit goes back as far as the 5.1 client so be aware of the issue even with older clients.

Affected Products:

IBM Tivoli Storage Manager Client version 5.1
IBM Tivoli Storage Manager Client version 5.2
IBM Tivoli Storage Manager Client version 5.3
IBM Tivoli Storage Manager Client version 5.4

Solution:

Apply client update package 5.4.1.2 (UK27738 and UK27739) :
http://www.ibm.com/support/docview.wss?uid=swg24016585

Apply client update package 5.3.5.3 (UK29248 and UK29249) :
http://www.ibm.com/support/docview.wss?uid=swg24016838

Apply client update packages 5.2.5.2 and 5.1.8.1 :
http://www.ibm.com/support/docview.wss?uid=swg24016985
http://www.ibm.com/support/docview.wss?uid=swg24016586

Original Advisory:
http://www-1.ibm.com/support/docview.wss?uid=swg21268775

TSM AIX Performance Issue With ML05

I found this APAR interesting and thought I would pass it along. There seems to be a bug in the TSM client for AIX when updated to ML05 and Direct I/O is in use on the filesystems. I can't explain the details well enough so read the post here. This bug can cause significant increases in backup time. They give a good example in the APAR description.

Have You Checked Your Drive Firmware Lately?

Well the weekend is over and I thought I would update you on the problem SAP system.  In the last update we had discussed how this one TSM server was consistently having its mounts go into a RESERVED and hanging. The only thing we could do was cycle the TSM server. So after talking to support they stated it was a known problem listed in APAR IC49066 and fixed in the TSM server 5.3.3.2 release. So we upgraded the TSM server, updated ATAPE on the problem server to the same level as on the controller, and turned on SANDISCOVERY. The system came back up and started to mount tapes, albeit slowly. So we let it run and all seemed well for about the first 8 hours then all hell broke loose. TSM started taking longer and longer to mount tapes until it couldn’t mount anything. We began receiving the following errors also:

7/13/2006 9:38:45 AM ANR8779E Unable to open drive /dev/rmt25, error number=16.
7/13/2006 9:38:45 AM ANR8311E An I/O error occurred while accessing drive DR25 (/dev/rmt25) for SETMODE operation, errno = 9.

We reopened the problem ticket and talked to a number of Tivoli support reps and almost had to force them to have us run a trace. The original problem fix began Friday and here we were still trying to fix it well into early Sunday morning. So after getting the trace to Tivoli they looked it over and seemed perplexed at the errors. The errors seemed as if Tivoli was trying to mount tapes for drives that the library client was not pathed for. Also it seemed TSM was polling the library for drives but was unable to get a response so it went into a polling loop until it found a drive. This caused our mount queue to get as high as 100+ tape mounts waiting and the mounts that did complete sometimes took 30-45 minutes to do so. It was NUTS! So Sunday morning a new Tivoli Rep was assigned (John Wang). As we discussed the problem and Tivoli was trying to get a developer to analyze the trace John mentioned how he had seen this type of error before in a call about an LTO-1 library. He stated that it took 3 weeks to determine the problem but that the end result was that the firmware on the drives was down-level and causing the mount issues. So I checked my 3584’s web interface (I feel bad for all those people out there without web interfaces on their libraries) and found the drive firmware level at 57F7. This seemed down-level from what little information we had so I had my oncall person call 1-800-IBM-SERV and place a SEV-1 service call. The CE called me and we discussed the firmware level. When he saw how down-level it was he was surprised and lectured me on making sure we always check with CE’s before we do any changes to the environment. The CE then gathered his needed software and came to the account to update the drives. I brought all the TSM servers down and after 30 minutes the library had dismounted all tapes from the drives. The CE then proceeded to update the firmware, which actually only took 15-20 minutes. I expected longer. So we went from firmware level 57F7 to 64D0. Huge jump! So after the firmware upgrade I audited the library and brought the controller back up. Viola! It started mounting tapes as soon as the library initialized and the response was back to what it should have been. It’s now Tuesday morning and there have been no problems. So before you upgrade TSM be sure to have checked your libraries firmware (both library and drives). It could mean the difference between sink and swim!

Shared Library Problem Solved?

Well after trying numerous things to get the problematic shared library working I finally got support on the line to fix this problem. The other day the problem reoccurred after about a week. I thought I had fixed the problem by changing the IP/VLAN the servers communicated over, and I thought it was until it all came crashing down. So I upgraded the problem instance to the same release level as the controller (controller has to be at or higher then the client servers). That didn’t fix it! I turned RESETDRIVES to NO and that didn’t do it either (Thanks for the advice Scott). So we got Tivoli on the line and Andy Ruhl with Tivoli support worked with us to identify the problem. First thing we did was check the ATAPE level and it turned out the controller was at a higher version. So we updated it to the same level as the controller. Then Andy found this APAR IC49066 which interestingly enough describes my problem. Turns out there was a known issue with this type of behavior and although it was due to be fixed in 5.3.4 it was added to the 5.3.3.2 update. Supposedly the explanation states it occurs when the client is accessing two different controllers, but Andy stated it was a misprint and it applies to single controller instances as well. So we updated all 5 TSM servers to the fixed level and hope we don’t experience any more issues. So far so good!

TSM Client Issue!

I have been informed by Tivoli support personnel that a critical issue has been identified in the TSM backup/archive client that could affect data integrity. The issue involves the RESOURCEUTILIZATION option and how it works when backing up to tape. If you have RESCOURCEUTIL set higher than the default and the client backs up directly to tape there is an identified issue of data not being backed up, or archived, and incorrect deletion of files that were not archived without an error message being issued.

If any of the following conditions are met you are NOT affected by the problem:

• The RESOURCEUTILIZATION client option is either not explicitly set, or is set to 1 or 2. This option can be set either in the client option set or schedule on the server, or in the local client options.


• Data for the client node is only stored in random-access disk storage pools (such as the preconfigured storage pool, BACKUPPOOL).


• Image backup is used.


• NDMP backup (initiated by client or server) is used.


• Only the API client is used (including programs and products that use only the API client - i.e. TDP's).

A fix is scheduled for release at the end of April, in the meantime make sure you either set any client using a higher RESOURCEUTIL value to 2, or send the client's backup to disk. Also note that any backup to disk using the FILE devclass is impacted by this problem as it affects all sequential media backup types.

I have provided a link to the APAR listing at the Tivoli website.