Sony Develops 185TB Tape

Sony announced they have developed a tape medium and write process that can support 185TB per tape. Whoa, That's huge! Now if we can see it hit the market before some other storage strategy catchphrase becomes the "it" thing.  Check out the link below...."To the cloud!"

http://www.extremetech.com/computing/181560-sony-develops-tech-for-185tb-tapes-3700-times-more-storage-than-a-blu-ray-disc

Solaris Driver Issue

I have a Solaris (version 10) server that I am trying to configure LTO drives on for a Storage Agent. My problem is that I ran the procedure to assign the drives to the IBMtape driver but doc state a reboot is needed. Has anyone done a successful modunload/modload of the IBMtape driver thereby avoiding a reboot? The server is a production box and the DB takes 45 minutes to take down and come back up so reboots are usually scheduled and the next one is weeks off. Any ideas or help is appreciated.

Drive Matching

While working with a VTL I realized how much I hated matching drive definitions on a LAN-Free client or remote library client and decided to do something about it. So after searching the web and finding a helpful script to pull rmt and serial numbers from the tape drives discovered on the LAN-Free or Lib-Client I worked it into an AIX shell script that will match the drives rmt with the correct drive definition on the library manager. The key to this working is to have the library manager listed in the lib-clients dsm.sys and to define the following TSM script on the library manager.


-=-=-=-=-=-=-TSM Server Script-=-=-=-=-=-=-=-


def script drive_match desc="This script is used by a shell script to match drives to rmts to help create DEFINE PATH statements"
upd script drive_match "select 'define path $1 ' || drive_name || ' srct=server destt=drive libr=' || library_name || ' device=/dev/$2 -"
upd script drive_match " online=yes' from drives where DRIVE_SERIAL='$3' "


-=-=-=-=-=-=-UNIX Shell Script-=-=-=-=-=-=-=-


#!/bin/ksh
ID=`cat /home/tsmadmin/VARS/ADSMID`
PA=`cat /home/tsmadmin/VARS/ADSMPA`
touch ./drv_match.mac
cat /dev/null > ./drv_match.mac
clear
echo "This script will build the DEFINE PATH statements for a server"
echo ""
echo "Enter the Library Manager Name: \c"
read SERV
echo ""
echo "What is the TSM server name these paths are going to be generated for: \c"
read TSMSRV
echo ""
lsdev |grep fscsi | while read FSCS rest
 do
   TAPES=$(lsdev -p $FSCS | grep rmt | grep -v ALT |cut -f1 -d" ")
     for TAPE in $TAPES
       do
         SERIAL=$(lscfg -vl $TAPE | grep Serial | awk -F"." '{printf "%d", $NF}')
         FCS=$(echo $FSCS | sed 's/fscsi/fcs/')
         dsmadmc -id=$ID -pa=$PA -dataonly=yes -commadelimited -servern=$SERV run drive_match $TSMSRV $TAPE $SERIAL | grep -v ANR >> ./drv_match.mac
        done
 done
cat ./drv_match.mac


-=-=-=-=-End Script-=-=-=-=-=-

You'll notice I save the output as a macro so it can be reviewed before executed. If you'd like to run the macro automatically then just add a dsmadmc line at the end.

The following is some sample output:


define path tsm8 VTL3_00332 srct=server destt=drive libr=VTL3 device=/dev/rmt88 online=yes
define path tsm8 VTL3_00336 srct=server destt=drive libr=VTL3 device=/dev/rmt89 online=yes
define path tsm8 VTL3_00340 srct=server destt=drive libr=VTL3 device=/dev/rmt90 online=yes
define path tsm8 VTL4_00444 srct=server destt=drive libr=VTL4 device=/dev/rmt91 online=yes

Storage Agent Device Issue

So one of my co-workers was tasked with figuring out why a drive was shown in TSM as being assigned to a storage agent when the storage agent was not actually using it (we had even restarted the agent and the library controller still showed the agent as owner of the drive). As it turns out there is an issue with down level TSM device drivers being used between an AIX TSM server and Windows Storage agents. You can find the APAR here. Here is a brief rundown of the problem.

After a review with the Windows and AIX Device Driver development team, it was determined that one of the Windows STA's placed a persistent registration on a drive, failed the persistent reservation, and then failed to release the registration during the recovery path. The registration causes a problem for AIX but not Windows due to the differences in SCSI 2 and SCSI 3 used by Windows and AIX. This hasbeen fixed and the recommendation was to upgrade to the current Windows Device Driver - 6.1.9.3. If you are not at 6.1.9.3 or higher, this is most likely your causing your problem.

You can download the latest driver from here:

ftp://ftp.software.ibm.com/storage/devdrvr/Windows/

Here is the tsm apar I mentioned:

http://www-01.ibm.com/support/entdocview.wss?uid=swg1IC57753&NotUpdateReferer=

Tivoli Storage Manager 5.2 and Capacity on Demand gotcha.

A customer recently commissioned a new frame for their 3584 library and installed some new tape drives. TSM was left running under the assumption that paths and drives could be taken offline whilst work was performed on the library to ensure availability to backup to the disk storage pools.

Unfortunately this is not the case when you are commissioning additional storage slots (frames or capacity on demand).

The first indication that something was astray was that after the paths and drives online was that the the inability to check in any more tapes as the library was still showing that it was full.

The new frame allowed for over 400 more slots so this also didn’t make sense. After some thought I remembered the ’show slots’ command. After running the command I noticed that I was only being shown 704 slots rather then ther full number of slots I knew that were available.

The diagnostics and administration tool "tapeutil" showed the new and updated element range which confirmed that the configuration from an operating point of view was correct.

So the problem obviously was to do with TSM:

tsm:> show slots 3584atl

PVR slot information for library L32ATL.
Library : L32ATL
Product Id : 03584L32
Support module : 4
Mount
count : 5
Drives : 16
Slots : 704
Changers : 2
Import/Exports :
10
Device : /dev/smc0
Drive 0, element 257
Drive 1, element 258
Drive 2, element 259
Drive 3, element 260
Drive 4, element 261
Drive 5, element 262
Drive 6, element 263
Drive 7, element 264
Drive 8, element 265
Drive 9, element 266
Drive 10, element 267
Drive 11, element 268
Drive 12, element 269
Drive 13, element 270
Drive 14, element 271
Drive 15, element 272

Changer 0, element 1
Changer 1, element 2

ImpExp 0, element number 769
ImpExp 1,
element number 770
ImpExp 2, element number 771
ImpExp 3, element number
772
ImpExp 4, element number 773
ImpExp 5, element number 774
ImpExp
6, element number 775
ImpExp 7, element number 776
ImpExp 8, element
number 777
ImpExp 9, element number 778

slot element range 1025 -
1728

After some research, I discovered that the element count is only queried once upon TSM startup and it is not possible to update the this information without restarting TSM.

tsm:> show slots 3584atl

PVR slot information for library L32ATL.
Library : L32ATL
Product Id : 03584L32
Support module : 4
Mount
count : 8
Drives : 20
Slots : 1130
Changers : 2
Import/Exports :
10
Device : /dev/smc0
Drive 0, element 257
Drive 1, element 258
Drive 2, element 259
Drive 3, element 260
Drive 4, element 261
Drive 5, element 262
Drive 6, element 263
Drive 7, element 264
Drive 8, element 265
Drive 9, element 266
Drive 10, element 267
Drive 11, element 268
Drive 12, element 269
Drive 13, element 270
Drive 14, element 271
Drive 15, element 272
Drive 16, element 273
Drive 17, element 274
Drive 18, element 275
Drive 19, element 276

Changer 0, element 1
Changer 1, element 2

ImpExp 0, element
number 769
ImpExp 1, element number 770
ImpExp 2, element number 771
ImpExp 3, element number 772
ImpExp 4, element number 773
ImpExp 5,
element number 774
ImpExp 6, element number 775
ImpExp 7, element number
776
ImpExp 8, element number 777
ImpExp 9, element number 778

slot element range 1025 - 2154

Have You Checked Your Drive Firmware Lately?

Well the weekend is over and I thought I would update you on the problem SAP system.  In the last update we had discussed how this one TSM server was consistently having its mounts go into a RESERVED and hanging. The only thing we could do was cycle the TSM server. So after talking to support they stated it was a known problem listed in APAR IC49066 and fixed in the TSM server 5.3.3.2 release. So we upgraded the TSM server, updated ATAPE on the problem server to the same level as on the controller, and turned on SANDISCOVERY. The system came back up and started to mount tapes, albeit slowly. So we let it run and all seemed well for about the first 8 hours then all hell broke loose. TSM started taking longer and longer to mount tapes until it couldn’t mount anything. We began receiving the following errors also:

7/13/2006 9:38:45 AM ANR8779E Unable to open drive /dev/rmt25, error number=16.
7/13/2006 9:38:45 AM ANR8311E An I/O error occurred while accessing drive DR25 (/dev/rmt25) for SETMODE operation, errno = 9.

We reopened the problem ticket and talked to a number of Tivoli support reps and almost had to force them to have us run a trace. The original problem fix began Friday and here we were still trying to fix it well into early Sunday morning. So after getting the trace to Tivoli they looked it over and seemed perplexed at the errors. The errors seemed as if Tivoli was trying to mount tapes for drives that the library client was not pathed for. Also it seemed TSM was polling the library for drives but was unable to get a response so it went into a polling loop until it found a drive. This caused our mount queue to get as high as 100+ tape mounts waiting and the mounts that did complete sometimes took 30-45 minutes to do so. It was NUTS! So Sunday morning a new Tivoli Rep was assigned (John Wang). As we discussed the problem and Tivoli was trying to get a developer to analyze the trace John mentioned how he had seen this type of error before in a call about an LTO-1 library. He stated that it took 3 weeks to determine the problem but that the end result was that the firmware on the drives was down-level and causing the mount issues. So I checked my 3584’s web interface (I feel bad for all those people out there without web interfaces on their libraries) and found the drive firmware level at 57F7. This seemed down-level from what little information we had so I had my oncall person call 1-800-IBM-SERV and place a SEV-1 service call. The CE called me and we discussed the firmware level. When he saw how down-level it was he was surprised and lectured me on making sure we always check with CE’s before we do any changes to the environment. The CE then gathered his needed software and came to the account to update the drives. I brought all the TSM servers down and after 30 minutes the library had dismounted all tapes from the drives. The CE then proceeded to update the firmware, which actually only took 15-20 minutes. I expected longer. So we went from firmware level 57F7 to 64D0. Huge jump! So after the firmware upgrade I audited the library and brought the controller back up. Viola! It started mounting tapes as soon as the library initialized and the response was back to what it should have been. It’s now Tuesday morning and there have been no problems. So before you upgrade TSM be sure to have checked your libraries firmware (both library and drives). It could mean the difference between sink and swim!

Help! Problem With Shared Library Client!

Ok, here is my problem.  I have a large shared library that supports 5 TSM servers. All of the TSM servers, save one, run without any problems. The one with problems is pathed to 32 of the 5 drives (they are designated just for its use) and periodically its mount requests go into a RESERVED state but never mount.  I have tried FORCESYNC'ing both sides to see if it was communication but that didn't work.  I have tried setting paths offline then back online and that didn't work either. The only thing that works is to cycle the TSM server then it comes back up and starts mounting tapes like nothing was wrong.  This happens on average every 24-48 hours.  This is a large DB client TSM instance so the reboots are hurting critical business apps backups. I don't see any glaring errors in the activity log or in the AIX errpt. Any suggestions would be gladly received.

LAN-Free In A Shared Library Environment

I had to setup a LAN-Free client recently and ran into some trouble with the tape environment.  As you all know I use a shared library environment where multiple TSM servers share a single large library through a library controller TSM instance.  This works great since it allows for a shared scratch pool and TSM can be bounced faster since the DB is less than 2GB.  The problem arises when trying to setup a LAN-Free agent in this complex environment.  If you have ever read the directions for LAN-Free you’ll notice they can be quite confusing since they switch the client port to 1502 to accommodate the LAN-Free agent’s default of 1500.  Now understand 1500 makes sense because the LAN-Free agent is really just a stripped down TSM server. Since the TSM server uses 1500 by default so does the agent.  I decided to use port 1502 for the agent but forgot to change it in the dsmsta.opt file. This kept me from being able to connect to the agent from the client since it was looking at port 1502 and the agent was still using port 1500.

Another issue the directions don’t fully explain is where to point the LAN-Free agent when setting it up.  I will explain how here and hopefully make it easy to understand.

1. Setup the dsm.sys client file with the following lines

ENABLELANFREE           YES
LANFREECOMMMethod       TCPIP
LANFREETCPServeraddress <loopback, DNS name, or IP>
LANFREETCPPort          1502

2. Define the agent to the node’s TSM server and the library controller as a server

define server storagent serverpassword=passw0rd hladdress=<loopback, DNS name, or IP> lladdress=1502

Also don’t forget to register the node if not registered already.

register node <nodename> <password> domain=<domain name>

3.  On the TSM library controller instance path the drives seen by the node/LAN-Free agent machine.  I am assuming you have connected the client node to the SAN environment and zoned a number of drives to the node.  When seen in the OS you can define the path. Make sure you map the paths correctly. In AIX this can be done by running the lscfg –vp | more command and noting the drive serials and noting them to their corresponding drive in TSM.

DEFine PATH STORAGENT <Drive Name> SRCType=SERVER AUTODetect=YES DESTType=DRIVE LIBRary=<Library Name> DEVIce=/dev/rmtX ONLine=YES

Note:   For ACSLS libraries more configuration parameters need to be set. I am showing an IBM hardware example since I would never use anything else!

4. On the client node go to the storage agent’s folder and update the dsmsta.opt file with the following

     tcpport 1502

5. Now from that storage agent directory run the following command

dsmsta setstorageserver myname=storagent mypassword=passw0rd myhladdress=<loopback,DNS name, or IP> servername=<node’s tsm server IP or DNS name> lladdress=1500 <or port used by node’s tsm server>

Note:  Notice I am pointing the agent to the node’s TSM server. Even though it is a library client and does not control the drives, by defining the agent to both the client TSM server and the controller instance a handoff from the library client to the library controller will occur automatically.  If you do not define the LAN-Free agent to the library controller the agent will fail.

You should now be able to test the configuration by starting the dsmsta in the foreground and then from another telnet window start a TSM command line client and run a backup(if using Windows open a dsmc client). If setup correctly you’ll see the agent contact the controller for a drive and mount a tape and commence a LAN-Free backup.

TSM Library Sharing

TSM is truly an Enterprise Backup tool and one of the key selling points is how well it allows you to consolidate and centralize the backup environment. One feature I have used extensively is the library sharing option. This feature allows multiple TSM servers to use the same library without conflict. It can be easier than partitioning the library, allows for dynamic allocation of tape drives, and allows for a single shared scratch pool. The key to using a shared library is creating a separate TSM instance as a dedicated library manager (whether on the same server as another TSM instance or on its own hardware). The library manager instance will not perform any backups and will not be a big instance. From my experience it will have very little impact on the physical server. When creating our library manager instance I was unsure how large to make the DB so I made it 2GB, and it currently is 4.6% utilized. The library it manages is a 9 frame 42 drive 3584. The library supports 3 TSM backup instances (2 on one server and 1 on a separate physical server) with a total of 951 clients. We put the library manager instance on the server with 2 instances, thereby creating a third TSM instance. By creating the third instance dedicated to managing the library you will have increased uptime and when you need to do maintenance or restart the instance the library manager is back up very quickly. This will not be the case if you make a production backup TSM instance the library manager due to a production TSM servers need to run expiration and usually having a larger log size making restart time take that much longer.

The other nice feature of library sharing is the ability to dynamically allocate drives to specific TSM servers. I have seen some administrators take twenty drives and path ten to one TSM instance and ten to another but this defeats the purpose. If you setup the library manager and TSM backup instances correctly then you’ll allow every instance connected to the library manager to see all the drives. This coupled with the device class mount limit setting can allow you to change drive usage on the fly. So say you have two TSM servers TSM-A and TSM-B using a twenty drive 3584 managed by a library manager instance called TSM-C. Each TSM instance sees all twenty drives but you have their device class mount limits set to ten. Now each TSM instance can only use ten drives out of the twenty. Lets say on a specific day TSM-A server is in need of more drives and TSM-B is currently only using three drives. You could update the TSM device class on each to allow TSM-A fifteen drives and TSM-B five drives.

Recently we purchased some newer hardware to replace our 9 frame LTO-1 library and are looking at using two small AIX servers using HACMP as the library controller since the new library will need to be available at all times and will be supporting at least six TSM servers. Unfortunately I have had to partition this new library due to application demands. The problem we are experiencing with this new library is that the logical portioning requires us to use the Virtual I/O option in the 3584. This option allows the library to automatically insert volumes when placed in the I/O door. This would not be a problem if the library did not then require us to assign the volumes to a specific logical library. This is done through the library and not through TSM, which adds another process to our tape management. I would have preferred to not have partitioned the library and allowed a TSM library manager instance to handle allocation of tape drives, but alas I am not able to at this time (still awaiting the HACMP instances).

FYI: Using Mixed Media In An LTO Library

We recently upgraded one of our 3584's at work with some LTO2 drives. This is the first attempt at a mixed media environment and according to IBM and the following Redbook technote here, you must work out the issues with MOUNTLIMITS or all LTO2 drives could end up in use when you need them since they can also read LTO1 media. So we set our mountlimits accordingly, but it does not help that we have more LTO1 drives than LTO2. So by setting the mount limit to the number of LTO1 drives didn't stop TSM from accessing the LTO2 drives. This they did not explain well and they left one crucial piece out of the puzzle. When using a mixed media library if you do not specificly state which media format to use TSM will use both LTO1 and LTO2 media in a LTO designated storage pool. You want me to explain further? Ok here is how it affected us. We added LTO2 drives and an additional 2 frames to our library and setup the devclasses accordingly, setting the FORMAT to DRIVES so it would use the highest format available by the drive assigned. Well since we didn't partition the library TSM is going to grab whatever drive comes available and will mount the appropriate scratch tape. So if TSM assigns an LTO1 drive then you'll use an LTO1 tape. The only way you can force the LTO2 media to be used is to set the FORMAT setting in the devclass to ULTRIUM2 or ULTRIUM2C (w/Compression). So we didn't think about that and were bit by it when we ran out of LTO1 scratch. We didn't catch it due to our script only looking for scratch in the library not being able to designate between the two media types (which you can really only do if a different vol series is used for labeling). So without the LTO1 scratch we basically lost 2/3 of the drives and didn't know it. So I had to go switch our script to monitor both scratch types and we had to force the LTO's to their appropriate format. Once I realized what was happening it was a "NO BRAINER" that TSM would work that way. The big problem is that TSM did not seperate out the media format for LTO2 so it would be a different devclass type like they did with the 3592's. So be aware how TSM works and make sure you don't make the same mistake I did.