|| Rate this content|
HP StorageWorks Tape Storage - Performance Troubleshooting
Information in this document is taken from the HP Support document titled HP Surestore and StorageWorks - Performance Troubleshooting and Using Performance Assessment Tools.
Tape drives are getting faster and it is not always easy to make the most of their performance in complex systems. Debugging performance problems can be tricky at the best of times. This document aims to provide the knowledge to help identify where performance related problems are in a tape backup system and gives guidance on how they can be fixed.
Note that this document applies equally to backup systems whatever type of tape drive is used. However, the examples used refer to HP's Ultrium and SDLT tape drives, as these are the fastest and most challenging to configure correctly.
Performance will always be limited by one or more bottlenecks in the system, of which the tape drive is just one part. The goal is to make the tape drive the bottleneck. That way the system will achieve the performance figures as advertised on the drive's spec sheet. This puts the emphasis on the rest of the system. The flow of data throughout the system must be fast enough to provide the tape drive with data at its desired rates.
High-speed tape drives such as the HP Ultrium 460, 920, 960, and 1840 are so fast this can be a difficult goal to achieve. If backup performance is not matching the data sheet of the tape drive, then there is a bottleneck somewhere else in the system.
How fast should a tape drive go?
The answer to this is determined by multiplying the native data rate of the tape drive by the compression ratio of the data being backed up. For example, 2:1 compressible data on an HP Ultrium 460 (30 MB/sec native) should achieve a data rate of 60 MB/sec.
Note that the internal pipeline of the tape drive might limit the ultimate data rate. In the case of the HP Ultrium 460, this is around 90 MB/sec. This is unlikely to be a limiter except in the fastest of systems.
Compression ratio can vary a great deal.
More information regarding compression is available in the HP Support document titled HP Surestore/StorageWorks- Data Compression Information. Click here to view "HP Surestore/StorageWorks- Data Compression Information" (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50244) .
For simplicity, data is assumed to compress at 2:1, which can be used as a working default. A better alternative is to use the actual compression ratio of data, which can be found by using Library and Tape Tools (LTT) to calculate it directly from the compression logs of the tape drive. With the expected data rate in mind, the next step is to measure the other parts of the system to ensure they can transfer data at least as fast as that.
Tape drive performance
This is a good place to start, as the tape drive is often incorrectly considered to be the bottleneck. This can be established with (or otherwise) certainty by checking how fast the server can access the tape drive independent of the disk subsystem and application using Devperf in newer versions of L&TT. It works by transferring pre-loaded data directly from the memory of the server to the tape drive. It can also use different compression ratios.
If the expected data rates cannot be achieved, then any of the following factors may be apparent:
Disk and file system performance
There is a significant difference between the raw data rate of a disk and the rate at which files can be read from a file system. This is because traversing a file system requires multiple, random disk accesses; whereas, a continuous read is limited only by the optimized data rates of the disk. The difference between these two modes becomes more significant as file size reduces. For file systems where the files are typically smaller than 64 KB, sequential (or disk image) backups should be considered to achieve the data rates supported by high speed tape drives.
To measure the data rate from the disk subsystem, use L&TT Sysperf, and select the method used by the backup - file-by-file or sequential. This tool gives the actual read transfer rate that the disk system can provide, which can be compared with the expected data rate. If the disk subsystem does not transfer data fast enough, then it is a bottleneck. This could be down to any of the following factors:
As changes are made to the disk subsystem, re-run Sysperf to measure its performance.
For more detailed information about how the file system and disk configuration affect backup performance of the SDLT or Ultrium, see the HP Support document titled HP Surestore and StorageWorks - How File System and Disk Configuration Affect Backup Performance with a DLT or Ultrium Drive. Click here to view "HP Surestore and StorageWorks - How File System and Disk Configuration Affect Backup Performance with a DLT or Ultrium Drive" (http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=lpg50032) .
Backup application configuration
At this point it has been shown that neither the disk subsystem or tape drive are the performance limiters in the system. The next, most likely contender is the backup application itself.
These are HP's recommendations for achieving high performance with backup applications:
For information about HP's recommendations regarding compatibility, see the HP Backup Compatibility web page. Click here to go to the "HP Backup Compatibility" web page (http://www.hp.com/products1/storage/compatibility/index.html) .
Server capacity and capability
The server is central to the backup process as it runs the backup software and the data is passed into and out of the server's main memory as it is read from disk subsystem and written to tape.
To see if the server is a possible bottleneck, use one of the readily available system monitoring tools, such as PerfMon, while performing a backup. Ensure there are still resources free. Watch for the following:
Also, check for the following:
One thing to watch for is the allocation of IRQs for the HBAs for both the tape drive and the disk subsystem. If either of them share an IRQ (interrupt) with something else, then there will be additional overhead in the management of interrupts. This can reduce performance significantly. The best way to find out how the IRQs are allocated is to do so in the BIOS. This is also the best place to change them if it is desired to do it manually. Do not use the Windows device manager as this does not always give accurate IRQ allocation.
There are a number of possible workarounds:
One other thing to watch for on Windows operating systems, is the Removable Storage Manager (RSM). This system service polls the drive regularly, which interrupts the flow of data and therefore reduces the overall throughput. The workaround is to disable the service, but beware - re-connecting any storage devices will automatically turn RSM back on again, including devices via USB.
Make sure to disable the RSM through the Registry hack in the device drivers (for example LTO.SYS, in the Robot drivers for MSL G3 libraries).
HBA settings are set as per the recommendations in the EBS Design Guide. For more information about EBS, see the HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features web page. Click here to go to the "HP StorageWorks Enterprise Backup Solutions (EBS)- Overview & features" web page (http://h18000.www1.hp.com/products/storageworks/ebs/) .
How important is performance?
The answer to this question depends on what is needed. If after following the advice above, the transfer rate of the tape drive still cannot be reached - and this really can be a challenge with the high transfer rates supported by HP's Ultrium and SDLT tape drives - then it is not necessarily a problem unless that performance is needed. The key requirement may just be that the backup completes within the available time window. As upgrades are done on the server, disk system, network or other parts of the whole system, better use of the performance of the tape drive will be possible. If the maximum performance is not being achieved at the moment, at least feel good about investing in a backup solution that will match the system well into the future.
The following table shows the native data transfer rates of a range of HP's currently supported tape drives. HP's Ultrium drives also have a feature called Adaptive Tape Speed (ATS), which matches the speed of the tape to the data rate of the server. It is fully variable between its lowest and highest rates and eliminates tape repositions within this range - thus reducing wear even if the server cannot keep up.
NOTE: The data compression engines of the VS80, VS160 and DLT80 products do not have a linear relationship between compression ratio and transfer rate. It is not recommended to use HPTapeperf with these products for performance measurements.
If a data rate can be achieve above the minimum ATS speed, then the drive will stream. For example, it will be operating at minimum wear rates. As a last resort, turn data compression off. This would be more likely to stream, although it would use more tape for the same amount of data.