Home

Solid State Drive Caching in the IBM XIV Storage

1. If you already have an XIV system but it is not yet equipped with SSDs you can use the stem statistics function to analyze a workload To use this function click Monitor Statistics In In the area at bottom of the Monitor window that opens Figure 4 6 you can select filters to separate the data I 2 Read Hit MemHit 7 64 512 KB 2 IOPS Write a Miss C SSD Hit 08 KB gt 512 KB Latency Rew 2 H M a 8 64 KB JA BW Figure 4 6 Filter panel for the statistics monitor These filters are divided by the type of transaction reads or writes cache properties hits versus misses cache memory hits SSD cache hits or the transfer size of l O as detected by the XIV system As shown in Figure 4 6 the filter panel now also supports SSD Caching To determine if a workload might benefit from SSD Caching filter the I Os as follows 1 For l Os select Read 2 For Cache select Miss 3 For the request size select I Os that represents small requests in the range 0 64 KB To make multiple selections hold down the Ctrl key 4 Select IOPS The resulting I Os are displayed in the filtered view and are all candidates for SSD acceleration To help you visualize the results Figure 4 7 on page 25 and Figure 4 8 on page 25 represent two views of a workload Solid State Drive Caching in the IBM XIV Storage System Figure 4 7 represents the total IOPS for read and write requ
2. gt IBM XIV Storage website http www ibm com systems storage disk xiv index html gt System Storage Interoperability Center SSIC http www ibm com systems support storage config ssic index jsp O Copyright IBM Corp 2012 All rights reserved 55 Help from IBM IBM Support and downloads ibm com support IBM Global Services ibm com services 56 Solid State Drive Caching in the IBM XIV Storage System Index A alerts 51 53 algorithms error correction code 5 error detection code 5 over provisioning 5 selective read cache 14 wear leveling 5 16 architectures traditional 11 B bad block mapping algorithm 5 bandwidth 11 block tracing facility 19 25 26 BI client 27 BI server 26 buffer 16 C cache learning 15 17 miss 24 node 14 15 prediction 27 cache management extended 16 caching 11 versus SSD tiering 10 COMPONENT_TEST_OF_SSD_HAS_FAILED 53 consistency group 46 Core Enterprise Resource Planning 23 D data loss 6 DB2 brokerage 22 disabled SSD Caching 41 48 53 Disk Magic 19 29 disk rebuild 18 DRAM cache 1 E enabled SSD Caching 41 48 53 erase block 6 erased 4 error correction code algorithm 5 error detection code algorithm 5 events SSD 53 extended cache 14 management 16 F flash cell 4 Copyright IBM Corp 2012 All rights reserved full redundancy 33 G garbage collection 16 green status 39 GUI support 37 H hash table 16 heat map 27 help s
3. Example 6 5 State of a volume XIV 1310039 Coruscant vol list x vol Res Fra Vol 02 XCLIRETURN STATUS SUCCESS COMMAND LINE vol_list x vol Res Fra Vol _02 gt OUTPUT volume id bec1240012f gt creator value itso gt creator category value storageadmin id value bec1240012f gt name value Res Fra Vol 02 size value 17 gt ssd caching value disabled gt lt use_ssd_caching default value yes gt lt volume gt lt OUTPUT gt lt XCLIRETURN gt The value of use_ssd_caching default parameter indicates whether the volume follows the default system state for SSD Caching gt If the value is yes the volume follows the default system state for SSD Caching gt If the value is no the volume does not inherit the default setting for SSD Caching If the global system setting for caching is changed the volume keeps its current ssd_caching value Chapter 6 GUI and XCLI support for solid state drives 49 50 Solid State Drive Caching in the IBM XIV Storage System Monitoring and alerting This chapter highlights the additional information that the IBM XIV Storage System can provide about installed solid state drives SSDs It includes information about performance and basic operational status It also provides information about alerts that were added to the XIV system software This chapter includes the following sections gt Monitoring SSDs gt Alerts Copyright IBM Co
4. O Copyright IBM Corp 2012 All rights reserved 9 3 1 SSD tiering versus caching When SSD was first adopted by storage system vendors they mostly used it as another faster storage tier XIV Gen3 departs from that approach by using SSDs exclusively as an extension of the system read cache Tiering Tiering is traditionally how SSDs are integrated in storage systems With the tiering approach Figure 3 1 additional storage software intelligence is necessary to unite independent tiers and manage the movement of data between the tiers This approach can often add cost and complexity Important SSD is not used as a tier in XIV Tiering is driven by policies that determine when to move data to a disk type If your environment is fairly static this approach can work well You must also consider the time it takes to relocate data in and out of a tier If data cannot be relocated quickly the tiered storage approach is not effective SSD as a Tier Multiple independent disk tiers provide different levels of service Data can be relocated between tiers Manually Automatically policy driven A memory cache serves all tiers XIV does NOT implement the Tier approach Figure 3 1 SSD used as a storage tier 10 Solid State Drive Caching in the IBM XIV Storage System Caching By using the SSD as an extension of the storage system cache the workload benefits from SSD technology in real time With SSD Caching illustrated
5. avgIOps 1000 pu j e 100 Read Write Seq Aligned 100 0 0 100 y Sm Md Lg VLg 100 0 0 0 read write ratio Aspects m O R MEE E e e RRE siitti is tes REHE H EE E A E E R 93GB H EHE REEF HE EHE EHE 1 255 H E PE EER FET A E EEE E R E s R PER po abba e I Eiin i uni iis e in Tail Bo BOR OBOROR OBORBBOR ORBOB ROB ORA RA A IEEEIIZIEIIII IIEZEEZEZEIIIIIZSEZNIISINIIIIIIIILIN I LEESI EII Figure 4 10 XIV host volume heat map The bottom part of Figure 4 10 shows the heat map The blue dots show the frequency access pattern for the same volume over 6 hours These hot spot areas pertain to a host volume and not an XIV physical disk The fundamental XIV concept of uniformly spreading data across all physical disks remains unchanged Other storage systems target hot spots on disks for relocation to SSDs However Chapter 4 Workload characteristics 27 the XIV system does not relocate any blocks at the physical disk layer but rather can predict which areas of the logical volume to cache on the SSD Cache prediction By using the same trace data IBM Research developed algorithms to predict how a workload can reduce read miss events when read cache is extended to 6 TB The program does the prediction without needing an XIV system equipped with SSD You can use the replay utility to validate the prediction and further validate the model Replay utility IBM service representatives have a progra
6. LTO the LTO Logo and the Ultrium logo are trademarks of HP IBM Corp and Quantum in the U S and other countries Java and all Java based trademarks and logos are trademarks or registered trademarks of Oracle and or its affiliates UNIX is a registered trademark of The Open Group in the United States and other countries Other company product or service names may be trademarks or service marks of others vi Solid State Drive Caching in the IBM XIV Storage System Preface This IBM Redpaper publication provides information about the implementation and use of solid state drives SSDs in IBM XIV Storage System XIV Generation 3 Gen3 running XIV software version 11 1 0 or later In the XIV system SSDs are used to increase the read cache capacity of the existing DRAM memory cache and are not used for persistent storage This paper begins with a high level overview of the SSD implementation in XIV and a brief review of the SSD technology with focus on the XIV system It explains the SSD Caching design and implementation in XIV Then it examines typical workloads that can benefit from the SSD Caching extension and introduces the tools and utilities to help you analyze and understand the workload In particular it highlights the block tracing facility that was designed and developed by IBM Research Then this paper explains the process that authorized IBM services representatives use to install SSD Caching It reviews the changes
7. This chapter provides high level overview of the SSD technology strictly focusing on aspects that are relevant to the use of SSDs in the XIV system This chapter includes the following sections gt Overview of solid state drives Endurance of solid state drives gt Solid state drive type in XIV Gens O Copyright IBM Corp 2012 All rights reserved 3 2 1 Overview of solid state drives 4 SSDs use semiconductor devices solid state memory to store data Unlike traditional hard disk drives HDD an SSD does not have a mechanical arm to read and write data An SSD does not have any moving parts Currently most of SSDs are based on NAND flash chips NAND flash chips are fast highly reliable widely available and are nonvolatile meaning they can store and retain data even without a power source A flash memory stores the information in an array of flash cells Each cell consists of a single floating gate transistor see Figure 2 1 that can store electrons The cell is programmed by increasing and decreasing the amount of charge placed on it e Source Figure 2 1 Flash transistor cell Two types of NAND flash memory architectures are available as explained in the following sections Single level cell NAND flash Single level cell flash SLC NAND flash memory stores 1 bit of data per memory cell Therefore a single voltage level is required for the device to detect one of the two possible stat
8. Caching at the system level To enable or disable the SSD Caching for the entire XIV system 1 From the toolbar in the main system view click Setting Figure 6 4 File view Tools Help y 9 CG settings eunen XCLI Bf Launch xiv Top itso XIV 1310039 C System Time 03 09pm Q Figure 6 4 Opening the system settings 2 In the Settings panel Figure 6 5 on the Parameters tab set Global SSD Caching to Enabled or Disabled Then click Update XIV 1310039 Coruscant Settings General iSCSI Name iqn 2005 10 com xivstorage 010039 Parameters Time Zone US Arizona NTP Server 9 11 107 12 SNMP DNS Primary 2002 90b e0f2 208 202 55fffe5d dbcc Misc DNS Secondary Use IPv6 Global SSD Caching Enabled Enabled ls Disabled Figure 6 5 System settings for Global SSD Caching By default the newly created volume follows the system level setting 40 Solid State Drive Caching in the IBM XIV Storage System System level settings If SSD Caching is disabled from the system level settings it is disabled by default for every new volume that is defined If SSD Caching is enabled from the system level settings it is enabled by default for every new volume that is defined Setting SSD Caching at the volume level The Volume and Snapshots view in the XIV Storage Management GUI includes a new SSD field that indicates the SSD Caching status for the volumes and defined in the system By default the new SSD field is not included in the vi
9. Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default If you select Properties the Volume Properties window Figure 6 11 opens This panel shows a new SSD Caching State field which indicates whether SSD Caching for the volume is enabled or disabled and whether it is the system Default Volume Properties Name abby 10 Size 51 GB 100823184 Blocks Used Capacity 0 0 GB 0 MiB Size on Disk 51 GB Serial Number 23 Consistency Group one Pool test pool Locked Status 55D Caching State Figure 6 11 Volume Properties window Chapter 6 GUI and XCLI support for solid state drives 44 If you select Change SSD Caching State in Figure 6 10 the Change SSD Caching State window Figure 6 12 opens In this window you can change the status of the SSD Caching for that volume Change SSD Caching State Volume abby_10 Default Enabled Recommended It is recommended to use the default system 55D caching state To change this go to Menu Tools Settings System Parameters CO Manual Enable Disable Using the manual option overrid
10. E i asimov 038 51GB ick k amp adams_127 51GB Unlock M ssimov 0r 51GB Change SSD Caching State L3 adams 065 541GB amp asimov_040 Uem HL adde 51 cal Create Mirrored Snapshot amp asimov 109 51GB Map selected volumes amp adams 067 51GB Map selected volumes manually i asimov 042 54GB View Volume Mapping Figure 6 14 Random Volumes Selection After you select the necessary volumes right click the highlighted volumes and select Change SSD Caching State Tip If volumes are part of a consistency group they do not all need the same SSD Caching setting 6 1 3 SSD Caching performance statistics The Performance Statistics view Figure 6 15 on page 47 was updated to reflect support for SSD Caching Two new metrics and filters are included for read I O transactions gt Mem Hit parameter which indicates read I O from the main DRAM cache gt SSD Hit parameter indicates read I O from the extended SSD cache 46 Solid State Drive Caching in the IBM XIV Storage System Figure 6 15 illustrates a common trend when you enable SSD Caching Total Memory Hit red line goes up thanks to SSD Caching Read Miss blue line goes down and SSD Hit purple line goes up proportionally All Interfaces N Hit X Miss X Memory Hit X SSD Hit X Hit Miss o J in a en Hit Miss IOPS 49134 SSD Hit IOPS 19265 Memory Hit IOPS 17562 Miss IOPS 12307 IOPS 36827 4T 00 17 10 1720 17 30
11. Figure 4 8 IBM WebSphereQ application 22 Solid State Drive Caching in the IBM XIV Storage System Core Enterprise Resource Planning The Financial Database Workload of Core Enterprise Resource Planning ERP shows an increase in IOPS of two times with the addition of SSD Caching in XIV as illustrated in Figure 4 4 Core ERP IOPS gt CRM and Financial DB Workload 70 30 8k 20 TB database XIV Gens WwS5sD AIV Gens Figure 4 4 Financial data application Medical record applications Medical record applications are used to access patient data which is a critical activity for the operation of a hospital Therefore the application has specific performance requirements The database used to store the patient data generates a random read workload that must have good performance For such a situation enabling the SSD cache for only the database volumes allows the full benefit of SSD Caching in the XIV system to be available to the medical database Testing shows that a significant reduction in read latency is possible See Figure 4 5 Medical Record Application Server RT Healthcare EMR Workload 100 random I O 6 TB database Read latency at 20K IOPS AIV Gen3 l gt XIV Gens WSS Figure 4 5 Medical records application Chapter 4 Workload characteristics 23 4 2 Analyzing the workload on XIV The XIV system offers several approaches to analyze the workload 4 2 1 Using the system monitor 24
12. cache for the requested I O If the requested I O exists in the extended cache it is served to the host through the main cache The I O operation is now complete and is recorded as an SSD hit Read data lt 64 KB Read Miss Operation Host requests random data its notin DRAM D Checks for hit on SSD extended cache If data isin SSD data is moved from SSD to DRAM Else forwards the read request Sh dens unmodified to disk drives pages to buffers 2 Disk drives send data to DRAM for log Data copied onto 512KB buffer page When buffer filled up it is sequentially destaged to the SSD fog structured Sequential I O detection bypasses SSD Sequential pre fetch goes direct to disk Sequential pre fetch is fast No need to put sequential data in SSD Large block 64KB reads probably sequential anyway structured write Check SSD extended cache for hit If hit Serves Data 4k read hit granularity SSD Extended Cache SAS Disks Figure 3 4 Cache learning If the operation results in a true read miss not in the DRAM cache and not in SSD extended cache the request is forwarded as unmodified to the disk SAS layer The I O is retrieved from the disk and served to the host by using the main cache From a host perspective the I O operation is now complete and is recorded as a read miss The related pages are copied to reserved buffers in the main cache Imp
13. in Figure 3 2 which is the XIV approach you do not need to relocate data Furthermore with XIV SSDs are used as read cache only When data in cache is no longer relevant it can be dropped and replaced by more relevant data SAS NL DISK DRAM 55D Main Extended Figure 3 2 SSD used as a cache extension With the caching approach the SSD can be used effectively in highly dynamic environments where data patterns are constantly changing For a holistic solution the caching architecture must also deliver good write performance Important XIV implements SSDs as an extended read cache This unique approach to handling SSD is possible on XIV thanks to the high efficiency of its Peripheral Component Interconnect Express PCle buses between SSD and memory or processor With this approach XIV can take full advantage of SSD performance but on traditional storage the SSD might saturate the buses memory and processor 3 2 XIV versus traditional storage architectures Traditional storage systems require the storage administrator to address most of the following questions gt Do we optimize our system for I O per second IOPS or bandwidth gt How do we lay out our storage gt Do we optimize for writes or reads or do we optimize for capacity gt How do we cope with different workload characteristics or workloads that shifts from random to sequential Specifically how do we handle online transaction processing OLTP se
14. management with XCLI 47 overview 13 performance statistics 46 setting 40 state 43 State field 43 volume level 41 ssd_caching_disable command 47 SSD_CACHING_DISABLED 53 ssd_caching_enable command 47 SSD_CACHING_ENABLED 53 SSD_COMPONENT_TEST_STARTED 53 SSD_HAS_FAILED 53 ssd list command 34 35 48 storage architectures traditional versus XIV 11 storage tier 1 system monitor 24 system statistics 19 system level settings 41 T tiering 10 12 policy 12 U use_ssd_caching_default parameter 49 V value key value pair 16 vol default ssd caching get command 48 vol list command 49 vol ssd caching set command 48 volume level SSD caching 41 VOLUME SET DEFAULT SSD CACHING 53 VOLUME SET SSD CACHING 53 W wear leveling algorithm 5 16 WebSphere data store 22 workload 19 characteristics 11 pattern 14 write amplification 5 destage 17 endurance 5 performance 17 write through caching 14 X XCLI 47 Y yellow status 39 58 Solid State Drive Caching in the IBM XIV Storage System Solid State Drive Caching in the IBM XIV Storage System Understand the architecture of solid state drive SSD Caching Learn about GUI and XCLI support and monitoring and alerts for SSDs View workloads that can benefit from SSD Caching This IBM Redpaper publication provides information about the implementation and use of solid state drives SSDs in IBM XIV Storage system XIV Generation 3 Gen3 running XIV soft
15. or the GUI The SSD installation by the IBM service representative is now completed Chapter 5 Installing and upgrading solid state drives 33 The XIV user or system administrator can now verify that the SSDs have a status of OK by using either of the following options gt Run the ssd list command on the XCLI Example 5 2 Example 5 2 Verifying the SSD status by using the CLI ssd list Component ID Status Currently Functioning Capacity Target Status Vendor Model Size Serial Firmware Fru Group Temperature 1 558D 2 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB87 MA40 99Y0720 41 1 SSD 3 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB84 MA40 99Y0720 41 1 SSD 8 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB7E MA40 99Y0720 42 1 SSD 6 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB96 MA40 99Y0720 40 1 SSD 4 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB8D MAZO 99Y0720 41 1 SSD 9 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200C237 MAZO 99Y0720 45 1 SSD 1 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB90 MA40 99Y0720 40 1 SSD 5 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB93 MA40 99Y0720 42 1 SSD 7 1 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200C184 MA40 99Y0720 43 gt Using the GUI as shown in Figure 5 3 All Systems View By My Groups gt XIV Fest 7820 gt System XIV Gen3 2TB SSD a Status OK om AAT Module 9 Status Coos EL Figure 5 8 SSD Cache Sta
16. period At any time the area of randomness can shift on the storage system Knowing the moving hit window for an application is critical in seeding a correct caching architecture The block tracing facility relies on a cumulative I O distribution function to determine the areas of a volume that are accessed most frequently That information can help to predict how to use SSD Caching when it is installed The block tracing facility has minimal performance impact on the XIV system The overall CPU usage is around 0 1 and no disk I Os are performed In addition IBM personnel can offload the trace that is taken on an XIV system at a customer site They can replay it in an IBM lab on an XIV system of the same capacity but that has SSD Caching to validate the predictions Components and processes of the XIV block tracing facility Figure 4 9 highlights the components of the block tracing facility Management IP1 br 2s Management IP2 Production Hosts 4 i Management IP3 EA EA Logs accumulate on the client Block tracing process polls the interface nodes and logs all lO requests from the BT client host l EE connects to When a client collection is started the any Management IP BT client starts to record the requests Figure 4 9 Block tracing facility components The statistics recording process BT server runs on the XIV system The XIV system includes up to six interface modules which are also referred to as i
17. see Chapter 3 Caching architecture and implementation on page 9 Solid State Drive Caching in the IBM XIV Storage System Table 2 1 summarizes the major MLC and SLC differences The third column shows the functions that implemented in the XIV system software that compensate for reduced tolerance These functions are also described in this paper Table 2 1 SLC versus MLC Operation MLC XIV software features added in v11 1 0 Write cycles Log structured write XIV software wear leveling Read speed Higher Lower Limits SSD use to only random operations High bandwidth workloads are handled by the SAS disk layer Density Reduces costs and increases usable footprints Chapter 2 Technology overview of solid state drives 7 8 Solid State Drive Caching in the IBM XIV Storage System Caching architecture and implementation SSD Caching is a deterministic cache layer that can adapt to mixed workloads Extended SSD Caching in XIV Gen3 specifically targets small block random read input output I O XIV relies on its InfiniBand back end to support low latency writes and high throughput This chapter explains the solid state drive SSD Caching architecture of IBM XIV Storage System Gen3 It also explains the implementation of SSD Caching and how it cooperates with core caching to extend its capabilities This chapter includes the following sections gt SSD tiering versus caching gt XIV versus traditional storage architectures
18. version get Version 11 1 0 RC1 p20120213 094718 32 Solid State Drive Caching in the IBM XIV Storage System 9 2 Installing SSD After the system has the appropriate XIV software version the IBM service representative can install the SSDs in your XIV Gen3 system Restriction Only an authorized IBM service representative can install SSDs The IBM service representative installs SSD by using the following tasks 1 From the IBM XIV Storage Management GUI verify that the XIV system status on the lower right side of the window shows Full Redundancy 2 From the rear of the rack insert an SSD into each data and interface module as illustrated in Figure 5 2 Orient the SSD so that the arrow on the pull tab points upward Push the release tab to the side and insert the SSD into the opening in the carrier assembly Release the tab to secure the SSD in place Important Install SSDs in every data and interface module starting from the lowest module in the rack and working upward solid state disk Pull tab A d i E EBEE om E Eb EE Retention tab Carrier assembly i XIV 10226 Figure 5 2 Inserting an SSD into the module 3 Using an XCLI command strictly reserved for the IBM service representative enable the flash cache and discover the SSD modules 4 Run a component test on each SSD When the component test is over and is successful all SSDs are ready 5 Phase in all SSDs by using the XCLI
19. 17 40 17 50 18 00 13 10 18 20 18 30 18 40 18 50 19 00 19 10 19 20 19 30 19 40 19 50 20 00 20 10 Lo 16 Feb 2012 A i I f r r Interfaces a Read a Hit O 64 512 KB amp IOPS CN Volumes p j Day O Write 2 Miss 0 8 KB gt 512 KB C Latency roo ML E Pos Es 5 Om 0j oo CRW 2 H M P 8 84 KB All JBW Figure 6 15 Performance Statistics view showing the Mem Hit and SSD Hit parameters For more information about SSD performance see Chapter 4 Workload characteristics on page 19 6 2 SSD Caching management with XCLI New commands are included in the XCLI that match the possible actions described for the GUI For example the help search ssd command shows a list of all commands related to SSDs Example 6 1 Example 6 1 List of commands related to SSDs XIV 1310039 Coruscant gt gt help search ssd Category Name Description system ssd caching disable Disables Flash Caching system ssd caching enable Enables SSD Caching system ssd list Lists SSDs used as flash cache in the system system vol default ssd caching get Gets the Default Stateof the SSD Caching system vol default ssd caching set Sets aDefault State for SSD Caching system vol ssd caching set Overrides the Default SSD Caching State for a Volume The ssd caching disable and ssd caching enable commands listed in Example 6 1 are restricted commands that can be used only with authority from an IBM service repres
20. 3 5 3 Upgrading SSD after a module upgrade 0 0 cc no 35 Chapter 6 GUI and XCLI support for solid state drives 37 641 SSD Caching SUDDOM AN XIV toes awe este deba a wea eS ee Me 38 6 1 1 Viewing and managing the SSD by using the GUl 38 6 1 2 Seling 59D Caching sow tano Bea ct oe a ee ame E ee lard 40 6 1 3 SSD Caching performance statistics 0 0 ees 46 6 2 SSD Caching management with XCLI 0 0 ces 47 Chapter 7 Monitoring and alerting 0 0 cc ees 51 71 MONIONING SSDSs PIE 52 Tee PICMG hemo ye der rd a ae a wey dao ados 53 Copyright IBM Corp 2012 All rights reserved lii Related DUDIICallOnS 4 239 rS Ede ERIT Ee d idee dado sau e taie 55 IBIVINFA COD OOK S espa tocara aaa ai aaa see ima 55 Other DUDICAIONS dou dco dv ia aa a ae eS ee ead Gelb edere 55 Online Tesoul6es 9 esce ar uto A ur Bs doses oe ex aoo ac oca d Doe Et Te e EU d 55 ICID MOM tai br iu mee epos e 56 O A E E E A O MM 57 Iv Solid State Drive Caching in the IBM XIV Storage System Notices This information was developed for products and services offered in the U S A IBM may not offer the products services or features discussed in this document in other countries Consult your local IBM representative for information on the products and services currently available in your area Any reference to an IBM product program or service is not intended to state or
21. Columns Volumes and Snapshots Table Columns Hidden Columns Visible Columns Used Capacity MiB Deletion Priority Created On Master Creator Serial Number Figure 6 8 SSD field moved to the Visible Columns area b To commit the change click Update You return to the Volumes and Snapshots view with the new SSD column added Figure 6 9 File View Tools Help m o Add Volumes All Systems View By My Groups gt XIV 1310039 C gt Volumes and Snapshots itso System Time 05 44pm OQ Size GB Used GB Consi abby 10 test pool asimov 125 mirror pool asimov 105 mirror pool abby 12 test pool asimov 038 mirror pool adams 127 mirror pool asimov 107 mirror pool adams 065 mirror pool asimov 040 mirror la asimov_117 mirror_pool asimov_109 mirror_pool adams_067 mirror_pool asimov_042 mirror_pool asimov_044 mirror_pool asimov_101 mirror_pool asimov_046 mirror_pool e oarnbat cirrus amp 08 test pool adams 079 mirror pool cirrus8 10 test pool abby 02 test pool asimov 097 mirror pool asimov 099 mirror pool asimov 032 mirror pool adams 023 mirror pool asimov 127 mirror pool asimov 034 mirror pool Solid State Drive Caching in the IBM XIV Storage System Created Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled
22. Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Enabled Default Changing the SSD Caching state If you right click a volume row you can click either Change SSD Caching State or Properties to see the volume properties Figure 6 10 File View Tools Help nm o wf Add Volumes All Systems View By My Groups gt XIV 1310039 C gt Volumes and Snapshots Size GB asimov_125 Delete asimov_105 abby_12 Format asimov_038 Rename adams_127 Create a Consistency Group With Selected Volumes asimov_107 adams_065 asimov_040 Add To Consistency Group Remove From Consistency Group Move To Consistency Group asimov_117 Move to Pool asimov_109 adams_067 Create Snapshot Create Snapshot Advanced Overwrite Snapshot asimov_042 asimov_044 asimov_101 Copy this Volume asimov_046 Restore cirrus8 08 adams 079 Lock H Inlo Change SSD Caching State cirrus8 10 abby_02 a Create Mirror asimov_097 5 Create Mirrored Snapshot asimov_099 a Map selected volumes adams_023 Map selected volumes manually asimov 127 View Volume Mapping asimov 034 as asimov_032 CALELLA Figure 6 10 Right clicking a volume row Enabled Default Enabled Default Enabled
23. E Solid State Drive Caching in the IBM XIV Storage System Understand the architecture of solid state drive SSD Caching Learn about GUI and XCLI support and monitoring and alerts for SSDs View workloads that can benefit from SSD Caching Bertrand Dufrasne In Kyu Park Francesco Perillo Hank Sautter Stephen Solewin Anthony Vattathil ome hedpaper l International Technical Support Organization Solid State Drive Caching in the IBM XIV Storage System May 2012 REDP 4842 00 Note Before using this information and the product it supports read the information in Notices on page v First Edition May 2012 This edition applies to the IBM XIV Storage System Gen3 with software Version 11 1 x O Copyright International Business Machines Corporation 2012 All rights reserved Note to U S Government Users Restricted Rights Use duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp Contents NOUCOS M FC V Trademarks usa doe Y Ce e de deis A e a artisti eU R vi aid Me TC rrr vii The team who wrote this paper n naana aaaea vii Now you can become a published author too o ooooooocooooroonooo viii Comments WECOME ac ua oo tie UU e a dd IX Stay connected to IBM Redbooks o o oocooocooonon lr IX Chapter 1 Executive summary o o ooooo ee ees 1 Chapter 2 Technology overview of solid state drives 3 2 1 Ov
24. Gen3 that ships can take advantage of a concurrent code upgrade that enables the use of the SSD extended cache SSD extended cache is available for all Gen3 systems both fully and partially populated Every module in the XIV must be upgraded with the extended cache For more information about IBM XIV Storage System Gen3 with software release 11 1 see the announcement letter at http www ibm com common ssi rep ca 1 897 ENUS112 031 ENUS112 031 PDF For more information about XIV see the XIV page at http www ibm com systems storage disk xiv Copyright IBM Corp 2012 All rights reserved 1 2 Solid State Drive Caching in the IBM XIV Storage System Technology overview of solid state drives In storage environments today the cost of a storage system is as much a concern as the performance that it delivers Starting with its first generation the IBM XIV Storage System and its software work with off the shelf therefore lower cost components The need for controlling costs is the reason that the XIV Gen3 software was designed to accommodate the less expensive multilevel cell MLC solid state drive SSD technology The integration of SSDs in XIV increases random read performance and maintains enterprise level reliability and resiliency As explained in this chapter the XIV system software prevents some shortcomings of the MLC SSD technology which render the XIV system independent of the specifics of any particular SSD vendor
25. IV volumes always engage all available physical disks spindles This core concept of the XIV architecture delivers excellent write performance without generating hot spots Areduced read load on the disks thanks to SSD Caching also yields more efficient destaging of dirty data allowing better write throughput to XIV gt XIV Gens uses InfiniBand as its back end protocol and can acknowledge writes out of cache three times faster than previous generation gt XIV uses an intelligent write caching algorithm that allows for small random writes to be combined into larger more spindle friendly writes Write performance By using InfiniBand and intelligent write destage algorithms across all physical disks XIV Gen3 delivers high performance writes Chapter 3 Caching architecture and implementation 17 Single I O write performance occurs at DRAM speed because all writes are acknowledged in DRAM To be precise writes are acknowledged after they are mirrored safely at the secondary cache node 3 2 4 SSD failure If an SSD starts to fail or fails completely it is phased out similar to any of the other XIV grid components If the SSD is phased out and not failed its data is invalidated If SSD is phased back in not all data is lost When an SSD is failed the system initializes all of the metadata again so that when the SSD is phased back in no integrity issues are expected Data is not redistributed which is the case after a module
26. MLC SSDs 2 2 Endurance of solid state drives The life of a flash is limited by the number of write erase operations that can be performed on the flash To extend the lifetime of a drive and to ensure data integrity on the drive SSDs use a built in dynamic or static wear leveling algorithm and bad block mapping algorithm These algorithms are further complemented by the over provisioning algorithm the error detection code algorithm and the error correction code algorithm to ensure data reliability These algorithms are implemented in various electronic components of the drive gt Wear leveling algorithm The goal of the wear leveling algorithm is to ensure that the same memory blocks are not overwritten too often With this mechanism the flash controller distributes the erase and write cycles across all the flash memory blocks gt Bad block algorithm The bad block algorithm detects faulty blocks during operations and flags them These flagged blocks are excluded from the rotation and replaced with good blocks so that the data does not go into bad blocks gt Over provisioning algorithm In combination with the wear leveling algorithm the over provisioning algorithm is used to further alleviate write endurance issues The over provisioning algorithm minimizes the amount of write amplification needed A flash memory must be erased before it can be rewritten and a whole erase block 512 KB 2 MB must be erased in a single erase op
27. Service OK PSUs OK Fans Ak SSD SSD 1 Ready e 2 omes e 0 m s E Figure 5 4 Module status after additional module installation As explained in 5 2 Installing SSD on page 33 the IBM service representative runs a component test for the additional SSDs and then enables them The status of the additional SSDs is now displayed as OK You can verify the status in the XIV GUI as shown in Figure 5 3 on page 34 or by using the ssd list XCLI command as shown in Example 5 2 on page 34 Chapter 5 Installing and upgrading solid state drives 35 36 Solid State Drive Caching in the IBM XIV Storage System GUI and XCLI support for solid state drives This chapter describes the changes made to the IBM XIV Storage Management GUI and XIV command line interface XCLI commands to accommodate the new solid state drive SSD Caching feature It includes the following sections gt SSD Caching support in XIV gt SSD Caching management with XCLI Copyright IBM Corp 2012 All rights reserved 37 6 1 SSD Caching support in XIV After the IBM service representative installs and enables the SSDs the XIV system can use them as an extension of the read cache The SSD cache automatically adjusts to the workload providing a performance boost for small I O random reads The storage administrator does not need to perform any additional configuration or tuning However the storage administrator has the flexibility to perf
28. aching 3 Determine how many of those read requests are small less than 64 KB random reads Here the smaller the read I O blocks the better the workload benefits from SSD Caching In summary a workload with I Os that are small block random read misses can benefit from SSD Caching in the XIV system 4 1 1 Workload performance benefits 20 SSD Caching in the XIV system is quick to respond to changes in workload characteristics and provides increased performance in the following situations gt O operations per second IOPS increase of up to three times for an online transaction processing OLTP workload gt IOPS increase of up to six times for a highly random workload gt A latency reduction of almost three times for a medical records application Solid State Drive Caching in the IBM XIV Storage System Testing shows that an IOPS increase of six times and 1 ms sublatency is possible when the working data set fits in the SSD cache as illustrated in Figure 4 1 Best Case Random Reads Performance BDO0GEB X 10 Volumes 278 SAS Drives 100 Random Read Response Time ms ATI 4OXKK OOQOR ODO 100000 120000 140000 16000 180000 OPS Best Case Data Base Open Workload Performance 5006GB X 10 Volumes TB SAS Drives 100 Random Read hesporee Time ms 00006 40000 680000 Ss0000 100000 120000 140000 160000 120000 00000 IO Ps Figure 4 1 Random read performance Again generally random read activity
29. aken was to enable or disable SSD Caching for that particular volume Chapter 7 Monitoring and alerting 53 54 Solid State Drive Caching in the IBM XIV Storage System Related publications The publications listed in this section are considered particularly suitable for a more detailed discussion of the topics covered in this paper IBM Redbooks The following IBM Redbooks publications provide additional information about the topic in this document Note that some publications referenced in this list might be available in softcopy only gt IBM XIV Storage System Architecture Implementation and Usage SG24 7659 gt IBM XIV Storage System Copy Services and Migration SG24 7759 gt IBM XIV Storage System Host Attachment and Interoperability SG24 7904 You can search for view download or order these documents and other Redbooks Redpapers Web Docs draft and additional materials at the following website ibm com redbooks Other publications These publications are also relevant as further information sources gt IBM XIV Storage System Planning Guide GC27 3913 gt IBM XIV Storage System Product Overview GC27 3912 gt IBM XIV Storage System User Manual GC27 3914 gt IBM XIV Storage System XCLI Utility User Manual GC27 3915 Online resources These websites are also relevant as further information sources gt IBM XIV Storage System Information Center http publib boulder ibm com infocenter ibmxiv r2 index jsp
30. and upgrades are nondisruptive procedures This chapter includes the following sections gt Prerequisites gt Installing SSD gt Upgrading SSD after a module upgrade Copyright IBM Corp 2012 All rights reserved 31 5 1 Prerequisites Before the IBM service representative can install SSDs ensure that XIV storage software firmware Version 11 1 0 or later is installed on the XIV system The IBM service representative also upgrades the XIV system code However you can check your system software level before ordering the SSDs by using the XIV Storage Management GUI or the XIV command line interface XCLI gt Checking the system version by using the GUI To check the code level on GUI click Settings on system dashboard larger window in Figure 5 1 In the Settings window that opens you can read the system version inset in Figure 5 1 bivbs t Storage Management XIV Fest 7820119 Settings General System Name XIV Fest 7820119 Parameters System Version 11 1 0 RC 1 p20120213 094718 System ID S N 7820119 20119 SNMP Machine Model Machine Type 114 2810 Misc IP Hostname 1 9 11 209 234 IP Hostname 2 IP Hostname 3 Figure 5 1 Checking the system code level by using the GUI gt Checking the system version by using the XCLI To check the system version open the XCLI window and enter the version get command as shown as Example 5 1 Example 5 1 Checking the system code level by using XCLI gt gt
31. ary Solid state drives SSDs are still a relatively new addition to storage systems SSDs are typically used to greatly reduce response time and increase random read input output I O operations per second IOPS when compared to traditional hard disk drives HDD The IBM XIV Storage System Gen3 with software version 11 1 supports the use of SSDs However in the XIV system SSDs are used to increase the read cache capacity of the existing DRAM cache They are not used for persistent storage which departs from the approach used by many other storage systems that use SSDs as another faster storage tier Important In XIV Gen3 SSDs are used solely as an extension of the system read cache XIV Gen3 uses 400 GB multilevel cell MLC SSDs increasing the total cache by 6 TB for a fully populated system SSD Caching in IBM XIV specifically targets l O intensive online transaction processing OLTP workloads With its sophisticated cache algorithms and use of the XIV massive internal bandwidth this extended cache can be used to accelerate target workload performance from two to three times faster With certain workloads it can accelerate target workload performance up to six times faster The caching algorithm is embedded in the XIV system software firmware or microcode and makes the integration of SSDs transparent to the user or storage administrator No tuning of the cache is required to achieve the potential performance boost Every XIV
32. d libraries SAN storage virtualization and storage software Steve has been working on the XIV product line since March of 2008 He holds a BS degree in electrical engineering from the University of Arizona where he graduated with honors Anthony Vattathil is a IBM Corporate Architect in California He has over 12 years of experience in the storage field His areas of expertise include large scale SAN design and technical knowledge of various storage systems Tony is also well versed on various Linux and UNIX system platforms He has written extensively on storage performance and is one of the performance leads for the XIV system Thanks to the following people for their contributions to this project gt For their contribution and support Shimon Ben David Brian Carmody Orli Gan Haim Helman Aviad Offer Ohad Rodeh Jim Sedgwick Yijie Zhang IBM Corporation gt For access to equipment and resources Rachel Mikolajewski George Thomas Peter Wendler IBM System Level Test Lab in Tucson AZ Now you can become a published author too Here s an opportunity to spotlight your skills grow your career and become a published author all at the same time Join an ITSO residency project and help write a book in your area of expertise while honing your experience using leading edge technologies Your efforts will help to increase product acceptance and customer satisfaction as you expand your network of technical contacts and relationshi
33. dom OLTP workload that runs from 8 a m to 8 p m and uses a database The size of the database is constantly growing and growing quickly User access patterns vary throughout the day and directly affect which volumes or regions of the databases are most active In this situation tiering cannot be proactive enough to move the corresponding blocks in a real time manner By the time relevant blocks are moved to SSD the hot region is elsewhere To complicate matters more the same database might be backed up by using a multithreaded copy that shifts the workload characteristic of a large block sequential read A storage system equipped with many nearline SAS NL SAS delivers high throughput Conversely traditional NL SAS RAID groups do not perform well for small block random I O An option is to keep the entire database on SSD but this option is too costly In such situations the XIV architecture helps in the following ways gt XIV has a predefined immutable disk configuration that enables the implementation of a single optimized caching algorithm gt XIV has a selective read cache algorithm that tunes itself to reduce random access to the disk layer or forwards bandwidth intensive I O operations to the disk layer XIV can dynamically tune its cache for random and sequential workloads The storage administrator must be concerned with only the capacity allocation After the storage administrator assigns storage to the host the XIV system s
34. earch ssd command 47 host write 13 14 hot page 16 I O collection client program 27 reads 24 statistics 20 I O operations per second IOPS 11 InfiniBand 17 inodes 26 interface nodes 26 IOPS I O operations per second 11 K key value pair 16 L logical erase block 6 log structured write 16 M medical record application 23 Mem Hit parameter 52 metadata 13 MLC multilevel cell 1 3 MLC NAND flash 4 6 module primary 14 secondary 14 monitoring SSD 52 moving hit window 26 multilevel cell MLC 1 3 N NAND Flash 4 O OLTP online transaction processing 1 12 57 online transaction processing OLTP 1 12 over provisioning algorithm 5 P prefetch 16 random read 16 primary module 14 programmed 4 R RAID levels 12 random read prefetch 16 SSD Caching enabled 15 read l Os 24 read miss 14 15 read prefetch random 16 recording process 26 red status 39 Redbooks website 55 Contact us ix replay program 28 S SAS disk rebuild 18 secondary module 14 selective read cache algorithm 14 sequential read 14 single level cell NAND flash 4 SLC NAND flash 4 6 solid state drive SSD See SSD SSD alerts 53 cache learning 15 cache map 15 cached data updates 17 events 53 failure 6 18 field addition of 41 GUI support 37 Hit parameters 52 installation 31 33 monitoring 52 tiering versus caching 10 update 16 upgrade 31 XCLI support 37 SSD Caching disabled 37 40 41 48 53 enabled 37 40 41 48 53
35. ecuted successfully default disabled If the default status is enabled SSD Caching is enabled on all the volumes in the system unless manually changed Otherwise if the default status is disabled SSD Caching is disabled for all the volumes in the system Options for an enabled state With the SSD Caching state enabled you can choose to explicitly disable any volume that must not be included in the extended caching To change the SSD Caching state globally for the system issue one of the following commands gt Set the SSD Caching to enabled vol default ssd caching set default enabled gt Set the SSD Caching to disabled vol default ssd caching set default disabled Use the vol ssd caching set command to set the SSD Caching status for a specific volume and eventually override the system default setting You must specify the vol and state parameters when issuing the command as shown in Example 6 4 Example 6 4 SSD Caching set for a volume XIV 1310039 Coruscant vol ssd caching set vol Res Fra Vol 01 state enabled Command executed successfully XIV 1310039 Coruscant gt gt vol ssd caching set vol Res Fra Vol 01 state disabled Command executed successfully 48 Solid State Drive Caching in the IBM XIV Storage System You can also use the vol list command with the x flag and the vol parameter to view all of the volume properties The properties now include two additional values ssd caching and use ssd caching default Example 6 5
36. entative The IBM service representative uses these commands to bring SSDs online when they are Chapter 6 GUI and XCLI support for solid state drives 47 phased into the XIV system at installation time For more information see Chapter 5 Installing and upgrading solid state drives on page 31 Use the ssd list command to access a list of SSDs that are used as flash cache in the system Example 6 2 shows the results of using this command Example 6 2 List of SSDs XIV 1310039 Coruscant ssd list Component ID 1 SSD 3 1 1 SSD 2 1 1 SSD 6 1 1 SSD 4 1 1 SSD 1 1 1 SSD 5 1 Status Currently Functioning Capacity Target Status Vendor Model Size Serial Firmware Fru Group Temperature OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB81 MAZO 99Y0720 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB78 MAZO 99Y0720 40 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB71 MAZO 99Y0720 47 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB95 MAZO 99Y0720 47 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB7A MAZO 99Y0720 39 OK yes 512GB XIV MTFDDAA512MAR 1KAAB 488378 0200BB7D MA40 99Y0720 43 The vol default ssd caching get command checks the default SSD Caching setting either enabled or disabled as shown in Example 6 3 Example 6 3 Checking the SSD Caching state XIV 1310039 Coruscant gt gt vol default ssd caching get Command executed successfully default enabled XIV 1310039 Coruscant gt gt vol default ssd caching get Command ex
37. eration Therefore typically more physical space is used than required by the amount of logical data to be written which is known as write amplification Over provisioning is the difference between the physical capacity of the flash memory and the logical capacity that detected by an application or host The SSD has more capacity than is usable gt Error detection code and error correction code algorithm The error detection code and error correction code algorithms maintain data reliability by allowing single bit or multiple bit corrections to the data that is stored If the data is corrupted due to aging or during the programming process the error detection code and error correction code algorithms compensate for the errors to ensure the delivery of accurate data to the host application Chapter 2 Technology overview of solid state drives 5 2 3 Solid state drive type in XIV Gen3 6 XIV Gen3 uses enterprise quality MLC NAND flash SSDs As indicated previously the MLC NAND flash is slower and has a shorter lifetime than the SLC NAND flash These disadvantages can be alleviated by sophisticated algorithms which is the approach taken by XIV In addition to the inherent techniques to the SSD and that are listed 2 2 Endurance of solid state drives on page 5 XIV Gen3 with software version 11 1 implements patented algorithms These algorithms can for example restrict direct host access to SSD operations such as dealing with logical era
38. erview of solid state drives u nan anaana aea 4 2 2 Endurance of solid state drives llle 5 2 3 Solid state drive type in XIV Gen8 lille 6 Chapter 3 Caching architecture and implementation 9 o SoD NennG Versus CaCniING 23 uuu goo ce Berta add a c Dr c cad 10 3 2 XIV versus traditional storage architectures llle 11 3 2 1 Overview of SSD Gaching 2244 34 ya peres A RI x dee P deg 13 3 2 2 Selective read cache algorithm for extended Cache ooooooooo 14 3 2 3 Random reads with SSD Caching enabled 0 000 eee ee eee 15 92 4 SoD fal E unas ait ot uon Bape oem Bee Wiis ieee Se d Ape UR 18 3 20 OAS CISKTCDUIG 3 9 ur stem uice potet dt Uter ee Me whl bel eae See tee 18 Chapter 4 Workload characteristicS o ee eee 19 4 1 Workload tor SSD Cachi osea opes ec Meee hoe seo eee ar n 20 4 1 1 Workload performance benefits a 20 4 1 2 Typical applications and performance benefits llus 22 4 2 Analyzing the workload on XIV o o oooocooocoooo ne 24 4 2 1 Using the system monitor llle 24 4 2 2 Using the XIV block tracing facility llle 26 Ao DISK Tage 1 s mera c E adi our atus facti ost ace qu o tah Co 29 Chapter 5 Installing and upgrading solid state drives 31 SX PEE QUISIMOS p PRETI 32 5 2 WNStalliNG SSD 1 pe aa feat SP doo sados 3
39. es of the memory If a current is detected the bit is stored as 0 meaning programmed lf no current is detected the bit is 1 meaning erased Thanks to this rather simple architecture SLC NAND flash can offer quick read and write capabilities long term durability and rather simple error correction algorithms Besides the limited programming density SLC NAND flash has one other disadvantage That is itis considerably more expensive per bit when compared to other NAND technologies The higher cost is due to each cell storing only 1 bit of data However SLC NAND flash is the ideal choice when performance is the driver and cost is not an issue Multilevel cell NAND flash Multilevel cell flash MLC NAND flash memory can store at least 2 bits of data per memory cell Therefore multiple voltage levels are necessary to decipher between the four possible states of memory Solid State Drive Caching in the IBM XIV Storage System Each memory cell can have the following range of states 00 Fully programmed 01 Partially programmed 10 Partially erased 11 Fully erased Two advantages of this increase in bits per cell are that the memory capacity is greater and production costs are reduced As a result MLC NAND flash can be sold at a much lower price per bit than SLC NAND flash and it offers twice the density However the MLC NAND flash is slower and has a shorter lifetime than the SLC NAND flash Important The XIV Storage System Gen3 uses
40. es the default 55D caching state Figure 6 12 Changing the SSD Caching state of the volume If the Default setting is selected the volume follows the current system settings You can override the Default setting by selecting Manual Then the volume no longer follows the current default system settings In this case you can select either of the following options gt Enable to enable SSD Caching for the selected volume gt Disable to disable SSD Caching for the selected volume Tip The overall system status setting for SSD Caching is Default enabled as shown in Figure 6 12 You can select more than one volume in the Volumes and Snapshots view if you need to change the SSD Caching State for a list of volumes gt Click the first volume to select it The volume becomes highlighted in yellow as shown in Figure 6 10 on page 43 Solid State Drive Caching in the IBM XIV Storage System gt To select a sequential list of volumes hold down the Shift key and click the last volume that you want to include All the volumes in the range are selected and highlighted as shown in Figure 6 13 FE sa cU WW Pu PEPE Figure 6 13 E b Name asimov 121 abby_08 asimov 123 asimov 103 asimov 031 abby 10 asimov 125 asimov 105 abby 12 asimov 038 adams 127 asimov 107 adams 065 asimov 040 asimov 117 asimov 109 adams 67 asimov 042 asimov 044 Systems View By My Groups gt XIV 1310039 C Vol
41. ests over a period Read a EIEREN 4 20 03 12 13 20 IOPS 20056 Random Read Workload 12 20 12 30 12 40 12 50 13 00 13 10 13 20 13 30 13 40 13 50 14 00 Figure 4 7 Total read and write IOPS Figure 4 8 shows only read requests with a size less than 64 KB that are cache read misses This figure represents the portion of the workload that can directly benefit from extended SSD Caching Read Miss 8 64 KB 20 03 12 13 20 o IOPS 13270 All Os displayed here are candidates for SSD Caching 12 20 12 30 12 40 12 50 13 20 Figure 4 8 Read cache misses You can also do in depth analyses of a workload that is running on an XIV system by using the new XIV block tracing feature Chapter 4 Workload characteristics 25 4 2 2 Using the XIV block tracing facility An XIV Gen3 system with software version 11 1 but not yet equipped with SSD Caching includes a unique feature known as block tracing Block tracing is for use by IBM technical staff and is meant to deliver additional client value by allowing a deeper understanding of an application workload pattern The block tracing facility which was conceived and developed by IBM Research collects usage statistics over time to show the application access pattern against a host volume This pattern helps identify the moving hit window The moving hit windows refers to the active candidates for SSD Caching for a time
42. ew Adding the SSD field To include the SSD field in the view 1 Open the Volumes and Snapshots view from the main XIV GUI 2 Right click anywhere on the field bar Figure 6 6 and the Customize Columns window opens xiv XIV Storage Management File view Tools Help y O i Add Volumes itso Il Systems View By My Groups Volumes and Snapshots System Time 10 09 am OQ Name Size GB lt Pool Res Fra Vol 04 B p Residency Francesco t Res Fra Vol 02 17 GB 0 GB Residency Francesco t Res Fra Vol 03 17 GB 0 GB Residency Francesco t Res Fra Vol 05 17 GB 0 GB Residency Francesco t Res Fra Vol 01 17 GB 0 GB Residency Francesco t adams 069 51 GB 0 GB mirror pool qe t abby 06 51 GB 0 GB test pool t asimov 121 51 GB 0 GB mirror pool oc t abby 08 51 GB 0 GB test pool t asimov 123 51 GB 0 GB mirror pool Figure 6 6 Volumes and Snapshots view 3 In the Customize Columns window Figure 6 7 a In the Hidden Columns area select SSD and then click the yellow right arrow indicator Customize Columns Volumes and Snapshots Table Columns Hidden Columns Visible Columns Used Capacity MiB Master Coupling Status Snapshot Formatted Deletion Priority Created On Master Snapshot Modification Status Created Figure 6 7 Customize Columns window Chapter 6 GUI and XCLI support for solid state drives 41 42 The SSD field moves to the right box as shown in Figure 6 8 Customize
43. he IBM International Technical Support Organization Experts from IBM Customers and Partners from around the world create timely technical information based on realistic scenarios opecific recommendations are provided to help you implement IT solutions more effectively in your environment For more information ibm com redbooks
44. he following sections gt Workload for SSD Caching gt Analyzing the workload on XIV Copyright IBM Corp 2012 All rights reserved 19 4 1 Workload for SSD Caching As mentioned previously in this paper in the XIV Storage System SSD Caching is implemented as a read cache exclusively and is not used as another storage tier Moreover SSD Caching improves small block random read performance SSD Caching in the XIV system can improve performance of any workload that is suffering from high latency due to a high amount of small block random read misses Consider a situation where you do not have an XIV system yet but you want to know whether you might benefit from by extending the XIV with SSD Caching On most storage systems you can inspect l O statistics and determine whether a workload can benefit from SSD Caching If the storage system has granular reporting you can take the following steps 1 Sort your workload by read and write percentages The read portion of the workload can potentially benefit from the SSD Caching A typical candidate workload for SSD Caching consists of 80 reads and 20 writes 2 Determine how many of the read requests were misses or rather where the data was not in the cache You can gather this information by looking at the read miss rate which determines how many read requests go to disk and are not serviced out of cache The more read misses you have the better your workload benefits from SSD C
45. he node pulls up to 64 KB from the disk layer and stages it into an SSD Even a small 4 KB random read miss can trigger a 64 KB prefetch The proactive cache builds are then inspected over several cycles If the additional pages that were pulled into SSD are accessed they are retained in the SSD cache and marked as hot Extended cache management XIV extended cache layer is indexed in a memory resident hash table XIV reserves 0 396 of the size of the total extended cache capacity to save the hash table As part of the shutdown process the contents of the hash table are dumped to the SSDs Upon startup the data is validated and repopulated into a hash table in memory The hash data structure consists of key value pair records In a key value pair a key is the disk page location and a value represents the location where data is stored on the SSD On a DRAM read miss the key is used to determine whether there is an SSD hit If an SSD hit exists the value is used to retrieve the data from SSD and satisfy the I O request Solid State Drive Caching in the IBM XIV Storage System Updates to SSD cached data XIV does not allow hosts direct access to the SSD cache After the initial SSD cache learning occurs the SSD cache holds a transient copy of the data blocks When the extended cache must be updated because of a new host write the corresponding hash table entry is invalidated XIV handles the write updates as a normal write operation D
46. imply that only that IBM product program or service may be used Any functionally equivalent product program or service that does not infringe any IBM intellectual property right may be used instead However it is the user s responsibility to evaluate and verify the operation of any non IBM product program or service IBM may have patents or pending patent applications covering subject matter described in this document The furnishing of this document does not give you any license to these patents You can send license inquiries in writing to IBM Director of Licensing IBM Corporation North Castle Drive Armonk NY 10504 1785 U S A The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION AS IS WITHOUT WARRANTY OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF NON INFRINGEMENT MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you This information could include technical inaccuracies or typographical errors Changes are periodically made to the information herein these changes will be incorporated in new editions of the publication IBM may make improvements and or changes in the product s and or the progra
47. ined the GMU ATS team He now focuses on IBM strategic storage products including the IBM XIV IBM Storwize V7000 and IBM SAN Volume Controller Francesco Perillo is an IT Storage Specialist in IBM Italy He has over 6 years of experience in working on EMC and IBM Enterprise Storage and has been involved with the planning design implementation management and problem analysis of storage solutions His areas of expertise include the SAN infrastructure enterprise disk storage and IBM storage software Francesco received an engineering degree from Rome Tor Vergata University and holds two technical storage certifications from EMC Hank Sautter is a Consulting IT Specialist with Advanced Technical Support for IBM US and has been with IBM for 33 years He has 20 years of experience with IBM S 3900 and IBM O Copyright IBM Corp 2012 All rights reserved vii disk storage hardware and Advanced Copy Services functions His previous IBM experience includes working on IBM Processor microcode development and S 390 system testing Hank s areas of expertise include enterprise storage performance and disaster recovery implementation for large systems and open systems which he regularly writes and presents on He holds a Bachelor of Science BS degree in physics Stephen Solewin is an XIV Corporate Solutions Architect for IBM in Tucson Arizona He has 16 years of experience working on IBM storage including Enterprise and Midrange Disk LTO drives an
48. l 9 Leeden Listiliew Cont Modell XIV Gen3 SSD Base 1 XIV Gen3 52 Project WolSer 0o Eg Model General Interfaces Disk Page Open Workload a Open a XIV Gen3 55D Hame Av Gen3 SSD Hardware Type IBM IVY Gen X Manufacturer IBM m ED 360 Additional SSD Cache Number of modules 15 per module GB 400 GB Memory per Module GB 24 total B 6000 Description jr j Hardware Details Histor Bas Bepat Help Autosave iz off F 4 lool Log Figure 4 12 Using Disk Magic to model XIV Gen3 with SSD Caching Chapter 4 Workload characteristics 29 30 Solid State Drive Caching in the IBM XIV Storage System Installing and upgrading solid state drives This chapter outlines how an IBM service representative installs the solid state drive SSD Caching feature on an IBM XIV Storage System that is deployed in a client environment After they install and enable the XIV system modules the SSDs are ready for use No additional monitoring configuring or tuning is necessary The SSD Caching feature is supported automatically by the unique XIV caching algorithm which dynamically adapts to detected I O request patterns Ordering information You can order a system that has SSDs already installed which applies to fully or partially populated racks You can also order SSDs for a Gen3 system that is already installed Important SSD installation
49. m s described in this publication at any time without notice Any references in this information to non IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you Information concerning non IBM products was obtained from the suppliers of those products their published announcements or other publicly available sources IBM has not tested those products and cannot confirm the accuracy of performance compatibility or any other claims related to non IBM products Questions on the capabilities of non IBM products should be addressed to the suppliers of those products This information contains examples of data and reports used in daily business operations To illustrate them as completely as possible the examples include the names of individuals companies brands and products All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental COPYRIGHT LICENSE This information contains sample application programs in source language which illustrate programming techniques on various operating platforms You may copy modify and distribute these sample p
50. m to replay the collected traces from the customer s XIV system The replay program is used to issue the I Os as recorded in the log and within the same time frame on an XIV system that has the same capacity as the one on which the logs were recorded When the replay is done the IBM service representative can use the XIV statistics monitor to see how the workload performed Figure 4 11 shows a comparison of the original that is without SSDs read cache hits the predicted read cache hits and the actual read cache hits replay when the system is equipped with SSD Cache hit rate comparison v original 84 replay prediction 74 percent 12 10 12 35 13 00 1320 13 45 14 05 14 30 14 50 15 15 15 40 16 00 Figure 4 11 Predicting SSD cache hits compared to actual replay 28 Solid State Drive Caching in the IBM XIV Storage System 4 2 3 Disk Magic Disk Magic now supports modeling the effect of an SSD read cache in XIV Gen3 configurations as of Disk Magic v9 2 0 Automatic cache modeling also provides an estimation of the expected SSD cache hit rate based on an existing workload By using the existing workload and the input or estimated SSD cache hit rate Disk Magic can predict the expected benefit to response time and throughput in any XIV configuration when SSD read cache is added Figure 4 12 i Disk Magic untitled dm2 elg amp File View Options Window Help O ca i ene i le lgs 20 lt leefez u
51. made to the XIV GUI and the XCLI to support SSD Caching Finally this paper provides a listing of the new alert generating events and monitoring options that are provided for SSD support This paper is intended for users who want an insight into the XIV SSD Caching implementation and architecture its capabilities and usage For more information about the IBM XIV Storage System see the IBM Redbooks publication IBM XIV Storage System Architecture Implementation and Usage SG24 7659 The team who wrote this paper This paper was produced by a team of specialists from around the world working at the International Technical Support Organization ITSO in San Jose CA Bertrand Dufrasne is an IBM Certified Consulting IT Specialist and Project Leader for IBM System Storage disk products at the ITOS in San Jose CA He has worked for IBM in various IT areas He has authored many IBM Redbooks publications and has developed and taught technical workshops Before joining the ITSO he worked for IBM Global Services as an Application Architect He holds a master degree in electrical engineering In Kyu Park is a Storage Technical Advisor and IT Specialist with the Advanced Technical Skills ATS team in for the IBM Growth Market Unit GMU He has 5 years of experience as a Storage Field Technical Sales Specialist in South Korea and covers storage products for the of financial and industrial sectors In August 201 1 he transferred to IBM China and jo
52. nterface nodes inodes The interface nodes are gateways to incoming host l Os An inode keeps records in memory for the last million I Os Solid State Drive Caching in the IBM XIV Storage System The statistics collections inodes support a special management command that extracts I O records from a specified recent time window This command is triggered by the I O collection client program The I O collection client program BT client runs on a PC that is connected to XIV through an Ethernet connection The I O collection program includes a daemon that polls the inodes periodically and collects new l Os The daemon also records them in a sorted fashion in a data log that is on the connected PC local disk for offline processing later The utility is a separate client program that is written in Java for portability Typically an IBM representative or IBM Technical Advisor can run it from their laptop at a customer site if needed Visualization and cache prediction The client program also includes a visualization utility The visualization utility graphically shows the moving hit window for a host volume that is defined on the XIV system Figure 4 10 By using the l O traces the visualization program calculates a coarse grained heat map whose granularity is 1 GB per 5 minutes UID 103176 Heat map legend Name inn01dt 24 aue a 0 05 27 27 30 30 41 Size GB 315 oy Reads i FAccessed GB 212 20 094 e High GB 466 80 20
53. oftware naturally optimizes how to service the host l Os 12 Solid State Drive Caching in the IBM XIV Storage System 3 2 1 Overview of SSD Caching SSDs are integrated into the XIV system as an extension of the dynamic random access memory DRAM primary cache Each module has 24 GB of DRAM cache and 400 GB of SSD cache In a 15 module configuration the caching layer can be expanded by 6 TB of usable read cache XIV SSD Caching is implemented at the lowest level of the caching mechanism just before the I O is issued to the disk Tip Cache data movement is inherent to the architecture of the XIV The caching layer is extended naturally by the introduction of the SSDs Figure 3 3 illustrates a cache implementation in XIV SSD mapped as Allocates SSD slots an extension of E M DEM C dynamically according to memory the detected workload patterns Data movement is inherent to the architecture of the system and happens naturally SAS DISKS Figure 3 8 XIV SSD extended caching architecture The SSD extended cache is targeted exclusively at improving random read performance The XIV system software firmware does not allow a host write request to go directly to an SSD cache All write I Os are staged and mirrored in the main cache only DRAM Important SSDs are used exclusively as a cache extension for random read l Os Host writes are not cached in SSDs When a host sends a write request to the system the XIV s
54. or disk failure The module with the failed SSD remains the primary module for its partition However because of the reduced cache the module limits its caching duties The degraded module continues to serve reads from its DRAM cache and to serve large sequential reads from disk All small read misses are redirected to the secondary modules which populate their SSD cache This behavior distributes the extended caching duties of the module with the failed SSD to all other modules balancing the use of the remaining extended cache across all modules For more information see also 7 2 Alerts on page 53 3 2 5 SAS disk rebuild 18 During a rebuild after a SAS disk failure in a module the data on an SSD in that module is not invalidated Rather it is gradually updated to contain the new data blocks similar to the way that the DRAM does Solid State Drive Caching in the IBM XIV Storage System Workload characteristics Solid state drive SSD Caching is an optional feature for IBM XIV Storage System Gen3 This chapter highlights the characteristics of workloads that can benefit from SSD Caching With this information you can determine whether your critical application workloads can benefit from SSD Caching on the XIV system The chapter also highlights the following tools and features that can help in making the determination gt XIV system statistics gt XIV block tracing facility gt Disk Magic This chapter includes t
55. orm optional actions such as the following examples gt Check the SSD status in each module gt Enable or disable SSD Caching at the system level gt Enable or disable SSD Caching at the volume level 6 1 1 Viewing and managing the SSD by using the GUI To view and manage the SSDs the XIV Storage Management GUI Version 3 1 is required You can check the health and the status of the SSDs in the main system view When you move the cursor over a module a window is displayed that shows the temperature and status of major module components When SSDs are present the SSD status is displayed at the bottom of the window If the SSD is operational a green OK status is displayed as shown in Figure 6 1 xiv XIV Storage Management oie 2 File View Tools Help 2 O j Settings MJ Launch xcii Bl Launch xivTop itso XIV 1310039 C System System Time 05 45 pm Q Ju Module 5 E g3 0 interface Temperature 29 C gt g Status OK i Data Service OK Interface Service OK Remote Service OK PSUs OK Fans OK SSD OK Figure 6 1 Module status with SSD 38 Solid State Drive Caching in the IBM XIV Storage System If SSDs are not installed the window does not include the SSD label at the bottom as shown in Figure 6 2 Module 8 p10hw_iscsi Temperature 12 C Status OK Data Service OK Interface Service OK Remote Service OK PSUs OK Fans OK Figure 6 2 Module
56. ortant By design any read larger than 64k bypasses the SSD cache Because the ratio of SSD to SAS drives is 1 to 12 in a module it is more efficient to serve large blocks from the spinning SAS drives given their better overall throughput Chapter 3 Caching architecture and implementation 15 16 Log structured write Two 512 KB buffers are available When a buffer is full it is destaged to an SSD as a log structured write which is illustrated in Figure 3 5 Data destage SSD are written to by using log structured writes Write Buffers in DRAM x 512 KB ias i SSD Map data structure 912 KB SSD cache mapis a hash data structure Buffers in DRAM Disks page location are keys that correlate to values On a read the location or key is used to locate the corresponding value or SSD page Figure 3 5 Log structured writes A log structured write is an IBM XIV patented algorithm that allows buffered I Os to be destaged sequentially to SSD This operation minimizes the effects on SSDs of wear leveling and garbage collection algorithms Random read prefetch XIV is known for aggressive sequential prefetch capabilities By isolating a cache node to service only the disks in its respective module the cache node can be aggressive when doing a prefetch This aggressive prefetch capability is extended to random l Os with the introduction of SSD Caching When a random read miss is detected the cac
57. performance when read l Os are served from SSDs freeing some cycles on the SAS drives 3 2 2 Selective read cache algorithm for extended cache 14 Modules interface or data are the basic building blocks of the XIV system architecture and are responsible for storing a portion of the overall data set They manage disk drives and handle all storage features including snapshots caching and prefetching Each module interface and data in the XIV system incorporates a cache node Cache nodes allocate SSD pages when the appropriate workload patterns are detected Application processing typically generates random I Os and sequential I Os The SSD cache mechanism is triggered only for random reads Caching operations are split into two groups main and extended The extended cache handles the caching of random read miss operations of less than 64 KB All sequential larger than 64 KB operations read prefetches are handled in the main DRAM cache All host writes are handled in the main cache and destaged directly to disk For information about internal writes to the SSDs that are inherent to the cache staging mechanism see Updates to SSD cached data on page 17 Solid State Drive Caching in the IBM XIV Storage System 3 2 3 Random reads with SSD Caching enabled An SSD cache map is built as read misses occur in DRAM cache which is known as SSD cache learning and is illustrated in Figure 3 4 The cache node immediately checks the extended
58. ps Residencies run from two to six weeks in length and you can participate either in person or as a remote resident working from your home base Find out more about the residency program browse the residency index and apply online at ibm com redbooks residencies html viii Solid State Drive Caching in the IBM XIV Storage System Comments welcome Your comments are important to us We want our papers to be as helpful as possible Send us your comments about this paper or other IBM Redbooks publications in one of the following ways gt Use the online Contact us review Redbooks form found at ibm com redbooks gt Send your comments in an email to redbooks us ibm com gt Mail your comments to IBM Corporation International Technical Support Organization Dept HYTD Mail Station P099 2455 South Road Poughkeepsie NY 12601 5400 Stay connected to IBM Redbooks gt Find us on Facebook http www facebook com IBMRedbooks gt Follow us on Twitter http twitter com ibmredbooks gt Look for us on LinkedIn http www 1inkedin com groups home amp gid 2130806 gt Explore new Redbooks publications residencies and workshops with the IBM Redbooks weekly newsletter https www redbooks ibm com Redbooks nsf subscribe 0penForm gt Stay current on recent Redbooks publications with RSS Feeds http www redbooks ibm com rss html Preface Ix X Solid State Drive Caching in the IBM XIV Storage System Executive summ
59. receives the most benefit from SSD Caching This optimization happens automatically However if needed it is possible to enable or disable SSD Caching on a per volume basis If a specific host requires better performance for a random read workload the volumes associated with that host can be enabled when other volumes are disabled This action effectively provides all the SSD cache for the enabled hosts For more information see 6 1 2 Setting SSD Caching on page 40 Chapter 4 Workload characteristics 21 4 1 2 Typical applications and performance benefits SSD Caching offers performance benefits on several typical applications IBM DB2 brokerage applications Securities trading is an OLTP workload that generates random I O and benefits from increased IOPs Figure 4 2 illustrates that SSD Caching for the XIV system was tested with this workload and an increase of three times was possible in IOPS DB2 Brokerage IOPS Securities trading OLTP Workload gt Mixed block I O gt 12 TB database XIV ena AIV sens wiSSD Figure 4 2 IBM DB20 Brokerage application IBM WebSphere data store Web access of data is another example of an OLTP random workload This application benefits from the addition of SSD Caching Measurements show an increase of 2 times in IOPS as illustrated in Figure 4 3 WebSphere Data Store IOPS Web 2 0 OLTP Workload gt 80 20 4k gt 20 TB database AIV ten XIV Gens wies
60. rograms in any form without payment to IBM for the purposes of developing using marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written These examples have not been thoroughly tested under all conditions IBM therefore cannot guarantee or imply reliability serviceability or function of these programs O Copyright IBM Corp 2012 All rights reserved V Trademarks IBM the IBM logo and ibm com are trademarks or registered trademarks of International Business Machines Corporation in the United States other countries or both These and other IBM trademarked terms are marked on their first occurrence in this information with the appropriate symbol or indicating US registered or common law trademarks owned by IBM at the time this information was published Such trademarks may also be registered or common law trademarks in other countries A current list of IBM trademarks is available on the Web at http www ibm com 1legal copytrade shtml The following terms are trademarks of the International Business Machines Corporation in the United States other countries or both DB20 Redbooks logo 40 WebSphere IBM S 390 XIV Redbooks Storwize Redpaper System Storage The following terms are trademarks of other companies Linux is a trademark of Linus Torvalds in the United States other countries or both
61. rp 2012 All rights reserved 51 7 1 Monitoring SSDs In version 3 1 of the XIV Storage Management GUI and XIV command line interface XCLI you can view specific performance information about installed SSDs You can also differentiate between DRAM cache parameters and SSD cache parameters Figure 7 1 shows the Mem Hit and SSD Hit parameters XIV Storage Management lso File View Tools Help 9 admin All System View Ey My Groups ILL Statistics System Time 02 12 pm All Interfaces olo k A fh a en eZeraad Hit amp Miss Memory Hit amp SSD Hit Y Read 2000 1500 onl fy 7000 EE 6500 B sone AM 5500 i 3000 LJ 4500 i LA SSD Hit IOPS 1171 3300 1 Memory Hilt MOPS 5865 2004 Miss TOPs 131 gt a Hit OPS T036 2500 i PRA H 2000 1500 i 1000 08 F 11 00 1110 1120 1139 11 40 1150 1200 1210 1220 1230 12 40 1250 11430 13 10 1320 1130 13 40 1350 1400 1410 mE 5 Jan 2012 O 64512108 uops Hour Week Year B P08 KB C 9812 KB C Latency aia AE E o e M Es wn n a VBH 8 Custom BE FoU a NE E S m fare es TT total ED Figure 7 1 Mem Hit and SSD Hit parameters Hegardless of the other parameters that are being examined the Mem Hit and SSD Hit parameters are valid only for read data You can enable or disable SSD Caching at the system level or on a volume by volume ba
62. se blocks LEBs when erasing data An erase block is the smallest unit that a NAND flash can erase The XIV software controls writes to the SSD cache and avoids overwriting a particular spot on the SSD too often Writing to the SSD cache is sequential Data writing starts at the beginning of the SSD first memory page and then progresses sequentially to the end without skipping or jumping as illustrated in Figure 2 2 Each sequential write consists of a 512K block When the end of the SSD is reached a new cycle begins overwriting the previous data again from beginning to end This sequential and cyclic writing allows for uniform use of the SSD cells maximizing the SSD lifetime SSD Write Figure 2 2 SSD written to sequentially Furthermore because XIV does only 512K sequential writes to SSD or 4K random reads it gets a consistent and predictable response time from those SSD operations XIV understands the performance to expect from the SSDs at any time This feature is another advantage over systems that do any combination of random reads and writes and sequential reads and writes to SSD making it impossible for the system to have consistent performance characteristics from the SSDs Because XIV uses SSD exclusively as an extension of the read cache layer all data on the SSD is written to disk This way XIV can sustain multiple SSD failures without any implication of data loss For information about the SSD Caching algorithm
63. sis To see the status of the SSD Caching for each volume when using the XIV GUI follow the steps in 6 1 2 Setting SSD Caching on page 40 Alternatively when using the XCLI see 6 2 SSD Caching management with XCLI on page 47 52 Solid State Drive Caching in the IBM XIV Storage System 7 2 Alerts With adding SSDs comes the requirement to generate alerts about those SSDs Starting with version 11 1 0 the XIV system software adds the following events about SSDs COMPONENT TEST OF SSD HAS FAILED A major event that is seen only when IBM service personnel are working on the XIV SSD CACHING DISABLED An informational event that is generated when IBM service personnel issue the command to no longer use SSDs SSD CACHING ENABLED An informational event that is generated when IBM service personnel issue the command to start using SSDs SSD COMPONENT TEST STARTED An informational event that is seen only when IBM service personnel are working on the XIV SSD HAS FAILED A major event that is issued when the module detects that the SSD is no longer functioning VOLUME SET DEFAULT SSD CACHING An informational event that is issued when the default state for all volumes is changed The Description column shows if the action taken was to set the default to enabled or disabled VOLUME SET SSD CACHING An informational event that is generated when the status of a single volume is changed The Description column shows if the action t
64. ssions with random reads versus backups with large block sequential bandwidth requirements Chapter 3 Caching architecture and implementation 11 Consideration of storage tiering as part of the solution then leads to another set of questions Whattiering policy should we implement gt Can we move our large data set to SATA drives in time to service our high bandwidth reads gt Can we move the relevant blocks back to SSD to facilitate our production workload which is a random OLTP workload How much SSD capacity do we need Dowe have enough budget for a large SSD tier The XIV architecture obsoletes all these considerations providing a solution that you can deploy quickly with minimal planning From inception the XIV design principles have dictated the use of low cost components coupled with virtualized and flexible system software firmware allowing the integration of emerging technologies The integration of SSD at the caching layer is another example of adhering to those XIV core values Traditional systems with a global cache that must support multiple disk layouts RAID levels and disk types SSD Fibre Channel FC SAS SATA make caching less nimble and less aggressive To support the many variations the cache must make concessions For example optimizing on RAID 6 might have implications on how much read cache is available to do a prefetch on RAID 1 Consider an example of an application with a highly ran
65. status without SSDs Clicking a module number in the main system view Figure 6 1 on page 38 opens a full perspective view of the module and its components Figure 6 3 xiv XIV Storage Management File view Tools Help y f Settings MY Launch xcii Rf Launch xivrop All Systems View By My Groups gt XIV 1310039 C gt System System Time 05 46pm Q SSD 1 Status OK Module 5 m Status Figure 6 8 SSD status From this view you can also check the SSD status If an SSD is installed place the mouse cursor over the SSD in the module view The SSD can have one of the following statuses Agreen status indicates that the SSD is OK and phased in gt Ayellow status indicates that the SSD is phased out gt Ared status indicates that the SSD failed Chapter 6 GUI and XCLI support for solid state drives 39 6 1 2 Setting SSD Caching When the IBM service representative initially enables SSD Caching after installation SSD Caching becomes effective and enabled for all volumes that are defined in the XIV system New volumes that are created afterward by default also have caching enabled The storage administrator can change the default status to disabled or to specifically enable or disable SSD Caching at the volume level You can manually change SSD Caching state for specific volumes from the Volumes and Snapshots view Tip You can dynamically enable or disable SSD Caching at any time Setting SSD
66. tus verify After the SSDs are enabled you do not need any additional software or tuning considerations for efficient SSD utilization The SSD Caching feature is supported automatically by the unique XIV caching algorithm which dynamically adapts to detected l O request patterns 34 Solid State Drive Caching in the IBM XIV Storage System 5 3 Upgrading SSD after a module upgrade SSDs can also be installed on partially populated XIV system with fewer than 15 modules The only constraint is that SSDs must be installed for every module in the rack If you decide to order additional modules to complement a partially populated rack that is already configured with SSDs you must also order SSDs for the additional modules For example consider a situation where you have a 6 module XIV Gen3 System already equipped with SSDs Then you order three additional modules with SSDs The IBM service representative first installs the additional modules in the rack When the modules are phased in the XIV system starts a redistribution procedure After the redistribution procedure is finished the additional modules show an OK status in green but the SSD status is displayed as Ready not OK as illustrated in Figure 5 4 SS xiv Storage Management mj x File View Tools Help y O G Settings Launch XCLI technician XIV Fest 7820 System System Time 01 31 pm Q Module 9 93 0 interface Temperature 30 C Status OK Data
67. umes and Snapshots O Size GR __ LET IGRI Cons A Resize Delete Format Rename Add To Consistency Group Remove From Consistency Group Move To Consistency Group Move to Pool Create Snapshot Create Snapshot Advanced Overwrite Snapshot Copy this Volume Restore Lock Unlock Change SSD Caghing State Create Mirror Create Mirrored Snapshot Map selected volumes Map selected volumes manually View Volume Mapping Properties Continuous Volumes Selection Chapter 6 GUI and XCLI support for solid state drives Create a Consistency Group With Selected Volumes 45 gt To select specific volumes hold down the Ctrl key and click one volume at a time on as many volumes you want to include in your selection See Figure 6 14 le View Tools Help 5 o Tp Add Volumes Systems View By My Groups gt XIV 1310039 C Volumes and Snapshots Name Size GB sed GR onsi DOO Cr Resize Res Fra Vol 02 17 GB Delete Format Rename EB h 5 Ly a Create a Consistency Group With Selected Volumes ye m abby_06 51 GB Add To Consistency Group cal amp asimov_121 51 GB Remove From Consistency Group abby 08 51 GB Move To Consistency Group Move to Pool S E amp asimov_031 Create Snapshot 5 Create Snapshot Advanced KJ amp Overwrite Snapshot gt b asimov 105 51 GB Copy this Volume amp abby 12 51GB Restore
68. uring this write operation the updated main memory pages DRAM are copied asynchronously into the SSD buffers and staged for a log structured write to the SSD After the l Os are updated on the SSD the hash table entry SSD data map structure is revalidated and then subsequent extended cache hits can occur This mechanism prevents the extended cache from serving stale data When the SSD layer is being updated if a read request for the same block is received it is serviced from the main DRAM cache The pages in the main cache are not dropped until the update to the SSD effectively occurs see Figure 3 6 Write Operation Incoming write D All writes go to DRAM Main Cache l D If SSD has a copy of the data being modified then the data in SSD is old and Main Cache gt Mirror write to stale and a corresponding hash table DRAM secondary DRAM and subsequent hits from the extended cache tohost Writes are de staged from DRAM to disk normally and asynchronously 4 As writes are de staged from Main Cache to disk they are asynchronously staged on to the SSD buffer when less or equal to 64KB 5 Once buffer is full data is written to SSD and hash table is updated Extended Cache Figure 3 6 Cache updates after a disk write Write performance To deliver high performance write operations the XIV system uses the following functions or features that are inherent to its architecture Write operations to X
69. ware version 11 1 0 or later In the XIV system SSDs are used to increase the read cache capacity of the existing DRAM memory cache and not used for persistent storage This paper begins with a high level overview of the SSD implementation in XIV and a brief review of the SSD technology with focus on the XIV storage system It explains the SSD Caching design and implementation in XIV Then it examines typical workloads that can benefit from the SSD Caching extension and introduces the tools and utilities to help you analyze and understand the workload In particular it highlights the block tracing facility that was designed and developed by IBM Research Then this paper explains the process that authorized IBM services representatives use to install SSD Caching It reviews the changes made to the XIV GUI and the XCLI to support SSD Caching Finally this paper provides a listing of the new alert generating events and monitoring options that are provided for SSD support This paper is intended for users who want an insight into the XIV SSD Caching implementation and architecture its capabilities and usage For more information about the XIV Storage System see the IBM Redbooks publication BM XIV Storage System Architecture Implementation and Usage SG24 7659 REDP 4842 00 Redpaper INTERNATIONAL TECHNICAL SUPPORT ORGANIZATION BUILDING TECHNICAL INFORMATION BASED ON PRACTICAL EXPERIENCE IBM Redbooks are developed by t
70. ystem responds in the following way 1 Any of the interface modules that are connected to the host can service the request 2 The interface modules use the system configuration information metadata to determine the location of the primary module data module or interface module that houses the referenced data The data is written to the local DRAM cache of the primary module Chapter 3 Caching architecture and implementation 13 3 The primary module uses the system configuration information metadata to determine the location of the secondary module that houses the copy of the referenced data Again it can be a data module or an interface module The data is redundantly written to the local DRAM cache of the secondary module Additional information about writes A write is still acknowledged only after it is committed to the main cache of the secondary cache node The writes are flushed from the main cache to the disk layer as part of the normal write destage Destaging to SSD or SAS disks is done asynchronously so that the system does not have to wait for the SSDs or the SAS disks upon a write request Therefore when host writes are destaged to the SAS disks they might be cached in parallel in the SSD cache to serve future reads write through caching By using the extended cache for small block random reads only SSD Caching attempts to greatly reduce the random read access from the disks layer Therefore XIV obtains better write

Solid State Drive Caching in the IBM XIV Storage

Contents

Download Pdf Manuals

Related Search

Related Contents