Home

Oracle Exadata Database Machine Extending and Multi

1. Leaf Switch Connection Cable Length R5 IB3 within Rack 5 R5 IB3 P8A to R5 U1 P3A 3 meters R5 IB3 to Rack 1 R5 IB3 P10A to R1 U1 P7A 10 meters R5 IB3 to Rack 2 R5 IB3 P10B to R2 U1 P8A 10 meters R5 IB3 to Rack 3 R5 IB3 P11A to R3 U1 P9A 5 meters R5 IB3 to Rack 4 R5 IB3 P11B to R4 U1 P10A 5 meters R5 IB3 to Rack 6 R5 IB3 P8B to R6 U1 P4A 5 meters R5 IB3 to Rack 7 R5 IB3 P9A to R7 U1 P5A 5 meters R5 IB3 to Rack 8 R5 IB3 P9B to R8 U1 P6A 10 meters R5 IB2 within Rack 5 R5 IB2 P8A to R5 U1 P3B 3 meters Multi Rack Cabling Tables 2 31 Eight Rack Cabling Table 2 32 Cont Leaf Switch Connections for the Fifth Rack in a Eight Rack System Table 2 33 shows the cable connections for the sixth spine switch R6 U1 when cabling eight full racks together Table 2 33 Leaf Switch Connections for the Sixth Rack in a Eight Rack System Leaf Switch Connection Cable Length R5 IB2 to Rack 1 R5 IB2 P10A to R1 U1 P7B 10 meters R5 IB2 to Rack 2 R5 IB2 P10B to R2 U1 P8B 10 meters R5 IB2 to Rack 3 R5 IB2 P11A to R3 U1 P9B 5 meters R5 IB2 to Rack 4 R5 IB2 P11B to R4 U1 P10B 5 meters R5 IB2 to Rack 6 R5 IB2 P8B to R6 U1 P4B 5 meters R5 IB2 to Rack 7 R5 IB2 P9A to R7 U1 P5B 5 meters R5 IB2 to Rack 8 R5 IB2 P9B to R8 U1 P6B 10 meters Table 2 34 shows the cable connections for the seventh spine switch R7 U1 when cabling eight full racks together
2. 2 28 Leaf Switch Connections for the First Rack in a Eight Rack System 2 29 Leaf Switch Connections for the Second Rack in a Eight Rack System 2 30 Leaf Switch Connections for the Third Rack in a Eight Rack System 2 30 Leaf Switch Connections for the Fourth Rack in a Eight Rack System 2 31 Leaf Switch Connections for the Fifth Rack in a Eight Rack System 2 31 Leaf Switch Connections for the Sixth Rack in a Eight Rack System 2 32 Leaf Switch Connections for the Seventh Rack in a Eight Rack System 2 32 Leaf Switch Connections for the Eighth Rack in a Eight Rack System 2 33 Leaf Switch Connections for the Ninth Rack in the System 2 34 Leaf Switch Connections for the Tenth Rack in the System 2 35 Leaf Switch Connections for the Eleventh Rack in the System 2 36 Leaf Switch Connections for the Twelfth Rack in the System 2 37 Leaf Switch Connections for the Thirteenth Rack in the System 2 38 Leaf Switch Connections for the Fourteenth Rack in the System 2 40 Leaf Switch Connections for the Fifteenth Rack in the System
3. 2 41 Leaf Switch Connections for the Sixteenth Rack in the System 2 42 Leaf Switch Connections for the Seventeenth Rack in the System 2 43 Leaf Switch Connections for the Eighteenth Rack in the System 2 44 List of Figures 1 1 Server ComponentistorUpofide carrellata Ise Removing th Rack DOTS scenica a J Removing the Piller Panels rron u l u EA NaN vii vill Audience Preface This guide describes how to extend and cable Oracle Exadata Database Machine It includes information about the cables new server installation and cabling tables This preface contains the following topics Audience a Documentation Accessibility a Related Documentation Conventions This guide is intended for Oracle Exadata Database Machine customers and those responsible for data center site planning configuration and maintenance of Oracle Exadata Database Machine Documentation Accessibility For information about Oracle s commitment to accessibility visit the Oracle Accessibility Program website at http www oracle com pls topic lookup ctx acc amp 1id docacc Access to Oracle Support Oracle customers that have purchased support have access to electronic support through My Oracle Support For information visit http www oracle com pls topic lookup ctx acc amp id info or visit http www oracle com pls topic l
4. R2 IB2 to Rack 1 R2 IB2 to Rack 3 R2 IB2 P11A to R1 U1 P9B R2 IB2 P11B to R1 U1 P10B R2 IB2 P9B to R3 U1 P6B R2 IB2 P10A to R3 U1 P7B R2 IB2 P10B to R3 U1 P8B 5 meters 5 meters Table 2 5 shows the cable connections for the third spine switch R3 U1 when cabling three full racks together Table 2 5 Leaf Switch Connections for the Third Rack in a Three Rack System Leaf Switch R3 IB3 within Rack 3 R3 IB3 to Rack 1 Connection R3 IB3 P8A to R3 U1 P3A R3 IB3 P8B to R3 U1 P4A R3 IB3 P9A to R3 U1 P5A R3 IB3 P9B to R1 U1 P6A R3 IB3 P10A to R1 U1 P7A R3 IB3 P10B to R1 U1 P8A Cable Length 3 meters 5 meters R3 IB3 to Rack 2 R3 IB3 P11A to R2 U1 P9A R3 IB3 P11B to R2 U1 P10A 5 meters R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B R3 IB2 P8B to R3 U1 P4B R3 IB2 P9A to R3 U1 P5B 3 meters Multi Rack Cabling Tables 2 13 Four Rack Cabling Table 2 5 Cont Leaf Switch Connections for the Third Rack in a Three Rack System Four Rack Cabling Leaf Switch Connection Cable Length R3 IB2 to Rack 1 R3 IB2 P9B to R1 U1 P6B 5 meters R3 IB2 P10A to R1 U1 P7B R3 IB2 P10B to R1 U1 P8B R3 IB2 to Rack 2 R3 IB2 P11A to R2 U1 P9B 5 meters R3 IB2 P11B to R2 U1 P10B Table 2 6 shows the cable connections for the first spine switch R1 U1 when cabling four racks together Notes The spine switch is in U1 also referred to as IB1 for all racks
5. 0 DBFS_DG_CD_0 6 9 0 DBFS_DG_CD_1 0 1 REBALANCE POWER 32 The preceding commands return Diskgroup altered if successful Optional Monitor the current rebalance operation using the following command SQL gt SELECT FROM gvSasm_operation Re enable the appliance mode attribute if it was disabled in step 6 using the following commands SQL gt ALTER DISKGROUP data_diskgroup set attribute appliance mode TRUE SOL gt ALTER DISKGROUP recodiskgroup set attribute appliance mode TRUE SQL gt ALTER DISKGROUP dbfs_dg_diskgroup set attribute appliance mode TRUE Validating New Oracle Exadata Database Machine X3 2 Half Rack or Later Configuration After adding the grid disks to the Oracle ASM disk groups validate the configuration The following procedure describes how to validate the configuration 1 2 Log in as the root user on the first database server Check the core count using the following command deli g db _ group l root dbmcli e list dbserver attributes coreCount Review the storage server configuration using the following command dcli g cell group 1 celladmin cellcli e list cell attributes eighthrack The output should show FALSE Review the appliance mode for each disk group using the following commands SQL gt set lines 100 SQL gt column attribute format a20 SQL gt column value format a20 SQL gt column diskgroup format a20 SQL gt SELECT att name attribute upp
6. Table 2 34 Leaf Switch Connections for the Seventh Rack in a Eight Rack System Leaf Switch Connection Cable Length R6 IB3 within Rack 6 R6 IB3 P8A to R6 U1 P3A 3 meters R6 IB3 to Rack 1 R6 IB3 P9B to R1 U1 P6A 10 meters R6 IB3 to Rack 2 R6 IB3 P10A to R2 U1 P7A 10 meters R6 IB3 to Rack 3 R6 IB3 P10B to R3 U1 P8A 10 meters R6 IB3 to Rack 4 R6 IB3 P11A to R4 U1 P9A 5 meters R6 IB3 to Rack 5 R6 IB3 P11B to R5 U1 P10A 5 meters R6 IB3 to Rack 7 R6 IB3 P8B to R7 U1 P4A 5 meters R6 IB3 to Rack 8 R6 IB3 P9A to R8 U1 P5A 5 meters R6 IB2 within Rack 6 R6 IB2 P8A to R6 U1 P3B 3 meters R6 IB2 to Rack 1 R6 IB2 P9B to R1 U1 P6B 10 meters R6 IB2 to Rack 2 R6 IB3 P10A to R2 U1 P7B 10 meters R6 IB2 to Rack 3 R6 IB2 P10B to R3 U1 P8B 10 meters R6 IB2 to Rack 4 R6 IB2 P11A to R4 U1 P9B 5 meters R6 IB2 to Rack 5 R6 IB2 P11B to R5 U1 P10B 5 meters R6 IB2 to Rack 7 R6 IB2 P8B to R7 U1 P4B 5 meters R6 IB2 to Rack 8 R6 IB2 P9A to R8 U1 P5B 5 meters Leaf Switch Connection Cable Length R7 IB3 within Rack 7 R7 IB3 P8A to R7 U1 P3A 3 meters R7 IB3 to Rack 1 R7 IB3 P9A to R1 U1 P5A 10 meters R7 IB3 to Rack 2 R7 IB3 P9B to R2 U1 P6A 10 meters R7 IB3 to Rack 3 R7 IB3 P10A to R3 U1 P7A 10 meters Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Nine Rack Cabling Table 2 34 Cont Leaf Switch Connections for the Seventh Rack in a Eight Rack Leaf Switch Connection Cable Le
7. e Update the group files with existing and new servers using the following commands cat root new_group_files dbs_group gt gt root group_files dbs_group cat root new_group_files cell_group gt gt root group_files cell_group cat root new_group_files all_ group gt gt root group_files all_group 1 46 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware cat root new_group_files dbs_ib group gt gt root group_files dbs_ib group cat root new_group_files cell_ib group gt gt root group_files cell_ib group cat root new_group_files all_ib_ group gt gt root group_files all_ib_group f Make the updated group files the default group files using the following commands The updated group files contain the existing and new servers cp root group_files root cp root group_files opt oracle SupportTools onecommand g Putacopy of the updated group files in the root user oracle user and Grid Infrastructure user home directories and ensure that the files are owned by the respective users Modify the etc hosts file on the existing and new database server to include the existing InfiniBand IP addresses for the database servers and Exadata Storage Servers The existing and new priv_ib_hosts files can be used for this step Note Do not copy the etc hosts file from one server to the other servers Edit each host s file Run the setssh Linux sh script as
8. Leaf Switch Connection Cable Length R3 IB2 to Rack 7 R3 IB2 P10A to R7 U1 P7B 10 meters R3 IB2 to Rack 8 R3 IB2 P10B to R8 U1 P8B 10 meters Table 2 31 shows the cable connections for the fourth spine switch R4 U1 when cabling eight full racks together Table 2 31 Leaf Switch Connections for the Fourth Rack in a Eight Rack System Leaf Switch Connection Cable Length R4 IB3 within Rack 4 R4 IB3 P8A to R4 U1 P3A 3 meters R4 IB3 to Rack 1 R4 IB3 P10B to R1 U1 P8A 10 meters R4 IB3 to Rack 2 R4 IB3 P11A to R2 U1 P9A 5 meters R4 IB3 to Rack 3 R4 IB3 P11B to R3 U1 P10A 5 meters R4 IB3 to Rack 5 R4 IB3 P8B to R5 U1 P4A 5 meters R4 IB3 to Rack 6 R4 IB3 P9A to R6 U1 P5A 5 meters R4 IB3 to Rack 7 R4 IB3 P9B to R7 U1 P6A 10 meters R4 IB3 to Rack 8 R4 IB3 P10A to R8 U1 P7A 10 meters R4 IB2 within Rack 4 R4 IB2 P8A to R4 U1 P3B 3 meters R4 IB2 to Rack 1 R4 IB2 P10B to R1 U1 P8B 10 meters R4 IB2 to Rack 2 R4 IB2 P11A to R2 U1 P9B 5 meters R4 IB2 to Rack 3 R4 IB2 P11B to R3 U1 P10B 5 meters R4 IB2 to Rack 5 R4 IB2 P8B to R5 U1 P4B 5 meters R4 IB2 to Rack 6 R4 IB2 P9A to R6 U1 P5B 5 meters R4 IB2 to Rack 7 R4 IB2 P9B to R7 U1 P6B 10 meters R4 IB2 to Rack 8 R4 IB2 P10A to R8 U1 P7B 10 meters Table 2 32 shows the cable connections for the fifth spine switch R5 U1 when cabling eight full racks together Table 2 32 Leaf Switch Connections for the Fifth Rack in a Eight Rack System
9. 5 meters R3 IB3 to Rack 4 R3 IB3 P9A to R4 U1 P5A 5 meters R3 IB3 P9B to R4 U1 P6A R3 IB3 to Rack 5 R3 IB3 P10A to R5 U1 P7A 5 meters R3 IB3 P10B to R5 U1 P8A R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B 3 meters R3 IB2 P8B to R3 U1 P4B R3 IB2 to Rack 1 R3 IB2 P11A to R1 U1 P9B 5 meters R3 IB2 to Rack 2 R3 IB2 P11B to R2 U1 P10B 5 meters R3 IB2 to Rack 4 R3 IB2 P9A to R4 U1 P5B 5 meters R3 IB2 P9B to R4 U1 P6B 2 18 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Five Rack Cabling Table 2 12 Cont Leaf Switch Connections for the Third Rack in a Five Rack System Leaf Switch R3 IB2 to Rack 5 Connection R3 IB2 P10A to R5 U1 P7B R3 IB2 P10B to R5 U1 P8B Cable Length 5 meters Table 2 13 shows the cable connections for the fourth spine switch R4 U1 when cabling five full racks together Table 2 13 Leaf Switch Connections for the Fourth Rack in a Five Rack System Leaf Switch Connection Cable Length R4 IB3 within Rack 4 R4 IB3 P8A to R4 U1 P3A 3 meters R4 IB3 P8B to R4 U1 P4A R4 IB3 to Rack 1 R4 IB3 P10A to R1 U1 P7A 10 meters R4 IB3 P10B to R1 U1 P8A R4 IB3 to Rack 2 R4 IB3 P11A to R2 U1 P9A 5 meters R4 IB3 to Rack 3 R4 IB3 P11B to R3 U1 P10A 5 meters R4 IB3 to Rack 5 R4 IB3 P9A to R5 U1 P5A 5 meters R4 IB3 P9B to R5 U1 P6A R4 IB2 within Rack 4 R4 IB2 P8A to R4 U1 P3B 3 meters R4 IB2 P8B to R4 U1 P4B R4 IB2 to Rack 1 R4 IB
10. Adding Grid Disks to Oracle ASM Disk Groups in Oracle Exadata Database Machine X3 2 Eighth Rack The grid disks created in Creating Grid Disks in Oracle Exadata Database Machine X3 2 Eighth Rack must be added as Oracle ASM disks to their corresponding existing Oracle ASM disk groups The following procedure describes how to add the grid disks to Oracle ASM disk groups 1 Validate the following No rebalance operation is currently running x All Oracle ASM disks are active 2 Log in to the first database server as the owner who runs the Grid Infrastructure software 3 Set the environment to access the ASM instance on the server 4 Login to the ASM instance as the sysasm user using the following command S sqlplus as sysasm 5 Validate the current settings as follows SQL gt set lines 100 SQL gt column attribute format a20 SQL gt column value format a20 SQL gt column diskgroup format a20 SQL gt SELECT att name attribute upper att value value dg name diskgroup N FROM VSASM ATTRIBUTE att VSASM DISKGROUP DG N WHERE DG group_number att group_number AND att name LIKE tappliance mode ORDER BY att group_number The output should be similar to the following ATTRIBUTE VALUE DISKGROUP appliance mode TRUE DATAC1 appliance mode TRUE DBFS_DG appliance mode TRUE RECOC1 6 Disable the appliance mode attribute for any disk group that shows TRUE using the following commands SQL gt ALTER DISKGROUP data
11. Check the status of the cluster after adding the servers using a command similar to the following S cluvfy stage post nodeadd n N dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 dm02db06 dm02db07 dm02db08 N verbose Check that all servers have been added and have basic services running using the following command from any server crsctl stat res t Note It may be necessary to mount disk groups on the new servers using commands similar to the following The commands must be run as the oracle user S srvctl start diskgroup g data S srvctl start diskgroup g reco Releases 11 2 0 2 and later Do the following procedure a Manually add the CLUSTER_INTERCONNECTS parameter to the spfile for each Oracle ASM instance using the following command alter system set cluster_interconnects 192 168 10 x sid ASMx scope spfile b Restart the cluster on each new server c Verify the parameters were set correctly Task 8 Configuring Cell Alerts for New Exadata Storage Servers Cell alerts need to be configured for the new Exadata Storage Servers The configuration depends on the type of installation as follows When extending Oracle Exadata Database Machine Quarter Rack to Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Half Rack to Oracle Exadata Database Machine Full Rack Extending Oracle Exadata Database Machine 1 55 Configuring the New Hardware Manually configure cell
12. In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 6 Leaf Switch Connections for the First Rack in a Four Rack System 2 14 Oracle Exadata Database Leaf Switch Connection Cable Length R1 IB3 within Rack 1 R1 IB3 P8A to R1 U1 P3A 3 meters R1 IB3 P8B to R1 U1 P4A R1 IB3 to Rack 2 R1 IB3 P9A to R2 U1 P5A 5 m
13. Install Oracle Exadata Deployment Assistant on the first new database server See Also My Oracle Support note 888828 1 for information about Oracle Exadata Deployment Assistant Copy the configuration files to the first database server of the new servers in the opt oracle SupportTools onecommand directory This is the information completed in step 2 Run Oracle Exadata Deployment Assistant up to but not including the CreateGridDisk step and then run the SetupCellEmailAlerts step and the Automatic Service Request steps Note m The Oracle Exadata Deployment Assistant ValidateEnv step may display an error message about missing files pXX zip This is expected behavior because the files are not used for this procedure You can ignore the error message w When using capacity on demand Oracle Exadata Deployment Assistant has the SetUpCapacityOnDemand step This step uses the resourcecontrol command to set up the cores correctly Configure the Exadata Storage Servers cell disks and grid disks as described in Configuring Cells Cell Disks and Grid Disks with CellCLI in chapter 2 of Oracle Exadata Storage Server Software User s Guide Note Use the data collected from the existing system as described in Obtaining Current Configuration Information on page 1 6 to determine the grid disk names and sizes Reclaim disk space as described in Oracle Exadata Database Machine Installation and Configuraton Guide of
14. Se zam ee OE ST 11 Port B Oracle Database Servers 1 2 3 4 Exadata Storage Servers 1 2 7 When connecting up to eight racks together remove the seven existing inter switch connections between each leaf switch as well as the two connections between the leaf switches and the spine switch From each leaf switch distribute eight connections over the spine switches in all racks In multi rack environments the leaf switches inside a rack are no longer directly interconnected as shown in the following graphic Rack 2 Leaf Switch 1 Leaf Switch 2 As shown in the preceding graphic each leaf switch in rack 1 connects to the following switches Four connections to its internal spine switch Four connections to the spine switch in rack 2 The spine switch in rack 1 connects to the following switches Eight connections to both internal leaf switches a Eight connections to both leaf switches in rack 2 As the number of racks increases from two to eight the pattern continues as shown in the following graphic Multi Rack Cabling Tables 2 3 Understanding Multi Rack Cabling Rack 1 Rack 2 Rack 8 Spine Switch Spine Switch i tch Spine Swi Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 As shown in the preceding graphic each leaf switch has eight inter switch connections distributed over all spine switches Each s
15. find u01 app 11 2 0 grid type d user gi_owner_name perm utx perm g x perm o x In the preceding commands gi_owner_name is the name of the Grid Infrastructure owner and u01 app 11 2 0 grid is the Grid Infrastructure home If any directories are listed then ensure the group and others permissions are x The GI_HOME network admin samples GI_HOME crf admin run crfmond and SGI_HOME crf admin run crflogd directories may need the x permissions set Run the following command It is assumed that the Grid Infrastructure home is owned by the Grid Infrastructure user S dcli g old_db_nodes 1 root chown f gi_owner_name dba N u01 app 11 2 0 grid OPatch ocm bin emocmrsp Create a response file add cluster nodes rsp as the Grid Infrastructure user to add the new servers similar to the following RESPONSEFILE VERSION 2 2 1 0 0 CLUSTER_NEW_NODES dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 dm02db06 dm02db07 dm02db08 CLUSTER_NEW_VIRTUAL_HOSTNAMES dm0201 vip dm0202 vip dm0203 vip dm0204 vip N dm0205 vip dm0206 vip dm0207 vip dm0208 vip In the preceding file the host names dm02db01 through db02db08 are the new nodes being added to the cluster Note The lines listing the server names should appear on one continuous line They are wrapped in the document due to page limitations Ensure most of the files in the SGRID_ HOME rdbms audit and SGRID_ HOME log diag directories have been moved or d
16. sN celldisk CD 11 _N celladmin cellcli e hostname s Size datasize create griddisk N hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s size recosize create griddisk hostname s size recosize create griddisk hostname s size recosize create griddisk hostname s size recosize create griddisk N hostname s size recosize create griddisk hostname s size recosize create griddisk DBFS DG CD 06 _ hostname s celldisk CD 06 hostname s size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk X DBFS_DG_CD_07_ hostname s celldisk CD_07_ hostname sN size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk DBFS_DG_CD_08_ hostname s celldisk CD_08_ hostname sN size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk N DBFS DG CD_09_ hostname sN celldisk CD_09_ hostname sN size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk N DBFS_DG_CD_10_ hostname sN celldisk CD_10_ hostname sN size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk N DBFS_DG_CD_11_ hostname sN celldisk CD_11_N hostname sN size dbfssize Extending Oracle Exadata Database Machine 1 21 Extending the Hardware
17. 2 18 Leaf Switch Connections for the Fourth Rack in a Five Rack System 2 19 Leaf Switch Connections for the Fifth Rack in a Five Rack System 2 19 Leaf Switch Connections for the First Rack in a Six Rack System 2 20 Leaf Switch Connections for the Second Rack in a Six Rack System 2 21 Leaf Switch Connections for the Third Rack in a Six Rack System 2 22 Leaf Switch Connections for the Fourth Rack in a Six Rack System 2 22 Leaf Switch Connections for the Fifth Rack in a Six Rack System 2 23 Leaf Switch Connections for the Sixth Rack in a Six Rack System 2 23 Leaf Switch Connections for the First Rack in a Seven Rack System 2 24 Leaf Switch Connections for the Second Rack in a Seven Rack System 2 25 Leaf Switch Connections for the Third Rack in a Seven Rack System 2 25 Leaf Switch Connections for the Fourth Rack in a Seven Rack System 2 26 Leaf Switch Connections for the Fifth Rack in a Seven Rack System 2 26 Leaf Switch Connections for the Sixth Rack in a Seven Rack System 2 27 Leaf Switch Connections for the Seventh Rack in a Seven Rack System
18. If you are extending Oracle Exadata Database Machine X5 2 Oracle Exadata Database Machine X4 2 Oracle Exadata Database Machine X3 8 Full Rack or Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers half rack then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch The new servers need to be configured manually when extending Oracle Exadata Database Machine Quarter Rack to Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Half Rack to Oracle Exadata Database Machine Full Rack Refer to Task 2 Setting Up New Servers on page 1 42 for additional information Always load equipment into the rack from the bottom up so that the rack does not become top heavy and tip over Extend the rack anti tip bar to prevent the rack from tipping during equipment installation See Also Oracle Exadata Database Machine System Overview to view the rack layout Installing the Server Into a Rack in Sun Server X4 2L Installation Guide at http docs oracle com cd E36974_ 01 html E38145 z4000087165586 html scrolltoc The following tasks describes how to add the servers and cables 1 28 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware Task 1 Preparing for Installation The following procedure describes the pre installation steps 1 Identify the rack unit where the server will be installed Fill the first available unit s
19. U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 40 Leaf Switch Connections for the Thirteenth Rack in the System Leaf Switch Connection R13 IB3 to Rack 1 R13 IB3 P8B to R1 U1 P15A R13 IB3 to Rack 2 R13 IB3 P9A to R2 U1 P15A R13 IB3 to Rack 3 R13 IB3 P9B to R3 U1 P15A 2 38 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Fourteen Rack Cabling Table 2 40 Cont Leaf Switch Connections for the Thirteenth Rack in the System Leaf Switch Connection R13 IB3 to Rack 4 R13 IB3 P10A to R4 U1 P15A R13 IB3 to Rack 5 R13 IB3 P10B to R5 U1 P15A R13 IB3 to Rack 6 R13 IB3 P15A to R6 U1 P15A R13 IB3 to Rack 7 R13 IB3 P15B to R7 U1 P15A R13 IB3 to Rack 8 R
20. the minimum release for the servers is release 11 2 3 3 1 a When expanding Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers or Oracle Exadata Database Machine X2 2 with X4170 M2 and X4270 M2 servers with Sun Server X3 2 Oracle Database Servers and Exadata Storage Server X3 2 Servers the minimum release for servers is release 11 2 3 2 0 a When expanding Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers with Sun Fire X4170 M2 Oracle Database Servers and Exadata Storage Server with Sun Fire X4270 M2 Servers the minimum release for servers is release 11 22 2 0 The earlier servers may need to be patched to a later release to meet the minimum In addition earlier database servers may use Oracle Enterprise Linux release 5 3 Those servers need to be updated to the latest Oracle Linux release Additional patching considerations include the Grid Infrastructure and database home releases and bundle patch updates If new patches will be applied then Oracle recommends changing the existing servers so the new servers will inherit the releases as part of the extension procedure This way the number of servers being patched is lower Any patching of the existing servers should be performed in advance so they are at the desired level when the extension work is scheduled thereby reducing the total amount of work required during the extension See Also a My Oracle Support note 888828 1 for information about pat
21. when cabling three racks together Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 3 Leaf Switch Connections for the First Rack in a Three Rack System Leaf Switch R1 IB3 within Rack 1 R1 IB3 P8A to R1 U1 P3A 3 meters Connec
22. 5 meters R2 IB3 to Rack 5 R2 IB3 P10B to R5 U1 P8A 10 meters R2 IB3 to Rack 6 R2 IB3 P11Ato R6 U1 P9A 10 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B 3 meters R2 IB2 P8B to R2 U1 P4B R2 IB2 to Rack 1 R2 IB2 P11B to R1 U1 P10B 5 meters R2 IB2 to Rack 3 R2 IB2 P9A to R3 U1 P5B 5 meters R2 IB2 P9B to R3 U1 P6B R2 IB2 to Rack 4 R2 IB2 P10A to R4 U1 P7B 5 meters R2 IB2 to Rack 5 R2 IB2 P10B to R5 U1 P8B 10 meters R2 IB2 to Rack 6 R2 IB2 P11Ato R6 U1 P9B 10 meters Table 2 17 shows the cable connections for the third spine switch R3 U1 when cabling six full racks together Multi Rack Cabling Tables 2 21 Six Rack Cabling Table 2 17 Leaf Switch Connections for the Third Rack in a Six Rack System Leaf Switch Connection Cable Length R3 IB3 within Rack 3 R3 IB3 P8A to R3 U1 P3A 3 meters R3 IB3 P8B to R3 U1 P4A R3 IB3 to Rack 1 R3 IB3 P11A to R1 U1 P9A 5 meters R3 IB3 to Rack 2 R3 IB3 P11B to R2 U1 P10A 5 meters R3 IB3 to Rack 4 R3 IB3 P9A to R4 U1 P5A 5 meters R3 IB3 P9B to R4 U1 P6A R3 IB3 to Rack 5 R3 IB3 P10A to R5 U1 P7A 5 meters R3 IB3 to Rack 6 R3 IB3 P10B to R6 U1 P8A 10 meters R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B 3 meters R3 IB2 P8B to R3 U1 P4B R3 IB2 to Rack 1 R3 IB2 P11A to R1 U1 P9B 5 meters R3 IB2 to Rack 2 R3 IB2 P11B to R2 U1 P10B 5 meters R3 IB2 to Rack 4 R3 IB2 P9A to R4 U1 P5B 5 meters R3 IB2 P9B to R4 U1 P6B R3 IB2 to Rack 5 R3 IB2 P10A to
23. 8 Full Rack m Cabling Tables for Oracle Exadata Database Machine X2 2 and Oracle Exadata Database Machine X2 8 Full Rack m Cabling Tables for Oracle Exadata Storage Expansion Rack About Extending Oracle Exadata Database Machine You can extend Oracle Exadata Database Machine as follows All Oracle Exadata Database Machines can be extended from quarter rack to half rack All Oracle Exadata Database Machines can be extended from half rack to full rack Any combination of Oracle Exadata Database Machine half racks and full racks can be cabled together A Sun Datacenter InfiniBand Switch 36 switch and cables must be ordered before extending Oracle Exadata Database Machine X4 2 racks Extending Oracle Exadata Database Machine 1 1 About Extending Oracle Exadata Database Machine Notes The cable lengths shown in Chapter 2 Multi Rack Cabling Tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters a Earlier Oracle Exadata Racks can be extended with later Oracle Exadata Racks a When extending Oracle Exadata Database Machine Eighth Rack with Oracle Exadata Storage Expansion Rack make sure that there are two separate disk groups There should be one disk group for the drives in the Oracle Exadata Database Machine Ei
24. Cluster Task 6 Adding Grid Disks to Oracle ASM Disk Groups Task 7 Adding Servers to a Cluster Task 8 Configuring Cell Alerts for New Exadata Storage Servers Task 9 Adding Oracle Database ORACLE_HOME to the New Servers Task 10 Adding Database Instance to the New Servers Note The new and existing racks must be at the same patch level for Exadata Storage Servers and database servers including the operating system Refer to Reviewing Release and Patch Levels on page 1 8 for additional information Extending Oracle Exadata Database Machine 1 41 Configuring the New Hardware Task 1 Changing the Interface Names Earlier releases of Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers used BONDO and BONDI as the names for the bonded InfiniBand and bonded Ethernet client networks respectively In the current release BONDIBO and BONDETHO are used for the bonded InfiniBand and bonded Ethernet client networks If you are adding new servers to an existing Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers then ensure the database servers use the same names for bonded configuration You can either change the new database servers to match the existing server interface names or change the existing server interface names and Oracle Cluster Registry OCR configuration to match the new servers Do the following after changing the interface names 1 Edit the entries in etc sysctl conf f
25. Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 38 Leaf Switch Connections for the Eleventh Rack in the System Leaf Switch Connection R11 IB3 to Rack 1 R11 IB3 P8B to R1 U1 P13A R11 IB3 to Rack 2 R11 IB3 P9A to R2 U1 P13A R11 IB3 to Rack 3 R11 IB3 P9B to R3 U1 P13A R11 IB3 to Rack 4 R11 IB3 P10A to R4 U1 P13A R11 IB3 to Rack 5 R11 IB3 P10B to R5 U1 P13A R11 IB3 to Rack 6 R11 IB3 P13A to R6 U1 P13A R11 IB3 to Rack 7 R11 IB3 P13B to R7 U1 P13A R11 IB3 to Rack 8 R11 IB3 P8A to R8 U1 P13A R11 IB2 to Rack 1 R11 IB2 P8B to R1 U1 P13B R11 IB2 to Rack 2 R11 IB2 P9A to R2 U1 P13B R11 IB2 to Rack 3 R11 IB2 P9B to R3 U1 P13B 2 36 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Twelve Rack Cabling Table 2 38 Cont Leaf Switch Connections for the Eleventh Rack in the System Leaf Switch R11 IB2 to Rack 4 Connection R11 IB2 P10A to R4 U1 P13B R11 IB2 to Rack 5 R11 IB2 P10B to R5 U1 P13B R11 IB2 to Rack 6 R11 IB2 P13A to R6 U1 P13B R11 IB2 to Rack 7 R11 IB2 P13B to R7 U1 P13B R11 IB2 to Rack 8 Twelve Rack Cabling R11 IB2 P8A
26. ORACLE SID oracle ASM1 1 50 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware The Oracle base for ORACLE HOME u01 app 11 2 0 grid is u01 app oracle S SQLPLUS AS sysasm SQL gt ALTER DISKGROUP data ADD DISK 2 gt OF DATA dm0 1 eel 08 3 3 gt o DATA dm01cel09 4 gt o DATA dm0lcell10 5 gt 0 DATA dm0lcelll 6 gt o DATA dm01cel12 7 gt o DATA dm01cel13 8 gt 0 DATA dm01cell4 9 gt rebalance power 11 Notes If your system is running Oracle Database 11g release 2 11 2 0 1 then Oracle recommends a power limit of 11 so that the rebalance completes as quickly as possible If your system is running Oracle Database 11 release 2 11 2 0 2 then Oracle recommends a power limit of 32 The power limit does have an impact on any applications that are running during the rebalance Refer to Oracle Automatic Storage Management Administrator s Guide for information about the ASM_POWER_LIMIT parameter Ensure the ALTER DISKGROUP commands are run from different Oracle ASM instances That way the rebalance operation for multiple disk groups can run in parallel a Add disks to all disk groups including SYSTEMDG or DBFS_DG a When adding servers with 3 TB High Capacity HC disks to existing servers with 2TB disks it is recommended to follow the procedure in My Oracle Support note 1476336 1 to properly define the grid disks
27. P9A to R3 U1 P5B 5 meters R2 IB2 P9B to R3 U1 P6B R2 IB2 to Rack 4 R2 IB2 P10A to R4 U1 P7B 5 meters R2 IB2 P10B to R4 U1 P8B Table 2 8 shows the cable connections for the third spine switch R3 U1 when cabling four full racks together Table 2 8 Leaf Switch Connections for the Third Rack in a Four Rack System Leaf Switch R3 IB3 within Rack 3 Connection R3 IB3 P8A to R3 U1 P3A R3 IB3 P8B to R3 U1 P4A Cable Length 3 meters Multi Rack Cabling Tables 2 15 Five Rack Cabling Table 2 8 Cont Leaf Switch Connections for the Third Rack in a Four Rack System Leaf Switch Connection Cable Length R3 IB3 to Rack 1 R3 IB3 P10A to R1 U1 P7A 5 meters R3 IB3 P10B to R1 U1 P8A R3 IB3 to Rack 2 R3 IB3 P11A to R2 U1 P9A 5 meters R3 IB3 P11B to R2 U1 P10A R3 IB3 to Rack 4 R3 IB3 P9A to R4 U1 P5A 5 meters R3 IB3 P9B to R4 U1 P6A R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B 3 meters R3 IB2 P8B to R3 U1 P4B R3 IB2 to Rack 1 R3 IB2 P10A to R1 U1 P7B 5 meters R3 IB2 P10B to R1 U1 P8B R3 IB2 to Rack 2 R3 IB2 P11A to R2 U1 P9B 5 meters R3 IB2 P11B to R2 U1 P10B R3 IB2 to Rack 4 R3 IB2 P9A to R4 U1 P5B 5 meters Table 2 9 shows the cable connections for the fourth spine switch R4 U1 when R3 IB2 P9B to R4 U1 P6B cabling four full racks together Table 2 9 Leaf Switch Connections for the Fourth Rack in a Four Rack System Leaf Switch Connection Cable Le
28. Quarter Rack to Oracle Exadata Full Rack or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks Oracle Exadata Eighth Rack to Oracle Exadata Eighth Rack Oracle Exadata Eighth Rack to Oracle Exadata Half Rack or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks Oracle Exadata Eighth Rack to Oracle Exadata Full Rack or multiple Oracle Exadata Full Racks or Oracle Exadata Half Racks 2 8 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Understanding Multi Rack Cabling Notes a To cable additional Oracle Exadata Quarter Racks or Oracle Exadata Eighth Racks it is necessary to purchase additional Sun Datacenter InfiniBand Switch 36 switches Itis assumed that four of the six ports dedicated for external connectivity in Oracle Exadata Half Rack or Oracle Exadata Full Rack are unused and available for switch interconnections with Oracle Exadata Quarter Rack or Oracle Exadata Eighth Rack The six ports are 5B 6A 6B 7A 7B and 12A in each leaf switch The following graphic shows the cable connections for two Oracle Exadata Quarter Racks The leaf switches within each rack maintain their existing seven connections The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity Quarter Rack 1 Quarter Rack 2 Leaf Switch 1 Leaf Switch 1 Li i LIL m Led in ue eS Be hee Bee L k ma eet ry
29. R5 U1 P7B 5 meters R3 IB2 to Rack 6 R3 IB2 P10B to R6 U1 P8B 10 meters Table 2 18 shows the cable connections for the fourth spine switch R4 U1 when cabling six full racks together Table 2 18 Leaf Switch Connections for the Fourth Rack in a Six Rack System Leaf Switch Connection Cable Length R4 IB3 within Rack 4 R4 IB3 P8A to R4 U1 P3A 3 meters R4 IB3 P8B to R4 U1 P4A R4 IB3 to Rack 1 R4 IB3 P10B to R1 U1 P8A 10 meters R4 IB3 to Rack 2 R4 IB3 P11A to R2 U1 P9A 5 meters R4 IB3 to Rack 3 R4 IB3 P11B to R3 U1 P10A 5 meters R4 IB3 to Rack 5 R4 IB3 P9A to R5 U1 P5A 5 meters R4 IB3 P9B to R5 U1 P6A R4 IB3 to Rack 6 R4 IB3 P10A to R6 U1 P7A 5 meters R4 IB2 within Rack 4 R4 IB2 P8A to R4 U1 P3B 3 meters R4 IB2 P8B to R4 U1 P4B R4 IB2 to Rack 1 R4 IB2 P10B to R1 U1 P8B 10 meters R4 IB2 to Rack 2 R4 IB2 P11A to R2 U1 P9B 5 meters R4 IB2 to Rack 3 R4 IB2 P11B to R3 U1 P10B 5 meters R4 IB2 to Rack 5 R4 IB2 P9A to R5 U1 P5B 5 meters R4 IB2 P9B to R5 U1 P6B R4 IB2 to Rack 6 R4 IB2 P10A to R6 U1 P7B 5 meters 2 22 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Six Rack Cabling Table 2 19 shows the cable connections for the fifth spine switch R5 U1 when cabling six full racks together Table 2 19 Leaf Switch Connections for the Fifth Rack in a Six Rack System Leaf Switch Connection Cable Length R5 IB3 within Rack
30. alerts on the new Exadata Storage Servers Use the settings on the original Exadata Storage Servers as a guide To view the settings on the original Exadata Storage Servers use a command similar to the following dcli g new_cells_nodes 1 celladmin cellcli e list cell detail To view the alerts on the new Exadata Storage Servers use a command similar to the following dcli g new_cell_ nodes 1 root cellcli e ALTER CELL smtpServer mailserver example com smtpPort 25 smtpUseSSL false smtpFrom DBM dm01 smtpFromAddr storecell example com smtpToAddr dbm admins example com notificationMethod mail snmp N notificationPolicy N critical warning clearN snmpSubscriber host snmpserver example com port 162 AA in s E oe Note The backslash character is used as an escape character for the dcli utility and as a line continuation character in the preceding command a When cabling racks Use Oracle Exadata Deployment Assistant to set up e mail alerts for Exadata Storage Servers as the root user from the original rack to the new rack The utility includes the SetupCellEmailAlerts step to configure alerts Task 9 Adding Oracle Database ORACLE_HOME to the New Servers It is necessary to add the Oracle Database software directory ORACLE_HOME to the servers after the cluster modifications are complete and all the servers are in the cluster The following procedure describes how to add the
31. all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 42 Leaf Switch Connections for the Fifteenth Rack in the System Leaf Switch Connection R15 IB3 to Rack 1 R15 IB3 P8B to R1 U1 P17A R15 IB3 to Rack 2 R15 IB3 P9A to R2 U1 P17A R15 IB3 to Rack 3 R15 IB3 P9B to R3 U1 P17A R15 IB3 to Rack 4 R15 IB3 P10A to R4 U1 P17A R15 IB3 to Rack 5 R15 IB3 P10B to R5 U1 P17A R15 IB3 to Rack 6 R
32. allow any InfiniBand cable to experience extreme tension Do not pull or drag an InfiniBand cable Pulling on an InfiniBand cable can damage the cable internally Unroll an InfiniBand cable for its length Do not twist an InfiniBand cable more than one revolution for its entire length Twisting an InfiniBand cable can damage the cable internally Do not route InfiniBand cables where they can be stepped on or experience rolling loads A crushing effect can damage the cable internally Estimating InfiniBand Cable Path Lengths Cable paths should be as short as possible When the length of a cable path has been calculated select the shortest cable to satisfy the length requirement When specifying a cable consider the following Bends in the cable path increase the required length of the cable Rarely does a cable travel in a straight line from connector to connector Bends in the cable path are necessary and each bend increases the total length Bundling increases the required length of the cables Bundling causes one or more cables to follow a common path However the bend radius is different in different parts of the bundle If the bundle is large and unorganized and there are many bends one cable might experience only the inner radius of bends while another cable might experience the outer radius of bends In this situation the differences of the required lengths of the cables is quite substantial 1 4 Oracle Exadata Database Mac
33. and disk groups At this point of setting up the rack the new grid disks should be defined but need to be placed into disk groups Refer to the steps in My Oracle Support note 1476336 1 a If the existing Exadata Cells have High Performance HP disks and you are adding Exadata Cells with High Capacity HC disks or the existing Exadata Cells have HC disks and you are adding Exadata Cell HP disks then you must place the new disks in new disk groups It is not permitted to mix HP and HC disks within the same disk group 6 Monitor the status of the rebalance operation using a query similar to the following from any Oracle ASM instance SQL gt SELECT FROM GVSASM_OPERATION WHERE STATE RUN The remaining tasks can be done while the rebalance is in progress Task 7 Adding Servers to a Cluster The following procedure describes how to add servers to a cluster Extending Oracle Exadata Database Machine 1 51 Configuring the New Hardware Caution If Cluster Ready Services CRS manages additional services that are not yet installed on the new nodes such as Oracle Golden Gate then note the following tmay be necessary to stop those services on the existing node before running the addNode sh script tis necessary to create any users and groups on the new database servers that run these additional services It may be necessary to disable those services from auto start so that CRS does not try to start
34. are in the shipping kit 2 Optional Replace the side panels if they were removed for the upgrade as follows a Lift each side panel up and onto the side of the rack The top of the rack should support the weight of the side panel Ensure the panel fasteners line up with the grooves in the rack frame b Turn each side panel fastener one quarter turn clockwise using the side panel removal tool Turn the fasteners next to the panel lock clockwise There are 10 fasteners per side panel c Optional Lock each side panel The key is in the shipping kit The locks are located on the bottom center of the side panels d Connect the grounding straps to the side panels After closing the rack proceed to Configuring the New Hardware on page 1 41 to configure the new hardware Extending a Rack by Adding Another Rack Extending Oracle Exadata Database Machine by adding another rack consists of cabling and configuring the racks together Racks can be cabled together with no downtime During the cabling procedure the following should be noted Extending Oracle Exadata Database Machine 1 37 Extending the Hardware m There is some performance degradation while cabling the racks together This degradation results from reduced network bandwidth and the data retransmission due to packet loss when a cable is unplugged a The environment is not a high availability environment because one leaf switch will need to be off All traffic
35. costs or damages incurred due to your access to or use of third party content products or services except as set forth in an applicable agreement between you and Oracle Contents Prelace dann ix VASTO ala Gos RTRT PERI eee ROMEO Te AE SETA AIR RONN See cone ene Tee URE SESE Ry APE eee eee TTT IX Doeymeniaton Accessibliy aaacasa IX Relot dPbocumentationiga israele alri IX C HA GIO uuu bu s L a aun biuk wat bi nana Xi 1 Extending Oracle Exadata Database Machine About Extending Oracle Exadata Database Machine eeeeeseeceseeeneeceseceseeesseenseeesaeenaees 1 1 Preparing to Extend Oracle Exadata Database Machine i 1 3 Reviewing the Safety Precautions usura 1 3 Reviewing the InfiniBand Cable Precautions iseicnsraiiiani naana uaaiassu 1 4 Estimating InfiniBand Cable Path Lengthszu cinsinin iii 1 4 B ndine Iatnip a PC b uu E R N 1 5 Reviewing the Cable Management Arm Guidelines cece a 1 5 Obtaining Current Configuration Information i 1 6 Preparing the Network Confistralionisssiai aaa 1 7 Moving Audit and Diagnostie Files assesserings iisi 1 7 Reviews Releaseand Fateh Levels azoni onnenn TE S 1 8 Performing Preliminary Check anale 1 8 Prepatine 0 Add Server aio E A EA 1 9 Extending th Hardware osea E A A 1 10 Extending an Eighth Rack to a Quarter Rack in Oracle Exadata Database Machine X4 2 and Later 1 10 Extending an Eighth Rack to a Quarter Ra
36. dbm dat files to verify the network configuration The only errors that should occur are from the ping command to the SCAN addresses Cisco switch and Sun Datacenter InfiniBand Switch 36 switches See Also Configuring Servers Manually on page 1 44 Moving Audit and Diagnostic Files The files in the GRID_HOME rdbms audit directory and the GRID_ HOME log diagnostics directory should be moved or deleted before extending a cluster Oracle recommends moving or deleting the files a day or two before the planned extension because it may take time Extending Oracle Exadata Database Machine 1 7 Preparing to Extend Oracle Exadata Database Machine Reviewing Release and Patch Levels The new rack or servers most likely have a later release or patch level than the current rack In some cases you may want to update the current rack release to the later release In other cases you may want to stay at your current release and choose to reimage the new rack to match the current rack Whatever you choose to do ensure that the existing and new servers and Sun Datacenter InfiniBand Switch 36 switches are at the same patch level Note the following about the hardware and releases a When expanding Oracle Exadata Database Machine X4 2 with Sun Server X4 2 Oracle Database Serverss and Exadata Storage Server X4 2L Servers the minimum release for the servers is release 11 2 3 3 0 a When expanding with Oracle Exadata Database Machine X4 8 Full Rack
37. disk groups by using the following command select from gvSasm_operation No rows should be returned by the command Review the configuration settings such as the following All parallelism settings Backup configurations Standby site if any Service configuration Oracle Database File System DBFS configuration and mount points on new servers Installation of Oracle Enterprise HugePage Manager agents on new database servers HugePages settings Incorporate the new cell and database servers into Auto Service Request See Also Oracle Exadata Quick Installation Guide for ASR at http www oracle com technetwork systems asr documentation Update Grid Control to include the new nodes Extending Oracle Exadata Database Machine 1 59 Returning the Rack to Service 1 60 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide 2 Multi Rack Cabling Tables This appendix contains the tables for multi rack cabling It contains the following topics Understanding Multi Rack Cabling Two Rack Cabling Three Rack Cabling Four Rack Cabling Five Rack Cabling Six Rack Cabling Seven Rack Cabling Fight Rack Cabling Nine Rack Cabling Ten Rack Cabling Eleven Rack Cabling Twelve Rack Cabling Thirteen Rack Cabling Fourteen Rack Cabling Fifteen Rack Cabling Sixteen Rack Cabling Seventeen Rack Cabling Eighteen Rack Cabling Understanding Multi Rack Cabling Three Sun Datacenter InfiniBand Swi
38. following command deli g cell group 1 celladmin cellcli e create celldisk all 1 12 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware 4 Recreate the flash log using the following commands deli g cell group 1 celladmin cellcli e drop flashlog all force dcli g cell group 1 celladmin cellcli e create flashlog all 5 Expand the flash cache using the following command deli g cell group 1 celladmin cellcli e alter flashcache all Creating Grid Disks in Oracle Exadata Database Machine X4 2 Eighth Rack or Later Grid disk creation must follow a specific order to ensure the proper offset The order of grid disk creation must follow the same sequence that was used during initial grid disks creation For a standard deployment using Oracle Exadata Deployment Assistant the order is DATA RECO and DBFS_DG Create all DATA grid disks first followed by the RECO grid disks and then the DBFS_DG grid disks The following procedure describes how to create the grid disks Note The commands shown in this procedure use the standard deployment grid disk prefix names of DATA RECO and DBFS_DG The sizes being checked are on cell disk 02 Cell disk 02 is used because the disk layout for cell disks 00 and 01 are different from the other cell disks in the server 1 Checkthesizeofthe grid disks using the following commands Each cell should return the same size for the gr
39. from the front You need to keep it to the left side of the rack as far as possible while pulling the two power cords through the C bracket on the right side Slide the server in rack location U2 out to the locked service position This improves access to the rear of the switch during further assembly Install the slide rails from the rear of the rack into the C brackets on the switch pushing them up to the rack rail Attach an assembled cable arm bracket to the slide rail and using a No 3 Phillips screwdriver screw these together into the rack rail a Install the lower screw loosely with the cable arm bracket rotated 90 degrees downward This allows better finger access to the screw b Rotate the cable arm bracket to the correct position c Install the upper screw d Tighten both screws If available a screwdriver with a long shaft 16 inch 400mm will allow easier installation such that the handle is outside the rack and beyond the cabling Push the switch completely into the rack from the front routing the power cords through the cutout on the rail bracket Secure the switch to the front rack rail with M6 16mm screws Tighten the screws using the No 3 Phillips screwdriver Install the lower part of the cable management arm across the back of the switch Connect the cables to the appropriate ports See Also Oracle Exadata Database Machine System Overview for information about InfiniBand networking cables Instal
40. goes through the remaining leaf switch a Only the existing rack is operational and any new rack that is added is powered down x The software running on the systems cannot have problems related to InfiniBand restarts a Itis assumed that Oracle Exadata Database Machine Half Racks have three InfiniBand switches already installed The new racks have been configured with the appropriate IP addresses to be migrated into the expanded system prior to any cabling and there are no duplicate IP addresses a The existing spine switch is set to priority 10 during the cabling procedure This setting gives the spine switch a higher priority than any other switch in the fabric and is the first to take the Subnet Manager Master role whenever a new Subnet Manager Master is being set during the cabling procedure The following sections describe how to extend Oracle Exadata Database Machine with another rack Cabling Two Racks Together m Cabling Several Racks Together Cabling Two Racks Together The following procedure describes how to cable two racks together This procedure assumes that the racks are adjacent to each other In the procedure the existing rack is R1 and the new rack is R2 See Also Chapter 2 Multi Rack Cabling Tables Oracle Exadata Database Machine System Overview for the following cabling tables m Cabling Tables for Oracle Exadata Database Machine X4 2 m Cabling Tables for Oracle Exadata Database Machine
41. new servers Refer to Reviewing Release and Patch Levels on page 1 8 for additional information 2 Modifythe etc oracle cell network config cellip ora file on all database servers to have a complete list of all Exadata Storage Servers This can be done by modifying the file for one database server and then copying the file to the other servers The cellip ora file should be identical on all database servers When adding Exadata Storage Server X4 2L Servers the cellip ora file contains two IP addresses listed for each cell Copy each line completely to include the two Extending Oracle Exadata Database Machine 1 49 Configuring the New Hardware IP addresses and merge the addresses in the cellip ora file of the existing cluster The following is an example of the cellip ora file after expanding Oracle Exadata Database Machine X3 2 Half Rack to full rack using Exadata Storage Server X4 2L Servers servers cell 192 168 10 9 cell 192 168 10 10 cell 192 168 10 11 eell 192 168 10 12 cell 192 168 10 13 cell 192 168 10 14 cell 192 168 10 15 Cella 192 166210 Liles 6 Se l0 13 cell 192 168 10 1919216801020 cell 1592 16d LO 21 192 168 10 222 cell 192 168 10 23 192 168 10 24 Cell 192 168 10 25 192 168 10 26 cell 192 168 10 27 192 168 10 28 Cell 192 1L6081029 192 16810530 In the preceding example lines 1 through 7 are for the original servers and lines 8 through 14 are for the new servers Exad
42. on nodes dm01db01 dm01db01 No such file or directory PRVF 5431 Oracle Cluster Voting Disk configuration check If such an error occurs then set the environment variable as follows S export IGNORE_PREADDNODE_CHECKS Y Setting the environment variable does not prevent the error when running the cluvfy command but it does allow the addNode sh script to complete successfully If a database server was installed with a certain image and subsequently patched to a later image then some operating system libraries may be older than the version expected by the cluvfy command This causes the cluvfy command and possibly the addNode sh script to fail It is permissible to have an earlier version as long as the difference in version is minor For example glibc common 2 5 81 el5_8 2 versus glibc common 2 5 49 The versions are different but both are at version 2 5 so the difference is minor and it is permissible for them to differ Set the environment variable IGNORE _PREADDNODE CHECKS Y before running the addNode sh script to workaround this problem Extending Oracle Exadata Database Machine 1 53 Configuring the New Hardware 10 11 12 13 14 Ensure that all directories inside the Grid Infrastructure home on the existing server have their executable bits set To do this use the following commands as the root owner find u01 app 11 2 0 grid type d user root perm utx perm g x perm o x
43. or a third Sun Datacenter InfiniBand Switch 36 switch To extend Oracle Exadata Database Machine X4 2 racks and Oracle Exadata Database Machine X3 8 Full Racks with Exadata Storage Server X4 2L Servers order cables and a Sun Datacenter InfiniBand Switch 36 switch In a single rack the two leaf switches are interconnected using seven connections In addition each leaf switch has one connection to the spine switch The leaf switches connect to the spine switch as shown in the following graphic Oracle Database Server and Exadata Storage Server IB Ports wa 7 n mu 4 Leaf Switch 1 Leaf Switch 2 The Oracle Database servers and Exadata Storage Servers connect to the leaf switches as shown in the following graphic Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Understanding Multi Rack Cabling Rack 1 Port A Oracle Database Servers 1 2 3 4 Exadata Storage Servers 1 2 7 Leaf Switch 1 11 Port B Oracle Database Servers 5 6 7 8 Exadata Storage Servers 8 9 14 gt 7 w wa Mid so See 3 J gt ee See 3 ee o si ee 2 ee A are eee fe Le Lee Yr Seat renti eee er niet einen pene 4 om H G E 9 1 4 Srrrrr ry Le WIITISITISITZNIITTKTYXYYE EXEFE E E Cc Port A Oracle Database Servers 5 6 7 8 Exadata Storage Servers 8 9 14 Leaf Switch 2 a E ass a TT IT Er t AA oa a a L Ln L
44. performance disks and cells with high capacity disks are being added then it is necessary to create new disk groups for the high capacity disks When adding the same type of disk ensure that the grid disk sizes are exactly the same even if the new disks are larger than the existing ones For example if the existing disks are 3 TB and the additional disks are 4 TB then it is necessary to create grid disks that match the size on the 3TB disks A new disk group can be created using the extra 1 TB of disk space m In order to access Exadata Storage Servers in one Oracle Exadata Database Machine by another Oracle Exadata Database Machine when they are not running as a single cluster Exadata Storage Servers must have unique Oracle ASM disk group and failure group names on each Oracle Exadata Database Machine For 1 2 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Preparing to Extend Oracle Exadata Database Machine example for two Oracle Exadata Database Machines cabled together but run as separate clusters the following names should be unique Cell name Cell disk name Grid disk name Oracle ASM failure group name See Also Oracle Automatic Storage Management Administrator s Guide for information about renaming disk groups All equipment receives a Customer Support Identifier CSI Any new equipment for Oracle Exadata Database Machine has a new CSI Contact Oracle Support Services to recon
45. set correctly Returning the Rack to Service Use the following procedure to ensure the new hardware is correctly configured and ready for use 1 Run the opt oracle SupportTools ibdiagtools verify topology command to ensure that all InfiniBand cables are connected and secure See Also Oracle Exadata Database Machine Maintenance Guide for information about verifying the InfiniBand network configuration 2 Run the Oracle Exadata Database Machine HealthCheck utility using the steps described in My Oracle Support note 1070954 1 3 Verify the instance additions using the following commands srvctl config database d dbm srvctl status database d dbm 4 Check the cluster resources using the following command crsctl stat res t 5 Ensure the original configuration summary report from the original cluster deployment is updated to include all servers This document should include the calibrate and network verifications for the new rack and the InfiniBand cable checks verify topology and infinicheck 6 Conduct a power off test if possible If the new Exadata Storage Servers cannot be powered off then verify that the new database servers with the new instances can be powered off and powered on and that all processes start automatically 1 58 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Returning the Rack to Service Note Ensure the Oracle ASM disk rebalance process has completed for all
46. switch links between R1IB2 and R1 IB3 Cable leaf switch R1 IB2 according to Table 2 1 Power on leaf switch R1 IB2 Wait for three minutes for R1 IB2 to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show three switches R1 IB1 R1 IB2 and R1 IB3 Power off leaf switch R1 IB3 This causes all the database servers and Exadata Storage Servers to fail over their InfiniBand traffic to R1 IB2 Cable leaf switch R1 IB3 according to Table 2 1 Power on leaf switch R1 IB3 Wait for three minutes for R1 IB3 to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show three switches R1 IB1 R1 IB2 and R1 IB3 Power on all the InfiniBand switches in R2 Wait for three minutes for the switches to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show six switches R1 IB1 R1 IB2 R1 IB3 R2 IB1 R2 IB2 and R2 IB3 Ensure the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch Power on all servers in R2 Log in to spine switch R1 IB1 and lower its priority to 8 as follows a Use the disablesm command to stop Subnet Manager b Use the setsmpriority 8 command to set the priority to 8 c Use the enablesm command to restart Subnet Manager Ensure Subnet Manager Master is running
47. the following cases a Initial installation of all Oracle Exadata Database Machine Full Racks System address assignments and host names should be complete New Oracle Exadata Database Machine Full Racks are added to an existing cluster The new rack configuration should require unique host names and IP addresses for the new Oracle Exadata Database Machines The IP addresses on the same subnet cannot conflict with the existing systems K Two existing Oracle Exadata Database Machine Full Racks are clustered together You can assign host names and IP addresses only if Oracle Exadata Database Machines are already assigned unique host names and IP addresses or the entire cluster must be reconfigured The machines must be on the same subnet and not have conflicting IP addresses Ensure the IP addresses for the new servers are in the same subnet and do not overlap with the currently installed servers Ensure the firmware on the original switches are at the same level as the new switches using the nm2version command If the firmware is not at the same level then apply a firmware patch Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks can be cabled as follows Oracle Exadata Quarter Rack to Oracle Exadata Quarter Rack Oracle Exadata Quarter Rack to Oracle Exadata Half Rack or multiple Oracle Exadata Half Racks or Oracle Exadata Full Racks Oracle Exadata
48. this guide Verify the time is the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database servers Ensure the NTP settings are the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database servers Configure HugePages on the new servers to match the existing servers Ensure the values in the etc security limits conf file for the new database servers match the existing database servers Go to Task 4 Setting User Equivalence to continue the hardware configuration Configuring Servers Manually Use the following procedure to configure the servers manually 1 44 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware 1 Prepare the servers using the procedure described in Preparing the Servers in chapter 2 of Oracle Exadata Storage Server Software User s Guide 2 Ensure the InfiniBand and bonded client Ethernet interface names are the same on the new database servers as on the existing database servers 3 Configure the Exadata Storage Servers cell disks and grid disks as described in Configuring Cells Cell Disks and Grid Disks with CellCLI in chapter 2 of Oracle Exadata Storage Server Software User s Guide 4 Configure the database servers as described in Setting Up Configuration Files for a Database Server Host in chapter 2 of Oracle Exadata Stor
49. three Oracle Exadata Database Machine Full Racks The following table lists the spare cables for the switch Cable InfiniBand 2 meter cable InfiniBand 3 meter cable InfiniBand 5 meter cable Spare 0 10 Inter switch 7 Total 10 When connecting four or more racks it is necessary to purchase additional 10 meter optical QSFP to QSFP QDR InfiniBand cables Two QSFP SR transceivers are needed for each optical cable The following table lists the number of racks and cables needed to interconnect the racks 3 Gather the following tools n Pen Note pad Flashlight a Labels for cables and rack 3 Meter Cable 5 Meter Cables 10 Meter Cables Number of Racks Needed Needed Needed 2 16 16 0 3 18 30 0 4 16 40 8 5 20 42 18 6 24 46 26 7 28 44 40 8 16 52 60 m Cable ties Oracle recommends hook and loop straps Multi Rack Cabling Tables 2 7 Understanding Multi Rack Cabling See Also Oracle Exadata Database Machine Maintenance Guide for information about cable labels Determine a naming method for the rack prefixes For example if the original rack has the prefix dm01 then use the prefix dm02 for the second rack the prefix dm03 for the third rack and so on Verify the racks have unique host names and IP addresses All servers interconnected in the racks must have unique names and IP addresses Server names and IP addresses conventions may differ in
50. to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported 2 10 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Two Rack Cabling See Also Cabling Two Racks Together on page 1 38 Table 2 1 Leaf Switch Connections for the First Rack in a Two Rack System Leaf Switch R1 IB3 within Rack 1 Connection R1 IB3 P8A to R1 U1 P3A R1 IB3 P8B t
51. 0 meters R2 IB3 to Rack 8 R2 IB3 P11A to R8 U1 P9A 10 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B 3 meters R2 IB2 to Rack 1 R2 IB2 P11B to R1 U1 P10B 5 meters R2 IB2 to Rack 3 R2 IB2 P8B to R3 U1 P4B 5 meters R2 IB2 to Rack 4 R2 IB2 P9A to R4 U1 P5B 5 meters R2 IB2 to Rack 5 R2 IB2 P9B to R5 U1 P6B 10 meters R2 IB2 to Rack 6 R2 IB2 P10A to R6 U1 P7B 10 meters R2 IB2 to Rack 7 R2 IB2 P10B to R7 U1 P8B 10 meters R2 IB2 to Rack 8 R2 IB2 P11A to R8 U1 P9B 10 meters Leaf Switch Connection Cable Length R3 IB3 within Rack 3 R3 IB3 P8A to R3 U1 P3A 3 meters R3 IB3 to Rack 1 R3 IB3 P11A to R1 U1 P9A 5 meters R3 IB3 to Rack 2 R3 IB3 P11B to R2 U1 P10A 5 meters R3 IB3 to Rack 4 R3 IB3 P8B to R4 U1 P4A 5 meters R3 IB3 to Rack 5 R3 IB3 P9A to R5 U1 P5A 5 meters R3 IB3 to Rack 6 R3 IB3 P9B to R6 U1 P6A 10 meters R3 IB3 to Rack 7 R3 IB3 P10A to R7 U1 P7A 10 meters R3 IB3 to Rack 8 R3 IB3 P10B to R8 U1 P8A 10 meters R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B 3 meters R3 IB2 to Rack 1 R3 IB2 P11A to R1 U1 P9B 5 meters R3 IB2 to Rack 2 R3 IB2 P11B to R2 U1 P10B 5 meters R3 IB2 to Rack 4 R3 IB2 P8B to R4 U1 P4B 5 meters R3 IB2 to Rack 5 R3 IB2 P9A to R5 U1 P5B 5 meters R3 IB2 to Rack 6 R3 IB2 P9B to R6 U1 P6B 10 meters Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Eight Rack Cabling Table 2 30 Cont Leaf Switch Connections for the Third Rack in a Eight Rack System
52. 1 P1A R17 IB3 to Rack 2 R17 IB3 P9A to R2 U1 P1A R17 IB3 to Rack 3 R17 IB3 P9B to R3 U1 P1A R17 IB3 to Rack 4 R17 IB3 P10A to R4 U1 P1A R17 IB3 to Rack 5 R17 IB3 P10B to R5 U1 P1A R17 IB3 to Rack 6 R17 IB3 P1A to R6 U1 P1A R17 IB3 to Rack 7 R17 IB3 P1B to R7 U1 P1A R17 IB3 to Rack 8 R17 IB3 P8A to R8 U1 P1A R17 IB2 to Rack 1 R17 IB2 P8B to R1 U1 P1B R17 IB2 to Rack 2 R17 IB2 P9A to R2 U1 P1B R17 IB2 to Rack 3 R17 IB2 P9B to R3 U1 P1B Multi Rack Cabling Tables 2 43 Eighteen Rack Cabling Table 2 44 Cont Leaf Switch Connections for the Seventeenth Rack in the System Leaf Switch Connection R17 IB2 to Rack 4 R17 IB2 P10A to R4 U1 P1B R17 IB2 to Rack 5 R17 IB2 P10B to R5 U1 P1B R17 IB2 to Rack 6 R17 IB2 P1A to R6 U1 P1B R17 IB2 to Rack 7 R17 IB2 P1B to R7 U1 P1B R17 IB2 to Rack 8 R17 IB2 P8A to R8 U1 P1B Eighteen Rack Cabling Table 2 45 shows the cable connections for the switches when cabling eighteen racks together The cabling between racks 1 through 8 do not change Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 r
53. 1 R12 IB3 P8B to R1 U1 P14A R12 IB3 to Rack 2 R12 IB3 P9A to R2 U1 P14A R12 IB3 to Rack 3 R12 IB3 P9B to R3 U1 P14A R12 IB3 to Rack 4 R12 IB3 P10A to R4 U1 P14A R12 IB3 to Rack 5 R12 IB3 P10B to R5 U1 P14A R12 IB3 to Rack 6 R12 IB3 P14A to R6 U1 P14A R12 IB3 to Rack 7 R12 IB3 P14B to R7 U1 P14A Multi Rack Cabling Tables 2 37 Thirteen Rack Cabling Table 2 39 Cont Leaf Switch Connections for the Twelfth Rack in the System Leaf Switch Connection R12 IB3 to Rack 8 R12 IB3 P8A to R8 U1 P14A R12 IB2 to Rack 1 R12 IB2 P8B to R1 U1 P14B R12 IB2 to Rack 2 R12 IB2 P9A to R2 U1 P14B R12 IB2 to Rack 3 R12 IB2 P9B to R3 U1 P14B R12 IB2 to Rack 4 R12 IB2 P10A to R4 U1 P14B R12 IB2 to Rack 5 R12 IB2 P10B to R5 U1 P14B R12 IB2 to Rack 6 R12 IB2 P14A to R6 U1 P14B R12 IB2 to Rack 7 R12 IB2 P14B to R7 U1 P14B R12 IB2 to Rack 8 R12 IB2 P8A to R8 U1 P14B Thirteen Rack Cabling Table 2 40 shows the cable connections for the switches when cabling thirteen racks together The cabling between racks 1 through 8 do not change Notes a The spine switch is in U1 also referred to as IB1 for all racks Inthe following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and
54. 1 U1 P2B R18 IB2 to Rack 2 R18 IB2 P9A to R2 U1 P2B R18 IB2 to Rack 3 R18 IB2 P9B to R3 U1 P2B R18 IB2 to Rack 4 R18 IB2 P10A to R4 U1 P2B R18 IB2 to Rack 5 R18 IB2 P10B to R5 U1 P2B R18 IB2 to Rack 6 R18 IB2 P2A to R6 U1 P2B R18 IB2 to Rack 7 R18 IB2 P2B to R7 U1 P2B R18 IB2 to Rack 8 R18 IB2 P8A to R8 U1 P2B Multi Rack Cabling Tables 2 45 Eighteen Rack Cabling 2 46 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Glossary cell number An attribute of a cell The cell number is used with the IP block attribute to compute the range of IP addresses and the netmask assigned to the cell Cell Server CELLSRV See CELLSRV CELLSRV Cell Server a block server that runs on the cell and provides a read write application programming interface API for Exadata Storage Server and scheduling of I O on the disks client application Any application that directly uses the storage server to store persistent data such as Oracle Database database administrator A user responsible for managing individual databases In larger deployments this responsibility is separate from storage administration but database administrators and storage administrators cooperate in managing the storage related aspects of application support database resource plan A resource plan that specifies how CPU and I O resources are allocated among consumer groups in its database The plan is created using PL SQL commands data
55. 13 IB3 P8A to R8 U1 P15A R13 IB2 to Rack 1 R13 IB2 P8B to R1 U1 P15B R13 IB2 to Rack 2 R13 IB2 P9A to R2 U1 P15B R13 IB2 to Rack 3 R13 IB2 P9B to R3 U1 P15B R13 IB2 to Rack 4 R13 IB2 P10A to R4 U1 P15B R13 IB2 to Rack 5 R13 IB2 P10B to R5 U1 P15B R13 IB2 to Rack 6 R13 IB2 P15A to R6 U1 P15B R13 IB2 to Rack 7 R13 IB2 P15B to R7 U1 P15B R13 IB2 to Rack 8 R13 IB2 P8A to R8 U1 P15B Fourteen Rack Cabling Table 2 41 shows the cable connections for the switches when cabling fourteen racks together The cabling between racks 1 through 8 do not change Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 resp
56. 15 IB3 P17A to R6 U1 P17A R15 IB3 to Rack 7 R15 IB3 P17B to R7 U1 P17A R15 IB3 to Rack 8 R15 IB3 P8A to R8 U1 P17A R15 IB2 to Rack 1 R15 IB2 P8B to R1 U1 P17B R15 IB2 to Rack 2 R15 IB2 P9A to R2 U1 P17B R15 IB2 to Rack 3 R15 IB2 P9B to R3 U1 P17B R15 IB2 to Rack 4 R15 IB2 P10A to R4 U1 P17B R15 IB2 to Rack 5 R15 IB2 P10B to R5 U1 P17B R15 IB2 to Rack 6 R15 IB2 P17A to R6 U1 P17B R15 IB2 to Rack 7 R15 IB2 P17B to R7 U1 P17B R15 IB2 to Rack 8 R15 IB2 P8A to R8 U1 P17B Multi Rack Cabling Tables 2 41 Sixteen Rack Cabling Sixteen Rack Cabling Table 2 43 shows the cable connections for the switches when cabling sixteen racks together The cabling between racks 1 through 8 do not change Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata D
57. 2 P10A to R1 U1 P7B 10 meters R4 IB2 P10B to R1 U1 P8B R4 IB2 to Rack 2 R4 IB2 P11A to R2 U1 P9B 5 meters R4 IB2 to Rack 3 R4 IB2 P11B to R3 U1 P10B 5 meters R4 IB2 to Rack 5 R4 IB2 P9A to R5 U1 P5B 5 meters R4 IB2 P9B to R5 U1 P6B Table 2 14 shows the cable connections for the fifth spine switch R5 U1 when cabling five full racks together Table 2 14 Leaf Switch Connections for the Fifth Rack in a Five Rack System Leaf Switch Connection Cable Length R5 IB3 within Rack 5 R5 IB3 P8A to R5 U1 P3A 3 meters R5 IB3 P8B to R5 U1 P4A R5 IB3 to Rack 1 R5 IB3 P9A to R1 U1 P5A 10 meters R5 IB3 P9B to R1 U1 P6A R5 IB3 to Rack 2 R5 IB3 P10A to R2 U1 P7A 10 meters R5 IB3 P10B to R2 U1 P8A R5 IB3 to Rack 3 R5 IB3 P11A to R3 U1 P9A 5 meters R5 IB3 to Rack 4 R5 IB3 P11B to R4 U1 P10A 5 meters R5 IB2 within Rack 5 R5 IB2 P8A to R5 U1 P3B 3 meters R5 IB2 P8B to R5 U1 P4B Multi Rack Cabling Tables 2 19 Six Rack Cabling Table 2 14 Cont Leaf Switch Connections for the Fifth Rack in a Five Rack System Leaf Switch Connection Cable Length R5 IB2 to Rack 1 R5 IB2 P9A to R1 U1 P5B 10 meters R5 IB2 P9B to R1 U1 P6B R5 IB2 to Rack 2 R5 IB2 P10A to R2 U1 P7B 10 meters R5 IB2 P10B to R2 U1 P8B R5 IB2 to Rack 3 R5 IB2 P11A to R3 U1 P9B 5 meters R5 IB2 to Rack 4 R5 IB2 P11B to R4 U1 P10B 5 meters Six Rack Cabling Table 2 15 shows the cable connections for the first spine switch R1 U1 w
58. 32 SQL gt ALTER DISKGROUP dbfs_dg diskgroup ADD DISK 0 DBFS_DG_CD_0 6 9 0 DBFS_DG_CD_1 0 1 REBALANCE POWER 32 The preceding commands return Diskgroup altered if successful Optional Monitor the current rebalance operation using the following command SQL gt SELECT FROM gvSasm operation Re enable the appliance mode attribute if it was disabled in step 6 using the following commands SQL gt ALTER DISKGROUP data _diskgroup set attribute appliance mode TRUE SOL gt ALTER DISKGROUP recodiskgroup set attribute appliance mode TRUE SQL gt ALTER DISKGROUP dbfs_dg_diskgroup set attribute appliance mode TRUE Validating New Oracle Exadata Database Machine X3 2 Half Rack Configuration After adding the grid disks to the Oracle ASM disk groups validate the configuration The following procedure describes how to validate the configuration 1 2 Log in as the root user on the first database server Check the core count using the following command deli g db _ group l root dbmcli e list dbserver attributes coreCount Review the storage server configuration using the following command deli g cell group 1 celladmin cellcli e list cell attributes eighthrack The output should show FALSE Review the appliance mode for each disk group using the following commands SQL gt set lines 100 SQL gt column attribute format a20 SQL gt column value format a20 SQL gt column diskgroup fo
59. 5 R5 IB3 P8A to R5 U1 P3A 3 meters R5 IB3 P8B to R5 U1 P4A R5 IB3 to Rack 1 R5 IB3 P10A to R1 U1 P7A 10 meters R5 IB3 to Rack 2 R5 IB3 P10B to R2 U1 P8A 10 meters R5 IB3 to Rack 3 R5 IB3 P11A to R3 U1 P9A 5 meters R5 IB3 to Rack 4 R5 IB3 P11B to R4 U1 P10A 5 meters R5 IB3 to Rack 6 R5 IB3 P9A to R6 U1 P5A 5 meters R5 IB3 P9B to R6 U1 P6A R5 IB2 within Rack 5 R5 IB2 P8A to R5 U1 P3B 3 meters R5 IB2 P8B to R5 U1 P4B R5 IB2 to Rack 1 R5 IB2 P10A to R1 U1 P7B 10 meters R5 IB2 to Rack 2 R5 IB2 P10B to R2 U1 P8B 10 meters R5 IB2 to Rack 3 R5 IB2 P11A to R3 U1 P9B 5 meters R5 IB2 to Rack 4 R5 IB2 P11B to R4 U1 P10B 5 meters R5 IB2 to Rack 6 R5 IB2 P9A to R6 U1 P5B 5 meters R5 IB2 P9B to R6 U1 P6B Table 2 20 shows the cable connections for the sixth spine switch R6 U1 when cabling six full racks together Table 2 20 Leaf Switch Connections for the Sixth Rack in a Six Rack System Leaf Switch Connection Cable Length R6 IB3 within Rack 6 R6 IB3 P8A to R6 U1 P3A 3 meters R6 IB3 P8B to R6 U1 P4A R6 IB3 to Rack 1 R6 IB3 P9A to R1 U1 P5A 10 meters R6 IB3 P9B to R1 U1 P6A R6 IB3 to Rack 2 R6 IB3 P10A to R2 U1 P7A 10 meters R6 IB3 to Rack 3 R6 IB3 P10B to R3 U1 P8A 10 meters R6 IB3 to Rack 4 R6 IB3 P11A to R4 U1 P9A 5 meters R6 IB3 to Rack 5 R6 IB3 P11B to R5 U1 P10A 5 meters R6 IB2 within Rack 6 R6 IB2 P8A to R6 U1 P3B 3 meters R6 IB2 P8B to R6 U1 P4B R6 IB2 to Rack 1 R6 IB2 P9A to R1 U1 P5B 10 mete
60. 6 R1 IB3 P10B to R6 U1 P8A 10 meters R1 IB3 to Rack 7 R1 IB3 P11A to R7 U1 P9A 10 meters R1 IB3 to Rack 8 R1 IB3 P11B to R8 U1 P10A 10 meters R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 to Rack 2 R1 IB2 P8B to R2 U1 P4B 5 meters R1 IB2 to Rack 3 R1 IB2 P9A to R3 U1 P5B 5 meters R1 IB2 to Rack 4 R1 IB2 P9B to R4 U1 P6B 10 meters R1 IB2 to Rack 5 R1 IB2 P10A to R5 U1 P7B 10 meters R1 IB2 to Rack 6 R1 IB2 P10B to R6 U1 P8B 10 meters R1 IB2 to Rack 7 R1 IB2 P11A to R7 U1 P9B 10 meters R1 IB2 to Rack 8 R1 IB2 P11B to R8 U1 P10B 10 meters Multi Rack Cabling Tables 2 29 Eight Rack Cabling 2 30 Table 2 29 shows the cable connections for the second spine switch R2 U1 when cabling eight full racks together Table 2 29 Leaf Switch Connections for the Second Rack in a Eight Rack System Table 2 30 shows the cable connections for the third spine switch R3 U1 when cabling eight full racks together Table 2 30 Leaf Switch Connections for the Third Rack in a Eight Rack System Leaf Switch Connection Cable Length R2 IB3 within Rack 2 R2 IB3 P8A to R2 U1 P3A 3 meters R2 IB3 to Rack 1 R2 IB3 P11B to R1 U1 P10A 5 meters R2 IB3 to Rack 3 R2 IB3 P8B to R3 U1 P4A 5 meters R2 IB3 to Rack 4 R2 IB3 P9A to R4 U1 P5A 5 meters R2 IB3 to Rack 5 R2 IB3 P9B to R5 U1 P6A 10 meters R2 IB3 to Rack 6 R2 IB3 P10A to R6 U1 P7A 10 meters R2 IB3 to Rack 7 R2 IB3 P10B to R7 U1 P8A 1
61. B2 P10A to R1 U1 P7B 10 meters R5 IB2 to Rack 2 R5 IB2 P10B to R2 U1 P8B 10 meters R5 IB2 to Rack 3 R5 IB2 P11A to R3 U1 P9B 5 meters R5 IB2 to Rack 4 R5 IB2 P11B to R4 U1 P10B 5 meters R5 IB2 to Rack 6 R5 IB2 P9A to R6 U1 P5B 5 meters R5 IB2 to Rack 7 R5 IB2 P9B to R7 U1 P6B 5 meters Table 2 26 shows the cable connections for the sixth spine switch R6 U1 when cabling seven full racks together Table 2 26 Leaf Switch Connections for the Sixth Rack in a Seven Rack System Leaf Switch Connection Cable Length R6 IB3 within Rack 6 R6 IB3 P8A to R6 U1 P3A 3 meters R6 IB3 P8B to R6 U1 P4A R6 IB3 to Rack 1 R6 IB3 P9B to R1 U1 P6A 10 meters R6 IB3 to Rack 2 R6 IB3 P10A to R2 U1 P7A 10 meters R6 IB3 to Rack 3 R6 IB3 P10B to R3 U1 P8A 10 meters R6 IB3 to Rack 4 R6 IB3 P11A to R4 U1 P9A 5 meters R6 IB3 to Rack 5 R6 IB3 P11B to R5 U1 P10A 5 meters R6 IB3 to Rack 7 R6 IB3 P9A to R7 U1 P5A 5 meters R6 IB2 within Rack 6 R6 IB2 P8A to R6 U1 P3B 3 meters R6 IB2 P8B to R6 U1 P4B R6 IB2 to Rack 1 R6 IB2 P9B to R1 U1 P6B 10 meters R6 IB2 to Rack 2 R6 IB2 P10A to R2 U1 P7B 10 meters R6 IB2 to Rack 3 R6 IB2 P10B to R3 U1 P8B 10 meters R6 IB2 to Rack 4 R6 IB2 P11A to R4 U1 P9B 5 meters R6 IB2 to Rack 5 R6 IB2 P11B to R5 U1 P10B 5 meters R6 IB2 to Rack 7 R6 IB2 P9A to R7 U1 P5B 5 meters Table 2 27 shows the cable connections for the seventh spine switch R7 U1 when cabling seven full racks together Mul
62. B2 P11B to R7 U1 P10B 10 meters Table 2 22 shows the cable connections for the second spine switch R2 U1 when cabling seven full racks together Table 2 22 Leaf Switch Connections for the Second Rack in a Seven Rack System Leaf Switch Connection Cable Length R2 IB3 within Rack 2 R2 IB3 P8A to R2 U1 P3A 3 meters R2 IB3 P8B to R2 U1 P4A R2 IB3 to Rack 1 R2 IB3 P11B to R1 U1 P10A 5 meters R2 IB3 to Rack 3 R2 IB3 P9A to R3 U1 P5A 5 meters R2 IB3 to Rack 4 R2 IB3 P9B to R4 U1 P6A 5 meters R2 IB3 to Rack 5 R2 IB3 P10A to R5 U1 P7A 10 meters R2 IB3 to Rack 6 R2 IB3 P10B to R6 U1 P8A 10 meters R2 IB3 to Rack 7 R2 IB3 P11A to R7 U1 P9A 10 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B 3 meters R2 IB2 P8B to R2 U1 P4B R2 IB2 to Rack 1 R2 IB2 P11B to R1 U1 P10B 5 meters R2 IB2 to Rack 3 R2 IB2 P9A to R3 U1 P5B 5 meters R2 IB2 to Rack 4 R2 IB2 P9B to R4 U1 P6B 5 meters R2 IB2 to Rack 5 R2 IB2 P10A to R5 U1 P7B 10 meters R2 IB2 to Rack 6 R2 IB2 P10Bto R6 U1 P8B 10 meters R2 IB2 to Rack 7 R2 IB2 P11A to R7 U1 P9B 10 meters Table 2 23 shows the cable connections for the third spine switch R3 U1 when cabling seven full racks together Table 2 23 Leaf Switch Connections for the Third Rack in a Seven Rack System Leaf Switch Connection Cable Length R3 IB3 within Rack 3 R3 IB3 P8A to R3 U1 P3A 3 meters R3 IB3 P8B to R3 U1 P4A R3 IB3 to Rack 1 R3 IB3 P11A to R1 U1 P9A 5 me
63. B2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 21 Leaf Switch Connections for the First Rack in a Seven Rack System Leaf Switch Connection Cable Length R1 IB3 within Rack 1 R1 IB3 P8A to R1 U1 P3A 3 meters R1 IB3 P8B to R1 U1 P4A R1 IB3 to Rack 2 R1 IB3 P9A to R2 U1 P5A 5 meters R1 IB3 to Rack 3 R1 IB3 P9B to R3 U1 P6A 5 meters R1 IB3 to Rack 4 R1 IB3 P10A to R4 U1 P7A 10 meters R1 IB3 to Rack 5 R1 IB3 P10B to R5 U1 P8A 10 meters R1 IB3 to Rack 6 R1 IB3 P11A to R6 U1 P9A 10 meters R1 IB3 to Rack 7 R1 IB3 P11B to R7 U1 P10A 10 meters R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 P8B to R1 U1 P4B 2 24 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Seven Rack Cabling Table 2 21 Cont Leaf Switch Connections for the First Rack in a Seven Rack System Leaf Switch Connection Cable Length R1 IB2 to Rack 2 R1 IB2 P9A to R2 U1 P5B 5 meters R1 IB2 to Rack 3 R1 IB2 P9B to R3 U1 P6B 5 meters R1 IB2 to Rack 4 R1 IB2 P10A to R4 U1 P7B 10 meters R1 IB2 to Rack 5 R1 IB2 P10B to R5 U1 P8B 10 meters R1 IB2 to Rack 6 R1 IB2 P11A to R6 U1 P9B 10 meters R1 IB2 to Rack 7 R1 I
64. ER DISKGROUP reco_diskgroup set attribute appliance mode FALSE SQL gt ALTER DISKGROUP dbfs_dg_diskgroup set attribute appliance mode FALSE In the preceding commands data_diskgroup reco_diskgroup and dbfs_dg_diskgroup are the names of the DATA RECO and DBFS_DG disk groups respectively 7 Addthegrid disks to the Oracle ASM disk groups Table 1 2 shows the commands to create the grid disks based on rack type and disk group Adding the new disks requires a rebalance of the system 1 16 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware Table 1 2 Commands to Add Disk Groups When Extending Oracle Exadata Database Machine X4 2 Eighth Rack or Later Rack Commands Extreme Flash Oracle SQL gt ALTER DISKGROUP data_diskgroup ADD DISK 0 DATA_FD_0 4 7 Exadata Database Machine X5 2 REBALANCE POWER 32 SQL gt ALTER DISKGROUP reco_diskgroup ADD DISK 0 RECO_FD_0 4 7 REBALANCE POWER 32 SQL gt ALTER DISKGROUP dbfs_dg_diskgroup ADD DISK 0 DBFS_DG_FD_0 4 7 REBALANCE POWER 32 High Capacity Oracle Exadata Database Machine X5 2 or Oracle Exadata Database Machine X4 2 SQL gt ALTER DISKGROUP data_diskgroup ADD DISK o DATA_CD_0 6 9 0 DATA CD_1 0 1 REBALANCE POWER 32 SQL gt ALTER DISKGROUP reco_diskgroup ADD DISK o RECO_CD_0 6 9 N o RECO_CD_1 0 1 REBALANCE POWER 32 SQL gt ALTER DISKGROUP dbfs_dg diskgroup ADD DISK
65. If you are extending Oracle Exadata Database Machine X4 2 Oracle Exadata Database Machine X3 8 Full Rack or Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers half rack then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch Figure 1 1 shows the components in the server expansion kit Figure 1 1 Server Components for Upgrade 3 Lay out the cables for the servers Extending Oracle Exadata Database Machine 1 9 Extending the Hardware Unroll the cables and stretch them to remove the bends Apply the cable labels Oracle recommends labeling all cables before installation See Also Oracle Exadata Database Machine Maintenance Guide for information about cable labels Install the servers as described in Adding New Servers on page 1 28 Cable the servers as described in Cabling Database Servers on page 1 31 and Cabling Exadata Storage Servers on page 1 35 Extending the Hardware Oracle Exadata Database Machine can be extended from Oracle Exadata Database Machine Quarter Rack to Oracle Exadata Database Machine Half Rack from Oracle Exadata Database Machine Half Rack to Oracle Exadata Database Machine Full Rack and by cabling racks together The following sections describe how to extend the hardware Extending an Eighth Rack to a Quarter Rack in Oracle Exadata Database Machine X4 2 and Later Extending an Eighth Rack to a Quarter Rack in Oracle Exadata Database Ma
66. Multi Rack Cabling Guide Extending the Hardware Table 1 3 Commands to Create Disk Groups When Extending Oracle Exadata Database Machine X3 2 Eighth Rack Rack Commands High dcli g cell group 1 celladmin cellcli e create griddisk Performance DATA_CD_06_ hostname s celldisk CD_06_ hostname s size datasize Oracle Exadata dcli g cell_group 1 celladmin cellcli e create griddisk Database Machine X3 2 DATA CD _07_ hostname deli g cell_group 1 DATA _CD_08_ hostname dcli g cell group 1 DATA _CD_09_ hostname deli g cell_group 1 DATA CD 10 hostname deli g cell group 1 DATA CD 11 hostname deli g cell_group 1 RECO_CD_06_ hostname dcli g cell_group 1 RECO_CD_07_ hostname deli g cell group 1 RECO_CD_08_ hostname dcli g cell group 1 RECO_CD_09_ hostname dcli g cell_group 1 RECO CD_10_ hostname dcli g cell group 1 RECO CD_11_ hostname dcli g cell_group 1 s celldisk CD_07_ celladmin cellcli e s celldisk CD 08 celladmin cellcli e s celldisk CD_09_ celladmin cellcli e sN celldisk CD 10_N celladmin cellcli e sN celldisk CD 11_ celladmin cellcli e sN celldisk CD 06_N celladmin cellcli e sN celldisk CD 07_N celladmin cellcli e sN celldisk CD_08_ celladmin cellcli e sN celldisk CD_09_ celladmin cellcli e sN celldisk CD 10_N celladmin cellcli e
67. ORACLE Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide 12c Release 1 12 1 E51952 10 August 2015 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide 12c Release 1 12 1 E51952 10 Copyright 2008 2015 Oracle and or its affiliates All rights reserved Primary Author Caroline Johnston Contributors Doug Archambault Leo Agranonik Andrew Babb Bharat Baddepudi Michael Belef Nilesh Choudhury Henry Chow Kevin Closson Bob Cyphers Ravindra Dani Boris Erlikhman Jaime Figueroa Gurmeet Goindi Roger Hansen Shrikumar Hariharasubrahmanian Kevin Jernigan Sarat Karkarla Leslie Keller Frank Kobylanski Poojan Kumar Ren Kundersma Robert S Lee Sue Lee Jennifer Leung Yang Liu Juan Loaiza Barb Lundhild Catherine Luu Varun Malhotra Bagish Mishra Valarie Moore Philip Newlan Adrian Ng Dan Norris Michael Nowak Bharat Paliwal Umesh Panchaksharaiah Hector Pujol Sugam Pandey Dmitry Potapov Darryl Presley Vince Pulice Ashish Ray Samitha Samaranayake Richard Scales Olly Sharwood Jia Shi Kesavan Srinivasan Mahesh Subramaniam Michelle Sun Krishnadev Telikicherla Cliff Thomas Alex Tsukerman Shreyas Udgaonkar Kothanda Umamageswaran Doug Utzig James Viscusi Vern Wagman Zheren Zhang This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws Except as expr
68. ORACLE_HOME directory to the servers 1 Check the ORACLE_HOME bin directory for files ending in zero 0 such as nmb0 that are owned by the root user and do not have oinstall or world read privileges Use the following command to modify the file privileges chmod a r SORACLE HOME bin 0 2 Create the ORACLE_BASE directory for the database owner if it is different from the Grid Infrastructure owner using the following commands dcli g root new_group_files dbs_group 1 root mkdir p u01 app oracle dcli g root new group_files dbs_group 1 root chown oracle oinstall N u01 app oracle 3 Run the following command to set ownership of the emocmrsp file in the Oracle Database ORACLE_HOME directory deli g old_db_nodes 1 root chown f oracle dba N u01 app 11 2 0 grid OPatch ocm bin emocmrsp 4 Create a response file add db nodes rsp as the oracle owner to add the new servers similar to the following RESPONSEFILE VERSION 2 2 1 0 0 1 56 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware CLUSTER_NEW_NODES dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 i dm02db06 dm02db07 dm02db08 Note The lines listing the server names should appear on one continuous line The are wrapped in the document due to page limitations Add the Oracle Database ORACLE_HOME directory to the new servers by running the addNode sh script from an existing server as the databa
69. Quarter Rack or Oracle Exadata Database Machine Eighth Rack to another rack Extending an Oracle Exadata Database Machine X4 2 rack to another rack Note The steps in this procedure are specific to Oracle Exadata Database Machine They are not the same as the steps in the Sun Datacenter InfiniBand Switch 36 manual See Also Oracle Exadata Database Machine System Overview to view the rack layout 1 Unpack the Sun Datacenter InfiniBand Switch 36 switch components from the packing cartons The following items should be in the packing cartons Sun Datacenter InfiniBand Switch 36 switch Cable bracket and rackmount kit Cable management bracket and cover 1 26 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware 9 Di oe e 10 11 12 13 14 15 16 17 Two rack rail assemblies Assortment of screws and captive nuts Sun Datacenter InfiniBand Switch 36 documentation The service label procedure on top of the switch includes descriptions of the preceding items X5 racks only Remove the trough from the rack in RU1 and put the cables aside while installing the IB switch The trough can be discarded Install cage nuts in each rack rail in the appropriate holes Attach the brackets with cutouts to the power supply side of the switch Attach the C brackets to the switch on the side of the InfiniBand ports Slide the switch halfway into the rack
70. R1 IB3 P8A to R1 U1 P3A 3 meters R1 IB3 P8B to R1 U1 P4A R1 IB3 to Rack 2 R1 IB3 P9A to R2 U1 P5A 5 meters R1 IB3 P9B to R2 U1 P6A R1 IB3 to Rack 3 R1 IB3 P10A to R3 U1 P7A 5 meters 2 20 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Six Rack Cabling Table 2 15 Cont Leaf Switch Connections for the First Rack in a Six Rack System Leaf Switch Connection Cable Length R1 IB3 to Rack 4 R1 IB3 P10B to R4 U1 P8A 10 meters R1 IB3 to Rack 5 R1 IB3 P11A to R5 U1 P9A 10 meters R1 IB3 to Rack 6 R1 IB3 P11B to R6 U1 P10A 10 meters R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 P8B to R1 U1 P4B R1 IB2 to Rack 2 R1 IB2 P9A to R2 U1 P5B 5 meters R1 IB2 P9B to R2 U1 P6B R1 IB2 to Rack 3 R1 IB2 P10A to R3 U1 P7B 5 meters R1 IB2 to Rack 4 R1 IB2 P10B to R4 U1 P8B 10 meters R1 IB2 to Rack 5 R1 IB2 P11A to R5 U1 P9B 10 meters R1 IB2 to Rack 6 R1 IB2 P11B to R6 U1 P10B 10 meters Table 2 16 shows the cable connections for the second spine switch R2 U1 when cabling six full racks together Table 2 16 Leaf Switch Connections for the Second Rack in a Six Rack System Leaf Switch Connection Cable Length R2 IB3 within Rack 2 R2 IB3 P8A to R2 U1 P3A 3 meters R2 IB3 P8B to R2 U1 P4A R2 IB3 to Rack 1 R2 IB3 P11B to R1 U1 P10A 5 meters R2 IB3 to Rack 3 R2 IB3 P9A to R3 U1 P5A 5 meters R2 IB3 P9B to R3 U1 P6A R2 IB3 to Rack 4 R2 IB3 P10A to R4 U1 P7A
71. U1 P10B 10 meters Multi Rack Cabling Tables 2 17 Five Rack Cabling Table 2 11 shows the cable connections for the second spine switch R2 U1 when cabling five full racks together Table 2 11 Leaf Switch Connections for the Second Rack in a Five Rack System Leaf Switch Connection Cable Length R2 IB3 within Rack 2 R2 IB3 P8A to R2 U1 P3A 3 meters R2 IB3 P8B to R2 U1 P4A R2 IB3 to Rack 1 R2 IB3 P11B to R1 U1 P10A 5 meters R2 IB3 to Rack 3 R2 IB3 P9A to R3 U1 P5A 5 meters R2 IB3 P9B to R3 U1 P6A R2 IB3 to Rack 4 R2 IB3 P10A to R4 U1 P7A 5 meters R2 IB3 P10B to R4 U1 P8A R2 IB3 to Rack 5 R2 IB3 P11A to R5 U1 P9A 10 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B 3 meters R2 IB2 P8B to R2 U1 P4B R2 IB2 to Rack 1 R2 IB2 P11B to R1 U1 P10B 5 meters R2 IB2 to Rack 3 R2 IB2 P9A to R3 U1 P5B 5 meters R2 IB2 P9B to R3 U1 P6B R2 IB2 to Rack 4 R2 IB2 P10A to R4 U1 P7B 5 meters R2 IB2 P10B to R4 U1 P8B R2 IB2 to Rack 5 R2 IB2 P11A to R5 U1 P9B 10 meters Table 2 12 shows the cable connections for the third spine switch R3 U1 when cabling five full racks together Table 2 12 Leaf Switch Connections for the Third Rack in a Five Rack System Leaf Switch Connection Cable Length R3 IB3 within Rack 3 R3 IB3 P8A to R3 U1 P3A 3 meters R3 IB3 P8B to R3 U1 P4A R3 IB3 to Rack 1 R3 IB3 P11A to R1 U1 P9A 5 meters R3 IB3 to Rack 2 R3 IB3 P11B to R2 U1 P10A
72. X3 2 and Oracle Exadata Database Machine X3 8 Full Rack m Cabling Tables for Oracle Exadata Database Machine X2 2 and Oracle Exadata Database Machine X2 8 Full Rack m Cabling Tables for Oracle Exadata Storage Expansion Rack 1 Set the priority of the current active Subnet Manager Master to 10 on the spine switch as follows a Log in to any InfiniBand switch on the active system b Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch If it is not then follow the procedure in Oracle Exadata Database Machine Installation and Configuraton Guide c Log in to the spine switch 1 38 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware A O p N 9 10 11 12 13 14 15 16 17 18 19 d Use the disablesm command to stop Subnet Manager e Use the setsmpriority 10 command to set the priority to 10 f Usethe enablesm command to restart Subnet Manager g Repeat step b to ensure the Subnet Manager Master is running on the spine switch Ensure the new rack is near the existing rack The InfiniBand cables must be able to reach the servers in each rack Completely shut down the new rack R2 Cable the leaf switch in the new rack according to Table 2 2 Power off leaf switch R1 IB2 This causes all the database servers and Exadata Storage Servers to fail over their InfiniBand traffic to R1 IB3 Disconnect all inter
73. _diskgroup set attribute appliance mode FALSE SQL gt ALTER DISKGROUP reco_diskgroup set attribute appliance mode FALSE SQL gt ALTER DISKGROUP dbfs_dg_diskgroup set attribute appliance mode FALSE In the preceding commands data_diskgroup reco_diskgroup and dbfs_dg_diskgroup are the names of the DATA RECO and DBFS_DG disk groups respectively 7 Addthegrid disks to the Oracle ASM disk groups Table 1 4 shows the commands to create the grid disks based on rack type and disk group Adding the new disks requires a rebalance of the system 1 22 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware Table 1 4 Commands to Add Disk Groups When Extending an Oracle Exadata Database Machine X3 2 Eighth Rack Rack Commands Extreme Flash Oracle SQL gt ALTER DISKGROUP data_diskgroup ADD DISK 0 DATA_FD_0 4 7 Exadata Database Machine X5 2 REBALANCE POWER 32 SQL gt ALTER DISKGROUP reco_diskgroup ADD DISK 0 RECO_FD_0 4 7 REBALANCE POWER 32 SQL gt ALTER DISKGROUP dbfs_dg_diskgroup ADD DISK 0 DBFS_DG_FD_0 4 7 REBALANCE POWER 32 High Capacity Oracle Exadata Database Machine X5 2 or Oracle Exadata Database Machine X4 2 SQL gt ALTER DISKGROUP data_diskgroup ADD DISK o DATA_CD_0 6 9 0 DATA CD 1 0 1 REBALANCE POWER 32 SQL gt ALTER DISKGROUP reco_diskgroup ADD DISK o RECO_CD_0 6 9 o RECO_CD_1 0 1 REBALANCE POWER
74. a given database may need more storage for its data files and storage servers can automatically provision the space Alternatively additional storage can be explicitly assigned by the storage administrator storage realm A collection of cells and attached grid disks that are associated with a group of Oracle ASM instances in a cluster The clustered Oracle ASM instances manage the related set of disk groups Glossary 3 Storage realm Glossary 4 A adding Sun Datacenter InfiniBand Switch 36 switch 1 26 addNode sh script 1 54 B backslash character 1 56 bend radius requirements 1 5 C cable management arm 1 5 cabling bend radius 1 5 bundling InfiniBand 1 5 estimating cable length 1 4 InfiniBand precautions 1 4 multi rack 1 38 2 1 new database servers 1 31 new Exadata Storage servers 1 35 quarter racks 2 8 capacity on demand 1 44 cellip ora file 1 50 checking cell configuration 1 6 cell disk configuration 1 6 firmware version 1 6 flash logs 1 6 HugePages memory configuration 1 image history 1 6 InfiniBand switch information 1 6 IORM plans 1 6 IP addresses 1 6 CMA guidelines 1 5 configuring cell alerts 1 55 CSI 1 3 1 10 Customer Support Identifier 1 3 1 10 6 E Exachk utility 1 6 expansion kit when to order 1 9 extending Index cellip ora file 1 50 eighth rack in X3 2 or earlier racks 1 18 eighth rack in X4 2 or later racks 1 10 F files add cluster
75. achine X4 2 Eighth Rack or Later The grid disks created in Creating Grid Disks in Oracle Exadata Database Machine X4 2 Eighth Rack or Later must be added as Oracle ASM disks to their corresponding existing Oracle ASM disk groups The following procedure describes how to add the grid disks to Oracle ASM disk groups 1 Validate the following No rebalance operation is currently running x All Oracle ASM disks are active 2 Log in to the first database server as the owner who runs the Grid Infrastructure software 3 Set the environment to access the ASM instance on the server 4 Login to the ASM instance as the sysasm user using the following command S sqlplus as sysasm 5 Validate the current settings as follows SQL gt set lines 100 SQL gt column attribute format a20 SQL gt column value format a20 SQL gt column diskgroup format a20 SQL gt SELECT att name attribute upper att value value dg name diskgroup FROM VSASM_ ATTRIBUTE att VSASM DISKGROUP DG WHERE DG group_number att group_number AND att name LIKE appliance mode ORDER BY att group_number The output should be similar to the following ATTRIBUTE VALUE DISKGROUP appliance mode TRUE DATAC1 appliance mode TRUE DBFS_DG appliance mode TRUE RECOC1 6 Disable the appliance mode attribute for any disk group that shows TRUE using the following commands SQL gt ALTER DISKGROUP data_diskgroup set attribute appliance mode FALSE SQL gt ALT
76. achine Extending and Multi Rack Cabling Guide Extending the Hardware b Locate the levers on the inside of each slide rail just behind the back panel of the server They are labeled PUSH c Simultaneously push in both levers and slide the server into the rack until it stops in approximately 46 cm 18 inches d Verify that the cables and CMA retract without binding e Locate the slide rail release buttons near the front of each mounting bracket f Simultaneously push in both slide rail release buttons and slide the server completely into the rack until both slide rails engage 7 Dress the cables and then tie off the cables with the straps Oracle recommends that you dress the InfiniBand cables in bundles of eight or fewer 8 Slide each server out and back fully to ensure that the cables are not binding or catching 9 Repeat the procedure for all servers 10 Connect the power cables to the power distribution units PDUs Ensure the breaker switches are in the OFF position before connecting the power cables Do not plug the power cables into the facility receptacles now Closing the Rack The following procedure describes how to close the rack after installing new equipment 1 Replace the rack front and rear doors as follows a Retrieve the doors and place them carefully on the door hinges b Connect the front and rear door grounding strap to the frame c Close the doors d Optional Lock the doors The keys
77. ack database server is 12 If the number of cores on a database server configured as an eighth rack differs then contact Oracle Support Services Review the current configuration of the storage servers using the following command The expected output is TRUE deli g cell group 1 celladmin cellcli e LIST CELL attributes eighthrack Activating Database Server Cores in Oracle Exadata Database Machine X4 2 Eighth Rack or Later The following procedure describes how to activate the database server cores 1 2 Log in as the root user on the first database server Activate all the database server cores using the following dcli utility command on the database server group Extending Oracle Exadata Database Machine 1 11 Extending the Hardware 3 4 deli g db_group 1 root dbmcli e ALTER DBSERVER pendingCoreCount number_of_cores In the preceding command number_of_cores is the total number of cores to activate The value includes the existing core count and the additional cores to be activated The following command shows how to activate all the cores in Oracle Exadata Database Machine X5 2 Eighth Rack deli g db_group 1 root dbmcli e ALTER DBSERVER pendingCoreCount 36 Note The maximum number of total active cores for Oracle Exadata Database Machine X5 2 Eighth Rack is 36 The maximum number of total active cores for Oracle Exadata Database Machine X4 2 Eighth Rack is 24 Restart eac
78. acle Exadata Database Machine Installation and Configuraton Guide of this guide Verify the time is the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database servers Ensure the NTP settings are the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database servers Configure HugePages on the new servers to match the existing servers Ensure the InfiniBand and bonded client Ethernet interface names on the new database servers match the existing database servers Extending Oracle Exadata Database Machine 1 45 Configuring the New Hardware 11 Configure the rack as described in Oracle Exadata Database Machine Installation and Configuraton Guide You can use either the Oracle Exadata Deployment Assistant or Oracle Enterprise Manager Grid Control to configure the rack Notes Only run the Oracle Exadata Deployment Assistant up to the CreateGridDisks step then configure Exadata Storage Servers as described in Configuring Cells Cell Disks and Grid Disks with CellCLI in Oracle Exadata Storage Server Software User s Guide a When adding servers with 3 TB High Capacity HC disks to existing servers with 2TB disks it is recommended to follow the procedure in My Oracle Support note 1476336 1 to properly define the grid disks and disk groups At this point of setting up the rack it is only necessary to defi
79. age Server Software User s Guide 5 Reclaim disk space as described in Oracle Exadata Database Machine Installation and Configuraton Guide 6 Verify the time is the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database servers 7 Ensure the NIP settings are the same on the new servers as on the existing servers This check is performed for Exadata Storage Servers and database Servers 8 Configure HugePages on the new servers to match the existing servers 9 Goto Task 4 Setting User Equivalence to continue the hardware configuration Task 3 Setting Up a New Rack A new rack is configured at the factory However it is necessary to set up the network and configuration files for use with the existing rack Use the following procedure to configure the rack 1 10 Check the Exadata Storage Servers as described in Oracle Exadata Database Machine Installation and Configuraton Guide Check the database servers as described in Oracle Exadata Database Machine Installation and Configuraton Guide Perform the checks as described in Oracle Exadata Database Machine Installation and Configuraton Guide Verify the InfiniBand network as described in Oracle Exadata Database Machine Installation and Configuraton Guide Perform initial configuration as described in Oracle Exadata Database Machine Installation and Configuraton Guide Reclaim disk space as described in Or
80. agnostics When completing Oracle Exadata Deployment Assistant for the additional rack you are prompted for SCAN addresses However these SCAN addresses are not used because the SCAN address from the original rack are used Manually remove the new SCAN addresses from the generated installation files The software owner account names and group names as well as their identifiers must match the names and identifiers of the original rack Multi Rack Cabling Tables 2 5 Understanding Multi Rack Cabling a If the additional grid disks are used with existing disk groups then ensure the erid disk sizes for the new rack are the same as the original rack If the InfiniBand network consists of four or more racks cabled together then disable the Subnet Manager on the leaf switches a Verify the Master Subnet Manager is located on the spine switch a Oracle Exadata Database Machine Quarter Racks can be extended as follows Connect two Oracle Exadata Database Machine Quarter Racks together At least four of the six ports reserved for external connectivity are open on each leaf switch The six ports are 5B 6A 6B 7A 7B and 12A in each leaf switch Maintain the existing seven inter switch links between the leaf switches within each rack Connect the leaf switches between the racks with two links each using the ports reserved for external connectivity Connect one Oracle Exadata Database Machine Quarter Rack with one Oracle Exa
81. anagement applications It is not developed or intended for use in any inherently dangerous applications including applications that may create a risk of personal injury If you use this software or hardware in dangerous applications then you shall be responsible to take all appropriate fail safe backup redundancy and other measures to ensure its safe use Oracle Corporation and its affiliates disclaim any liability for any damages caused by use of this software or hardware in dangerous applications Oracle and Java are registered trademarks of Oracle and or its affiliates Other names may be trademarks of their respective owners Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International Inc AMD Opteron the AMD logo and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro Devices UNIX is a registered trademark of The Open Group This software or hardware and documentation may provide access to or information about content products and services from third parties Oracle Corporation and its affiliates are not responsible for and expressly disclaim all warranties of any kind with respect to third party content products and services unless otherwise set forth in an applicable agreement between you and Oracle Oracle Corporation and its affiliates will not be responsible for any loss
82. ances in the cluster Set the value of the initialization parameter cluster _database_instances in the spfile for each database to the total number of instances that will be in the cluster after adding the new servers The HugePages settings are correctly configured on the new servers to match the existing servers The following procedure describes how to add database instances to the new servers 1 Use a command similar the following from an existing database server to add instances to the new servers In the command the instance dbm9 is being added for server dm02db01 dbca silent addInstance gdbName dbm nodeList dm02db01 instanceName dbm9 sysDBAUSername sys Extending Oracle Exadata Database Machine 1 57 Returning the Rack to Service The command must be run for all servers and instances substituting the server name and instance name as appropriate Note If the command fails then ensure any files that were created such as redo log files are cleaned up The deleteInstance command does not clean log files or data files that were created by the addInstance command 2 Do the following procedure a Manually add the CLUSTER_INTERCONNECTS parameter to the spfile for each new Oracle instance The additions are similar to the existing entries but are the InfiniBand addresses corresponding to the server each instance runs on b Restart the instance on each new server c Verify the parameters were
83. and Patch Levels Performing Preliminary Checks Reviewing the Safety Precautions Before upgrading Oracle Exadata Database Machines read Important Safety Information for Sun Hardware Systems included with the rack Extending Oracle Exadata Database Machine 1 3 Preparing to Extend Oracle Exadata Database Machine Note Contact your service representative or Oracle Advanced Customer Services to confirm that Oracle has qualified your equipment for installation and use in Oracle Exadata Database Machine Oracle is not liable for any issues when you install or use non qualified equipment See Also a Oracle Exadata Database Machine Installation and Configuraton Guide for safety guidelines a Oracle Exadata Database Machine Safety and Compliance Guide for safety notices Reviewing the InfiniBand Cable Precautions Review the following InfiniBand cable precautions before working with the InfiniBand cables Fiber optic InfiniBand cables with laser transceivers must be of type Class 1 Do not allow any copper core InfiniBand cable to bend to a radius tighter than 127 mm 5 inches Tight bends can damage the cables internally Do not allow any optical InfiniBand cable to bend to a radius tighter than 85 mm 3 4 inches Tight bends can damage the cables internally Do not use zip ties to bundle or support InfiniBand cables The sharp edges of the ties can damage the cables internally Use hook and loop straps Do not
84. and config_sync_status are shown as Ok before continuing this procedure 3 Review the current configuration of the storage servers using the following command The expected output is TRUE deli g cell group 1 celladmin cellcli e LIST CELL attributes eighthrack 4 Ensure that flash disks are not used in Oracle ASM disk groups using the following command Flash cache is dropped and recreated during this procedure deli g cell group 1 celladmin cellcli e list griddisk attributes X asmDiskgroupName asmDiskName diskType where diskType FlashDisk and asmDiskgroupName null No rows should be returned by the command Activating Database Server Cores in Oracle Exadata Database Machine X3 2 Eighth Rack The following procedure describes how to activate the database server cores 1 Loginasthe root user on the first database server 2 Activate all the database server cores using the following dcli utility command on the database server group deli g db_group 1 root opt oracle SupportTools resourcecontrol core number_of_cores In the preceding command number_of_cores is the total number of cores to activate To activate all the cores enter A11 for the number of cores See Also a Oracle Exadata Database Machine Maintenance Guide for additional information about activating a subset of cores a Oracle Exadata Database Machine Licensing Information for information about licensing a subset of cores 3 Resta
85. ata Storage Server X4 2L Servers servers have two IP addresses each 3 Ensure the updated cellip ora file is on all database servers The updated file must include a complete list of all Exadata Storage Servers 4 Verify accessibility of all grid disks using the following command from one of the original database servers The command can be run as the root user or the oracle user S GRID _HOME grid bin kfod disks all dscvgroup true The output from the command shows grid disks from the original and new Exadata Storage Servers 5 Addthegrid disks from the new Exadata Storage Servers to the existing disk groups using commands similar to the following This step is only permitted when adding high performance disks to high performance disks or high capacity disks to high capacity disks oraenv ORACLE_SID oracle ASM1 The Oracle base for ORACLE_HOME u01 app 11 2 0 grid is u01 app oracle S sqlplus as sysasm SQL gt ALTER DISKGROUP data ADD DISK 2 gt 0 DATA dm02 3 gt rebalance power 11 In the preceding commands a full rack was added to an existing system The prefix for the new rack is dm02 and the grid disk prefix is DATA The following is an example in which Oracle Exadata Database Machine Half Rack was upgraded to Oracle Exadata Database Machine Full Rack The cell host names in the original system were named dm01cel01 through dm01ce107 The new cell host names are dm01cel08 through dm01ce114 0raenv
86. atabase Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 43 Leaf Switch Connections for the Sixteenth Rack in the System Leaf Switch Connection R16 IB3 to Rack 1 R16 IB3 P8B to R1 U1 POA R16 IB3 to Rack 2 R16 IB3 P9A to R2 U1 POA R16 IB3 to Rack 3 R16 IB3 P9B to R3 U1 POA R16 IB3 to Rack 4 R16 IB3 P10A to R4 U1 POA R16 IB3 to Rack 5 R16 IB3 P10B to R5 U1 POA R16 IB3 to Rack 6 R16 IB3 POA to R6 U1 POA R16 IB3 to Rack 7 R16 IB3 POB to R7 U1 POA R16 IB3 to Rack 8 R16 IB3 P8A to R8 U1 POA R16 IB2 to Rack 1 R16 IB2 P8B to R1 U1 P0B R16 IB2 to Rack 2 R16 IB2 P9A to R2 U1 POB R16 IB2 to Rack 3 R16 IB2 P9B to R3 U1 P0B R16 IB2 to Rack 4 R16 IB2 P10A to R4 U1 POB R16 IB2 to Rack 5 R16 IB2 P10B to R5 U1 POB R16 IB2 to Rack 6 R16 IB2 POA to R6 U1 POB R16 IB2 to Rack 7 R16 IB2 POB to R7 U1 POB 2 42 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Seventeen Rack Cabling Table 2 43 Cont Leaf Switch Connections for the Sixteenth Rack in the System R16 IB2 to Rack 8 R16 IB2 P8A to R8 U1 POB Seventeen Rack Cabling Table 2 44 show
87. ate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 37 Leaf Switch Connections for the Tenth Rack in the System Leaf Switch Connection R10 IB3 to Rack 1 R10 IB3 P8B to R1 U1 P12A R10 IB3 to Rack 2 R10 IB3 P9A to R2 U1 P12A R10 IB3 to Rack 3 R10 IB3 P9B to R3 U1 P12A R10 IB3 to Rack 4 R10 IB3 P10A to R4 U1 P12A R10 IB3 to Rack 5 R10 IB3 P10B to R5 U1 P12A R10 IB3 to Rack 6 R10 IB3 P12A to R6 U1 P12A R10 IB3 to Rack 7 R10 IB3 P12B to R7 U1 P12A R10 IB3 to Rack 8 R10 IB3 P8A to R8 U1 P12A R10 IB2 to Rac
88. ater 1 14 Commands to Add Disk Groups When Extending Oracle Exadata Database Machine X4 2 Eighth Rack or Later 1 17 Commands to Create Disk Groups When Extending Oracle Exadata Database Machine X3 2 Eighth Rack 1 21 Commands to Add Disk Groups When Extending an Oracle Exadata Database Machine X3 2 Eighth Rack 1 23 Leaf Switch Connections for the First Rack in a Two Rack System 2 11 Leaf Switch Connections for the Second Rack in a Two Rack System 2 11 Leaf Switch Connections for the First Rack in a Three Rack System 2 12 Leaf Switch Connections for the Second Rack in a Three Rack System 2 13 Leaf Switch Connections for the Third Rack in a Three Rack System 2 13 Leaf Switch Connections for the First Rack in a Four Rack System 2 14 Leaf Switch Connections for the Second Rack in a Four Rack System 2 15 Leaf Switch Connections for the Third Rack in a Four Rack System 2 15 Leaf Switch Connections for the Fourth Rack in a Four Rack System 2 16 Leaf Switch Connections for the First Rack in a Five Rack SyStem 2 17 Leaf Switch Connections for the Second Rack in a Five Rack System 2 18 Leaf Switch Connections for the Third Rack in a Five Rack System
89. base server A computer that runs client applications and databases It is connected to cells by some form of network such as Ethernet ILOM Integrated Lights Out Manager is a dedicated system of hardware and supporting software that allows you to manage a server independently of the operating system interdatabase plan Interdatabase resource plan manages multiple databases The plan is created using the CellCLI utility Integrated Lights Out Manager ILOM See ILOM Glossary 1 intradatabase plan Glossary 2 intradatabase plan See database resource plan I O Resource Management IORM See IORM IORM I O Resource Management allows workloads and databases to share Exadata Storage Servers according to user defined policies IP block An attribute of a cell The IP block is used with the cell number attribute to compute the range of IP addresses and the netmask assigned to the cell KVM switch Keyboard video and mouse switch is a hardware device that enables a user to control multiple computers from a single keyboard video monitor and mouse Management Server MS See MS MS Management Server an application server running on a cell device and exposing basic hardware configuration and management Network Interface Card NIC See NIC NIC Network interface card is a circuit board or card in a computer that enables computers to communicate over a network Oracle ASM Oracle Automatic Storage Mana
90. ches a My Oracle Support note 1284070 1 for information about upgrading the database servers Performing Preliminary Checks Perform a visual check of Oracle Exadata Database Machine physical systems before extending the hardware 1 Check the rack for damage Check the rack for loose or missing screws Check Oracle Exadata Database Machine for the ordered configuration Check that all cable connections are secure and well seated Check power cables 9 Di P oe N Ensure the correct connectors have been supplied for the data center facility power source 7 Check network data cables 1 8 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Preparing to Extend Oracle Exadata Database Machine 8 Check the site location tile arrangement for cable access and airflow 9 Check the data center airflow into the front of Oracle Exadata Database Machine Preparing to Add Servers Perform the following tasks before adding the servers 1 Unpack the Oracle Exadata Database Machine expansion kit 2 Unpack all Oracle Exadata Database Machine server components from the packing cartons The following items should be packaged with the servers Oracle Database servers or Exadata Storage Server Power cord packaged with country kit Cable management arm with installation instructions Rackmount kit containing rack rails and installation instructions Optional Sun server documentation and media kit Note
91. chine X3 2 Extending a Quarter Rack or Half Rack Extending a Rack by Adding Another Rack Note All new equipment receives a Customer Support Identifier CSI Any new equipment for Oracle Exadata Database Machine has anew CSI Contact Oracle Support Services to reconcile the new CSI with the existing Oracle Exadata Database Machine CSI Have the original instance numbers or serial numbers available as well as the new numbers when contacting Oracle Support Services Extending an Eighth Rack to a Quarter Rack in Oracle Exadata Database Machine X4 2 and Later Extending Oracle Exadata Database Machine X4 2 or later from an eighth rack to a quarter rack is done using software No hardware modifications are needed to extend the rack This procedure can be done with no downtime or outages other than a rolling database outage The following procedures in this section describe how to extend an Oracle Exadata Database Machine X4 2 or later eighth rack to a quarter rack Reviewing and Validating Current Configuration of Oracle Exadata Database Machine X4 2 Eighth Rack or Later Activating Database Server Cores in Oracle Exadata Database Machine X4 2 Eighth Rack or Later Activating Storage Server Cores and Disks in Oracle Exadata Database Machine X4 2 Eighth Rack or Later Creating Grid Disks in Oracle Exadata Database Machine X4 2 Eighth Rack or Later 1 10 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide E
92. cile the new CSI with the existing Oracle Exadata Database Machine CSI Have the original instance numbers or serial numbers available as well as the new numbers when contacting Oracle Support Services The InfiniBand network can be used for external connectivity The external connectivity ports in the Sun Datacenter InfiniBand Switch 36 switches can connect to media servers for tape backup data loading and client and application access Use the available ports on the leaf switches for external connectivity There are 12 ports per rack The available ports are 5B 6A 6B 7A 7B and 12A in each leaf switch For high availability connections connect one port to one leaf switch and the other port to the second leaf switch The validated InfiniBand cable lengths are as follows Up to 5 meters passive copper 4X QDR QSFP cables Up to 100 meters fiber optic 4X D QDR QSFP cables Preparing to Extend Oracle Exadata Database Machine Before extending any rack hardware review the safety precautions and cabling information and collect information about the current rack in this section This section contains the following topics Reviewing the Safety Precautions Reviewing the InfiniBand Cable Precautions Estimating InfiniBand Cable Path Lengths Bundling InfiniBand Cables Reviewing the Cable Management Arm Guidelines Obtaining Current Configuration Information Preparing the Network Configuration Moving Audit and Diagnostic Files Reviewing Release
93. ck in Oracle Exadata Database Machine X3 2 1 18 Extending a Quarter Rack Or HAER akren ta E O 1 24 Extending a Rack by Adding Another Rack 1 37 Confieurins the New Hardwat e si lilla 1 41 Returino the Rack To Service nn uccidi 1 58 2 Multi Rack Cabling Tables Understanding Multi Rack Cabling uio lella i 2 1 Preparing to Cable Racks Togelhera uu a Sp aa Qh E ias sa 2 5 Cabling Oracle Exadata Quarter Racks and Oracle Exadata Eighth Racks 2 8 Twoe Rack Cab u u alati a 2 10 PLCS Rack Ca Di 86 u u m a u nu nas iran digerire nia 2 12 Four RaekCablino cuii elle un u ahus 2 14 FivesRacheCablingu u suasana ha u aha laaya aaah qaku u sskasss 2 16 SI Qe CADIS u uu umanuan Sau D am au uma aaa 2 20 Seven RaekCablino a anale 2 24 Eisht RackCablinss lele eli dana 2 28 Nine Rack Gablinp ille atei 2 33 Ten Rack Caplio eenen A nta 2 35 ElevensRackCabline Rtl 2 36 Twelve RackCablino cina 2 37 Thirteen Rack Cablino crreaale ae 2 38 Fourteen Rack Cablins uu uu u a aasawa u aaa au asas 2 39 Hilleen Rack Cabling escrase N 2 40 Steen Back Ge 6 ib 8 ae ee Se m a errr ry 2 42 Seventeen RackGoblin gt srrirnn earn ri 2 43 EishteensRack CEablina i anelli 2 44 Glossary Index List of Tables vi 1 1 be der ai ol 1 AON MISNININNNNNNNNNINNNINNN O OO ND OQ ONS O I Commands to Create Disk Groups When Extending Oracle Exadata Database Machine X4 2 Eighth Rack or L
94. data Database Machine Half Rack or one Oracle Exadata Database Machine Full Rack At least four ports reserved for external connectivity are open on each leaf switch The spine switch in the Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack remains as the spine switch Maintain the existing seven inter switch links between the leaf switches within each rack Connect the leaf switches between the racks with two links each using the ports reserved for external connectivity Connect one Oracle Exadata Database Machine Quarter Rack with two or more Oracle Exadata Database Machine Half Racks or Oracle Exadata Database Machine Full Racks The racks are interconnected using a fat tree topology Connect each leaf switch in the quarter rack to the spine switch of each half rack or full rack using two links each The seven inter switch links between the leaf switches in the quarter rack are removed Note To connect more than one quarter rack to additional racks it is necessary to purchase Sun Datacenter InfiniBand Switch 36 switches for the quarter racks f you are extending Oracle Exadata Database Machine X4 2 Oracle Exadata Database Machine X3 8 Full Rack or Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers half rack then order the expansion kit that includes a Sun Datacenter InfiniBand Switch 36 switch When extending more than eighteen racks note the following To cable ni
95. e Database Server http docs oracle com cd E40591_01 htm1 E40317 index html Oracle Server X5 2 Service Manual http docs oracle com cd E41059_01 htm1 E48312 napsm html Oracle Server X5 2L Service Manual http docs oracle com cd E41033_01 htm1 E48325 cnpsm html Sun Datacenter InfiniBand Switch 36 User s Guide http docs oracle com cd E36265_01 index html Sun Flash Accelerator F20 PCIe Card User s Guide at http docs oracle com cd E19682 01 E21358 index html Sun Flash Accelerator F40 PCIe Card User s Guide at http docs oracle com cd E29748_01 htm1 E29741 index html Sun Flash Accelerator F80 PCIe Card User s Guide at http docs oracle com cd E41278_01 html E41251 index html Sun Storage 6 Gb SAS PCIe RAID HBA documentation at http docs oracle com cd E19221 01 E22410 1index html Oracle Integrated Lights Out Manager ILOM 3 0 Documentation at http docs oracle com cd E19860 01 index html m Cisco Catalyst 4948 switch Web site for information about MIBs at http www cisco com en US prod collateral switches ps5718 ps6021 produ ct_data_sheet0900aecd8017a72e html Cisco Catalyst 4948E F switch Web site for information about MIBs at http www cisco com en US prod collateral switches ps5718 ps6021 ps109 47 data_sheet_c78 598933 html a Avocent Web site for information about KVM switch MIBs at ftp ftp avocent com public product upgrades DS1800 MIB Files MPU_ Polaris TRAP MIB AVO_9 21 09 ASN Conventions The follow
96. e Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 28 Leaf Switch Connections for the First Rack in a Eight Rack System Leaf Switch Connection Cable Length R1 IB3 within Rack 1 R1 IB3 P8A to R1 U1 P3A 3 meters R1 IB3 to Rack 2 R1 IB3 P8B to R2 U1 P4A 5 meters R1 IB3 to Rack 3 R1 IB3 P9A to R3 U1 P5A 5 meters R1 IB3 to Rack 4 R1 IB3 P9B to R4 U1 P6A 10 meters R1 IB3 to Rack 5 R1 IB3 P10A to R5 U1 P7A 10 meters R1 IB3 to Rack
97. e straps Oracle recommends the InfiniBand cables should be dressed in bundles of eight or less Extend and then fully retract the server to check cable travel by sliding each server out and back fully to ensure that the cables are not binding or catching Repeat the procedure for the rest of the servers Connect the power cables to the power distribution units PDUs Ensure the breaker switches are in the OFF position before connecting the power cables Do not plug the power cables into the facility receptacles at this time Cabling Exadata Storage Servers After the new Exadata Storage Servers are installed they need to be cabled to the existing equipment The following procedure describes how to cable the new equipment in the rack 1 Notes The existing cable connections in the rack do not change a The blue cables connect to Oracle Database servers and the black cables connect to Exadata Storage Servers These network cables are for the NETO Ethernet interface port Attach and route the management cables on the CMA and rear panel one server at a time Do not slide out more than one server at a time Start from the bottom of the rack and work upward Longer hook and loop straps are needed when cabling three CAT5e cables or two TwinAx cables See Also Chapter 2 Multi Rack Cabling Tables Oracle Exadata Database Machine System Overview for the following cabling tables m Cabling Tables for Oracle E
98. ectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Multi Rack Cabling Tables 2 39 Fifteen Rack Cabling Table 2 41 Leaf Switch Connections for the Fourteenth Rack in the System Leaf Switch Connection R14 IB3 to Rack 1 R14 IB3 P8B to R1 U1 P16A R14 IB3 to Rack 2 R14 IB3 P9A to R2 U1 P16A R14 IB3 to Rack 3 R14 IB3 P9B to R3 U1 P16A R14 IB3 to Rack 4 R14 IB3 P10A to R4 U1 P16A R14 IB3 to Rack 5 R14 IB3 P10B to R5 U1 P16A R14 IB3 to Rack 6 R14 IB3 P16A to R6 U1 P16A R14 IB3 to Rack 7 R14 IB3 P16B to R7 U1 P16A R14 IB3 to Rack 8 R14 IB3 P8A to R8 U1 P16A R14 IB2 to Rack 1 R14 IB2 P8B to R1 U1 P16B R14 IB2 to Rack 2 R14 IB2 P9A to R2 U1 P16B R14 IB2 to Rack 3 R14 IB2 P9B to R3 U1 P16B R14 IB2 to Rack 4 R14 IB2 P10A to R4 U1 P16B R14 IB2 to Rack 5 R14 IB2 P10B to R5 U1 P16B R14 IB2 to Rack 6 R14 IB2 P16A to R6 U1 P16B R14 IB2 to Rack 7 R14 IB2 P16B to R7 U1 P16B R14 IB2 to Rack 8 R14 IB2 P8A to R8 U1 P16B Fifteen Rack Cabling Table 2 42 shows the cable connections for the switches when cabling fifteen racks together The cabling between racks 1 through 8 do not change 2 40 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Fifteen Rack Cabling Notes The spine switch is in U1 also referred to as IB1 for
99. eleted before extending a cluster Refer to My Oracle Support note 744213 1 If the installer runs out of memory The note describes how to edit the GRID_HOME oui ora param ini file and change the JRE MEMORY_OPTIONS parameter to Xms512m Xmx2048m Add the new servers by running the addNode sh script from an existing server as the Grid Infrastructure owner S cd SGRID_HOME oui bin S addNode sh silent responseFile path to add cluster nodes rsp Verify the grid disks are visible from each of the new database servers using the following command S GRID_HOME grid bin kfod disks all dscvgroup true 1 54 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware 15 16 17 18 19 20 Run the orainstRoot sh script as the root user when prompted using the dcli utility S dcli g new_db_nodes 1 root N u01 app oraInventory oralinstRoot sh Run the u01 app 11 2 0 grid root sh script on each server sequentially This simplifies the process and ensures that any issues can be clearly identified and addressed Note The node identifier is set in order of the nodes where the root sh script is run Typically the script is run from the lowest numbered node name to the highest Check the log file from the root sh script and verify there are no problems on the server before proceeding to the next server If there are problems then resolve them before continuing
100. end the rack anti tip bar to prevent the rack from tipping during equipment installation The following procedure describes how to install the sever 1 Read the service label on the top cover of the server before installing a server into the rack 2 Push the server into the slide rail assembly a Push the slide rails into the slide rail assemblies as far as possible b Position the server so the rear ends of the mounting brackets are aligned with the slide rail assemblies mounted in the equipment rack The callouts in the preceding image highlight the following 1 Mounting bracket inserted into slide rail 2 Slide rail release lever c Insert the mounting brackets into the slide rails and push the server into the rack until the mounting brackets encounter the slide rail stops approximately 30 cm 12 inches d Simultaneously push down and hold the slide rail release levers on each mounting bracket while pushing the server into the rack 1 30 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware e Continue pushing until the slide rail locks on the front of the mounting brackets engage the slide rail assemblies and you hear the click 3 Cable the new server as described in Cabling Exadata Storage Servers on page 1 35 Note Oracle recommends that two people push the servers into the rack one person to move the server in and out of the rack and another per
101. er att value value dg name diskgroup FROM VSASM_ATTRIBUTE att VSASM DISKGROUP DG WHERE DG group_number att group_number AND Extending Oracle Exadata Database Machine 1 17 Extending the Hardware att name LIKE tappliance mode ORDER BY DG group_number Validate the number of Oracle ASM disks using the following command SQL gt SELECT g name d failgroup d mode_status count FROM vSasm_diskgroup g vSasm_disk d WHERE d group_number g group_number N GROUP BY g name d failgroup d mode_status Extending an Eighth Rack to a Quarter Rack in Oracle Exadata Database Machine X3 2 Extending Oracle Exadata Database Machine X3 2 or earlier rack from an eighth rack to a quarter rack is done using software No hardware modifications are needed to extend the rack This procedure can be done with no downtime or outages other than a rolling database outage The following procedures in this section describe how to extend an Oracle Exadata Database Machine X3 2 eighth rack to a quarter rack Reviewing and Validating Current Configuration of Oracle Exadata Database Machine X3 2 Eighth Rack Activating Database Server Cores in Oracle Exadata Database Machine X3 2 Eighth Rack Activating Storage Server Cores and Disks in Oracle Exadata Database Machine X3 2 Eighth Rack Creating Grid Disks in Oracle Exadata Database Machine X3 2 Eighth Rack Adding Grid Disks to Oracle ASM Disk Groups in Oracle Exadata Database Machine X3 2 Eig
102. ers R8 IB2 to Rack 3 R8 IB2 P9B to R3 U1 P6B 10 meters R8 IB2 to Rack 4 R8 IB2 P10A to R4 U1 P7B 10 meters R8 IB2 to Rack 5 R8 IB2 P10B to R5 U1 P8B 10 meters R8 IB2 to Rack 6 R8 IB2 P11A to R6 U1 P9B 5 meters R8 IB2 to Rack 7 R8 IB2 P11B to R7 U1 P10B 5 meters Nine Rack Cabling Table 2 36 shows the cable connections for the switches when cabling nine racks together The cabling between racks 1 through 8 do not change Multi Rack Cabling Tables 2 33 Nine Rack Cabling Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respective
103. ers so less load is placed on the existing servers The following procedure describes how to add grid disk to existing Oracle ASM disk groups Notes tis assumed in the following examples that the newly installed Exadata Storage Servers have the same grid disk configuration as the existing Exadata Storage Servers and that the additional grid disks will be added to existing disk groups The information gathered about the current configuration should be used when setting up the grid disks Refer to Obtaining Current Configuration Information on page 1 6 for information about the existing grid disks Refer to Task 2 Setting Up New Servers on page 1 42 for information about configuring the grid disks a If the existing Exadata Cells have High Performance HP disks and you are adding Exadata Cells with High Capacity HC disks or the existing Exadata Cells have HC disks and you are adding Exadata Cell HP disks then you must place the new disks in new disk groups It is not permitted to mix HP and HC disks within the same disk group 1 Ensure the new Exadata Storage Servers are running the same version of software as Exadata Storage Servers already in use using the following command on the first database server dcli g dbs_group 1 root imageinfo ver Note If the software on the cells does not match then upgrade or patch the software to be at the same level This could be patching the existing servers or
104. ery of InfiniBand Cables on page 1 5 Connect the orange Ethernet cable to the KVM switch Connect the red and blue Ethernet cables to the Cisco switch Verify operation of the slide rails and CMA for each server as follows Note Oracle recommends that two people do this step One person to move the server in and out of the rack and another person to observe the cables and CMA a Slowly pull the server out of the rack until the slide rails reach their stops b Inspect the attached cables for any binding or kinks c Verify the CMA extends fully from the slide rails Push the server back into the rack as follows a Release the two sets of slide rail stops b Push in both levers simultaneously and slide the server into the rack The first stop in the set are levers located on the inside of each slide rail just behind the back panel of the server The levers are labeled PUSH The server slides approximately 46 cm 18 inches and stop c Verify the cables and CMA retract without binding d Simultaneously push or pull both slide rail release buttons and push the server completely into the rack until both slide rails engage The second stop 1 34 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware 19 20 21 22 in the set are the slide rail release buttons located near the front of each mounting bracket Dress the cables and then tie off the cables with th
105. es For example if rack Rn 1 was R4 then use Table 2 9 Leaf Switch Connections for the Fourth Rack in a Four Rack System 5 Complete the following procedure for each of the original racks a Power off leaf switch Rx IB2 This causes all the database servers and Exadata Storage Servers to fail over their InfiniBand traffic to Rx IB3 b Cable leaf switch Rx IB2 according to Chapter 2 Multi Rack Cabling Tables c Power on leaf switch Rx IB2 d Wait for three minutes for Rx IB2 to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show n 3 switches for IB1 IB2 and IB3 in racks R1 R2 Rn e Power off leaf switch Rx IB3 This causes all the database servers and Exadata Storage Servers to fail over their InfiniBand traffic to Rx IB2 f Cable leaf switch Rx IB3 according to Chapter 2 Multi Rack Cabling Tables Power on leaf switch Rx IB3 Wait for three minutes for Rx IB3 to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show n 3 switches for IB1 IB2 and IB3 in racks R1 R2 Rn All racks should now be rewired according to Chapter 2 Multi Rack Cabling Tables 1 40 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware 10 11 12 13 Power on all the InfiniBand switches in Rn 1 Wait for three minu
106. espectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 45 Leaf Switch Connections for the Eighteenth Rack in the System Leaf Switch Connection R18 IB3 to Rack 1 R18 IB3 P8B to R1 U1 P2A R18 IB3 to Rack 2 R18 IB3 P9A to R2 U1 P2A R18 IB3 to Rack 3 R18 IB3 P9B to R3 U1 P2A R18 IB3 to Rack 4 R18 IB3 P10A to R4 U1 P2A R18 IB3 to Rack 5 R18 IB3 P10B to R5 U1 P2A R18 IB3 to Rack 6 R18 IB3 P2A to R6 U1 P2A R18 IB3 to Rack 7 R18 IB3 P2B to R7 U1 P2A 2 44 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Eighteen Rack Cabling Table 2 45 Cont Leaf Switch Connections for the Eighteenth Rack in the System Leaf Switch Connection R18 IB3 to Rack 8 R18 IB3 P8A to R8 U1 P2A R18 IB2 to Rack 1 R18 IB2 P8B to R
107. essly permitted in your license agreement or allowed by law you may not use copy reproduce translate broadcast modify license transmit distribute exhibit perform publish or display any part in any form or by any means Reverse engineering disassembly or decompilation of this software unless required by law for interoperability is prohibited The information contained herein is subject to change without notice and is not warranted to be error free If you find any errors please report them to us in writing If this is software or related documentation that is delivered to the U S Government or anyone licensing it on behalf of the U S Government then the following notice is applicable U S GOVERNMENT END USERS Oracle programs including any operating system integrated software any programs installed on the hardware and or documentation delivered to U S Government end users are commercial computer software pursuant to the applicable Federal Acquisition Regulation and agency specific supplemental regulations As such use duplication disclosure modification and adaptation of the programs including any operating system integrated software any programs installed on the hardware and or documentation shall be subject to license terms and license restrictions applicable to the programs No other rights are granted to the U S Government This software or hardware is developed for general use in a variety of information m
108. eters R1 IB3 P9B to R2 U1 P6A R1 IB3 to Rack 3 R1 IB3 P10A to R3 U1 P7A 5 meters R1 IB3 P10B to R3 U1 P8A Machine Extending and Multi Rack Cabling Guide Four Rack Cabling Table 2 6 Cont Leaf Switch Connections for the First Rack in a Four Rack System Leaf Switch Connection Cable Length R1 IB3 to Rack 4 R1 IB3 P11A to R4 U1 P9A 10 meters R1 IB3 P11B to R4 U1 P10A R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 P8B to R1 U1 P4B R1 IB2 to Rack 2 R1 IB2 P9A to R2 U1 P5B 5 meters R1 IB2 P9B to R2 U1 P6B R1 IB2 to Rack 3 R1 IB2 P10A to R3 U1 P7B 5 meters R1 IB2 P10B to R3 U1 P8B R1 IB2 to Rack 4 R1 IB2 P11A to R4 U1 P9B 10 meters R1 IB2 P11B to R4 U1 P10B Table 2 7 shows the cable connections for the second spine switch R2 U1 when cabling four full racks together Table 2 7 Leaf Switch Connections for the Second Rack in a Four Rack System Leaf Switch Connection Cable Length R2 IB3 within Rack 2 R2 IB3 P8A to R2 U1 P3A 3 meters R2 IB3 P8B to R2 U1 P4A R2 IB3 to Rack 1 R2 IB3 P11A to R1 U1 P9A 5 meters R2 IB3 P11B to R1 U1 P10A R2 IB3 to Rack 3 R2 IB3 P9A to R3 U1 P5A 5 meters R2 IB3 P9B to R3 U1 P6A R2 IB3 to Rack 4 R2 IB3 P10A to R4 U1 P7A 5 meters R2 IB3 P10B to R4 U1 P8A R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B 3 meters R2 IB2 P8B to R2 U1 P4B R2 IB2 to Rack 1 R2 IB2 P11A to R1 U1 P9B 5 meters R2 IB2 P11B to R1 U1 P10B R2 IB2 to Rack 3 R2 IB2
109. f grid disk creation must follow the same sequence that was used during initial grid disks creation For a standard deployment using Oracle Exadata Deployment Assistant the order is DATA RECO and DBFS_DG Create all DATA grid disks first followed by the RECO grid disks and then the DBFS_DG grid disks The following procedure describes how to create the grid disks Note The commands shown in this procedure use the standard deployment grid disk prefix names of DATA RECO and DBFS_DG The sizes being checked are on cell disk 02 Cell disk 02 is used because the disk layout for cell disks 00 and 01 are different from the other cell disks in the server 1 Check the size of the grid disks using the following commands Each cell should return the same size for the grid disks starting with the same grid disk prefix deli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like DATA 02 deli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like RECO 02 deli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like DBFS DG 02 The sizes shown are used during grid disk creation 2 Create the grid disks for the disk groups using the sizes shown in step 1 Table 1 3 shows the commands to create the grid disks based on rack type and disk group 1 20 Oracle Exadata Database Machine Extending and
110. gement is the Oracle volume manager that supports software striping and mirroring of Oracle Database data using multiple disks Oracle Automatic Storage Management Oracle ASM See Oracle ASM resource consumer groups A group that categorizes user sessions according to resource requirements Resource consumer groups are different from user roles One database user can have different sessions assigned to different resource consumer groups Sessions are mapped to resource consumer groups through a series of hierarchical mapping rules For additional information about resource consumer groups see Oracle Database Administrator s Guide resource plan A plan that specifies how CPU and I O resources are to be distributed among various users resource consumer groups For additional information about resource plans see Oracle Database Administrator s Guide Storage realm Restart Server RS See RS RS Restart Server is a server that restarts the CELLSRV and MS services and monitors these services to verify when they need to be restarted Secure Shell SSH See SSH SSH Secure Shell is a network protocol that allows data to be exchanged over a secure channel between two computers storage pool A homogeneous collection of storage resources managed as a unit that supports multiple consumers for example databases storage provisioning Dynamic or static assignment of disk storage to the clients that need it For example
111. ghth Rack rack and one disk group for the drives in Oracle Exadata Storage Expansion Rack Multiple Oracle Exadata Database Machines can be run as separate environments and connect through the InfiniBand network If you are planning to use multiple Oracle Exadata Database Machines in this manner then note the following a All servers on the InfiniBand network must have a unique IP address When Oracle Exadata Database Machine is deployed the default InfiniBand network is 192 168 10 1 You must modify the IP addresses before re configuring the InfiniBand network Failure to do so causes duplicate IP addresses After modifying the network run the verify topology and infinicheck commands to verify the network is working properly You need to create a file that contains IP addresses for Exadata Storage Servers such as combined_cellip ora The following is an example of the commands cd opt oracle SupportTools ibdiagtools verify toplogy t fattree infinicheck c tmp combined_cellip ora b a When Oracle Exadata Database Machines run in separate clusters do not modify the cellip ora files The cellip ora file on a database server should only include the IP addresses for the Exadata Storage Servers used with that database server m Cells with disk types different from what is already installed can be added but the disk types cannot be mixed in the same Oracle ASM disk group For example if the existing disk groups all use high
112. gt opatch_oracle txt In the preceding commands GRID_HOME is the path for the Grid Infrastructure home directory and ORACLE_HOME is the path for the Oracle home directory Preparing the Network Configuration When adding additional servers or rack to an existing rack the IP addresses for the new servers are obtained using Oracle Exadata Deployment Assistant If adding additional servers to an existing rack then the application should only include the new servers If adding an additional rack then the new rack should use its own Oracle Exadata Deployment Assistant The exact Oracle ASM disk group configuration currently in use may not be reflected by the application This is not an issue as the grid disks and disk groups are configured manually All other items such as the Oracle home location and owner should be defined exactly as the existing configuration When adding Sun Server X4 2 Oracle Database Server Exadata Storage Server X4 2L Servers and Sun Server X4 8 Oracle Database Server the bonding configuration must match the existing servers in the rack The Oracle Exadata Deployment Assistant InfiniBand configuration page has an option to select the type of bonding Select the option for active active bonding or deselect the option for active passive bonding The configuration file generated by the application is used by Oracle Exadata Deployment Assistant After completing Oracle Exadata Deployment Assistant use the checkip sh and
113. h database server Note If this procedure is done in a rolling fashion with the database and Grid Infrastructure active then ensure the following before restarting the database server a All Oracle ASM grid disks are online a No Oracle ASM rebalance operations are active a Shut down the database and Grid Infrastructure in a controlled manner failing over services as needed Verify the following items on the database server after the restart completes and before proceeding to the next server a The database and Grid Infrastructure services are active a The number of active cores is correct Use the dbmcli e list dbserver attributes coreCount command to verify the number of cores See Also a Oracle Exadata Database Machine Maintenance Guide for additional information about activating a subset of cores a Oracle Exadata Database Machine Licensing Information for information about licensing a subset of cores Activating Storage Server Cores and Disks in Oracle Exadata Database Machine X4 2 Eighth Rack or Later The following procedure describes how to activate the storage server cores and disks 1 2 Log in as the root user on the first database server Activate the cores on the storage server group using the following command The command uses the dcli utility and runs the command as the celladmin user deli g cell group 1 celladmin cellcli e alter cell eighthRack false Create the cell disks using the
114. have the correct UID and GID values 5 Log in as the Grid Infrastructure owner on an existing host 6 Verify the Oracle Cluster Registry OCR backup exists using the following command ocrconfig showbackup 7 Verify that the additional database servers are ready to be added to the cluster using commands similar to following S cluvfy stage post hwos n N dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 dm02db06 dm02db07 dm02db08 SN 1 52 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware verbose S cluvfy comp peer refnode dm01db01 n dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 dm02db06 dm02db07 dm02db08 SN orainv oinstall osdba dba grep B 3 A 2 mismatched S cluvfy stage pre nodeadd n dm02db01 dm02db02 dm02db03 dm02db04 dm02db05 dm02db06 dm02db07 dm02db08 SN verbose fixup fixupdir home gi_owner_name fixup d In the preceding commands gi_owner_name is the name of the Grid Infrastructure owner dm02db01 through db02db06 are the new database servers and refnode is an existing database server Notes The second and third commands do not display output if the commands complete correctly An error about a voting disk similar to the following may be displayed ERROR PRVF 5449 Check of Voting Disk location 0 192 168 73 102 DATA _CD_00_dm01cel07 0 192 168 73 102 DATA_CD_00_dm01cel07 failed on the following nodes Check failed
115. he switch Note Overhead cabling details are not included in this guide For details on overhead cabling contact a certified service engineer Reviewing the Cable Management Arm Guidelines Review the following cable management arm CMA guidelines before routing the cables Remove all required cables from the packaging and allow cables to acclimate or reach operating temperature if possible This improves the ability to manipulate the cables Label both ends of each cable using a label stock that meets the ANSI TIA EIA 606 A standard if possible Begin the installation procedure in ascending order Only slide out one server at a time Sliding out more than one server can cause cables to drop cause problems when sliding the servers back Separate the installation by dressing cables with the least stringent bend radius requirements first The following bend radius requirements are based on EIA TIA 568 x standards and may vary from the manufacturer s requirements CAT5e UTP 4 x diameter of the cable or 1 inch 25 4 mm minimum bend radius AC power cables 4 x diameter of the cable or 1 inch 25 4 mm minimum bend radius TwinAx 5 x diameter of the cable or 1 175 inch 33 mm Quad Small Form factor Pluggable QSFP InfiniBand cable 6 x diameter of the cable or 2 inch 55 mm Extending Oracle Exadata Database Machine 1 5 Preparing to Extend Oracle Exadata Database Machine Fibe
116. hen cabling six racks together Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 15 Leaf Switch Connections for the First Rack in a Six Rack System Leaf Switch Connection Cable Length R1 IB3 within Rack 1
117. hine Extending and Multi Rack Cabling Guide Preparing to Extend Oracle Exadata Database Machine If you are routing the InfiniBand cable under the floor consider the height of the raised floor when calculating cable path length Bundling InfiniBand Cables When bundling InfiniBand cables in groups use hook and loop straps to keep cables organized If possible use color coordinated straps to help identify cables and their routing The InfiniBand splitter and 4X copper conductor cables are fairly thick and heavy for their length Consider the retention strength of the hook and loop straps when supporting cables Bundle as few cables as reasonably possible If the InfiniBand cables break free of their straps and fall free the cables might break internally when they strike the floor or from sudden changes in tension You can bundle the cables using many hook and loop straps Oracle recommends that no more than eight cables be bundled together Place the hook and loop straps as close together as reasonably possible for example one strap every foot 0 3 m If a cable breaks free from a strap then the cable cannot fall far before it is retained by another strap Floor and Underfloor Delivery of InfiniBand Cables Sun Datacenter InfiniBand Switch 36 switch accepts InfiniBand cables from floor or underfloor delivery Floor and underfloor delivery limits the tension in the InfiniBand cable to the weight of the cable for the rack height of t
118. hth Rack Validating New Oracle Exadata Database Machine X3 2 Half Rack Configuration Reviewing and Validating Current Configuration of Oracle Exadata Database Machine X3 2 Eighth Rack The following procedure describes how to review and validate the current configuration 1 2 Log in as the root user on the first database server Review the current configuration of the database servers using the following command deli g db_group 1 root opt oracle SupportTools resourcecontrol show The following is an example of the output from the command aAm01db01 INFO Validated hardware and OS Proceed dm01db01 dm0ldb01 system_bios_version 25010600 dm01db01 restore_status Ok dm0ldb01 config_sync_status Ok dm01db01 reset_to_defaults Off dm01db01 SHOW Number of cores active per socket 4 dm01db02 INFO Validated hardware and OS Proceed dm01db02 dm01db02 system_bios_version 25010600 dm01db02 restore_status Ok dm01db02 config sync status Ok dm01db02 reset_to_defaults Off dm01db02 SHOW Number of cores active per socket 4 1 18 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware Note The number of active cores in Oracle Exadata Database Machine X3 2 Eighth Rack database server is 4 If the number of cores on a database server configured as an eighth rack differs then contact Oracle Support Services Ensure the output for restore_status
119. id disks starting with the same grid disk prefix deli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like DATA 02 deli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like RECO 02 dcli g cell group 1 celladmin cellcli e list griddisk attributes name size where name like DBFS DG 02 The sizes shown are used during grid disk creation 2 Create the grid disks for the disk groups using the sizes shown in step 1 Table 1 1 shows the commands to create the grid disks based on rack type and disk group Extending Oracle Exadata Database Machine 1 13 Extending the Hardware Table 1 1 Commands to Create Disk Groups When Extending Oracle Exadata Database Machine X4 2 Eighth Rack or Later Rack Commands Extreme Flash dcli g cell group 1 celladmin cellcli e create griddisk Oracle DATA FD 04 hostname s celldisk FD_04 hostname s slze datasize Exadata Database Machine X5 2 1 14 deli g cell_group 1 DATA FD 05 _ hostname deli g cell group 1 DATA _FD_06_ hostname deli g cell_group 1 DATA FD 07_ hostname dcli g cell group 1 RECO_FD_04_ hostname deli g cell_group 1 RECO FD 05 hostname deli g cell group 1 RECO FD 06 _ hostname doli g cell group L RECO_FD_07_ hostname celladmin cellcli e s celldisk FD_05_ celladmin cellcli e
120. ighth Rack or Later Rack High Capacity Oracle Exadata Database Machine X5 2 or Oracle Exadata Database Machine X4 2 Commands dcli g cell_group 1 DATA CD _06_ hostname dcli g cell_group 1 DATA CD _07_ hostname dcli g cell group 1 DATA _CD_08_ hostname dcli g cell_group 1 DATA _CD_09_ hostname deli g cell group 1 DATA CD 10 hostname deli g cell_group 1 DATA CD 11 N hostname deli g cell group 1 RECO_CD_06_ hostname deli g cell group 1 RECO_CD_07_ hostname dcli g cell_group 1 RECO_CD_08_ hostname deli g cell group 1 RECO_CD_09_ hostname deli g cell_group 1 RECO CD_10_ hostname dcli g cell group 1 RECO CD_11_ hostname dcli g cell group 1 celladmin cellcli e s celldisk CD 06_N celladmin cellcli e sN celldisk CD 07_N celladmin cellcli e s celldisk CD 08_N celladmin cellcli e s celldisk CD 09_N celladmin cellcli e sN celldisk CD 10_N celladmin cellcli e s celldisk CD 11 _N celladmin cellcli e sN celldisk CD 06_N celladmin cellcli e sN celldisk CD 07_N celladmin cellcli e sN celldisk CD_08_ celladmin cellcli e s celldisk CD 09_N celladmin cellcli e s celldisk CD 10_N celladmin cellcli e sN celldisk CD 11 _N celladmin cellcli e create griddisk hostname s Size datasize create griddisk N hostname s size datasize c
121. ile on the database servers so that the entries for the InfiniBand network match The following is an example of the file entries before editing One set of entries must be changed to match the other set Found in X2 node net ipv4 neigh bondib0 locktime 0 net ipv4 conf bondib0 arp_ignore 1 net ipv4 conf bondib0 arp_accept 1 net ipv4 neigh bondib0 base_reachable_time_ms 10000 net ipv4 neigh bondib0 delay_first_probe_time 1 Found in V2 node net ipv4 conf bond0 arp_accept 1 net ipv4 neigh bond0 base reachable _time_ms 10000 net ipv4 neigh bond0 delay_first_probe_time 1 2 Save the changes to the sysctl conf file 3 Use the oifcfg utility to change the OCR configuration if the new names differ from what is currently in OCR The interface names for Exadata Storage Servers do not have to be changed 4 Continue configuring the new hardware as follows a If the hardware is new servers then go to Task 2 Setting Up New Servers to configure the servers If the hardware is a new rack then go to Task 3 Setting Up a New Rack to configure the rack See Also Oracle Exadata Database Machine Maintenance Guide for information about changing the InfiniBand network information Task 2 Setting Up New Servers New servers need to be configured when extending Oracle Exadata Database Machine Quarter Rack or Oracle Exadata Database Machine Half Rack The new servers do not have any configuration information and you cannot use O
122. ing text conventions are used in this document Convention Meaning boldface Boldface type indicates graphical user interface elements associated with an action or terms defined in text or the glossary italic Italic type indicates book titles emphasis or placeholder variables for which you supply particular values monospace Monospace type indicates commands within a paragraph URLs code in examples text that appears on the screen or text that you enter prompt The dollar sign prompt indicates a command run as the oracle user prompt The pound prompt indicates a command that is run as the root user xi xii 1 Extending Oracle Exadata Database Machine This chapter describes how to extend Oracle Exadata Database Machine It contains the following topics About Extending Oracle Exadata Database Machine Preparing to Extend Oracle Exadata Database Machine Extending the Hardware Configuring the New Hardware Returning the Rack to Service Note For ease of reading the name Oracle Exadata Rack is used when information refers to both Oracle Exadata Database Machine and Oracle Exadata Storage Expansion Rack See Also Chapter 2 Multi Rack Cabling Tables Oracle Exadata Database Machine System Overview for the following cabling tables m Cabling Tables for Oracle Exadata Database Machine X4 2 Cabling Tables for Oracle Exadata Database Machine X3 2 and Oracle Exadata Database Machine X3
123. k 1 R10 IB2 P8B to R1 U1 P12B R10 IB2 to Rack 2 R10 IB2 P9A to R2 U1 P12B R10 IB2 to Rack 3 R10 IB2 P9B to R3 U1 P12B R10 IB2 to Rack 4 R10 IB2 P10A to R4 U1 P12B R10 IB2 to Rack 5 R10 IB2 P10B to R5 U1 P12B R10 IB2 to Rack 6 R10 IB2 P12A to R6 U1 P12B R10 IB2 to Rack 7 R10 IB2 P12B to R7 U1 P12B Multi Rack Cabling Tables 2 35 Eleven Rack Cabling Table 2 37 Cont Leaf Switch Connections for the Tenth Rack in the System R10 IB2 to Rack 8 R10 IB2 P8A to R8 U1 P12B Eleven Rack Cabling Table 2 38 shows the cable connections for the switches when cabling eleven racks together The cabling between racks 1 through 8 do not change Notes a The spine switch is in U1 also referred to as IB1 for all racks Inthe following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X4 2 racks Oracle Exadata
124. l the upper part of the cable management arm Slide the server in rack location U2 back into the rack Install power cords to the InfiniBand switch power supply slots on the front Loosen the front screws to install the vented filler panel brackets Tighten the screws and snap on the vented filler panel in front of the switch Extending Oracle Exadata Database Machine 1 27 Extending the Hardware Adding New Servers Oracle Exadata Database Machine Quarter Rack can be upgraded to Oracle Exadata Database Machine Half Rack and Oracle Exadata Database Machine Half Rack can be upgraded to Oracle Exadata Database Machine Full Rack The upgrade process includes adding new servers cables and when upgrading to Oracle Exadata Database Machine X2 2 Full Rack Sun Datacenter InfiniBand Switch 36 switch A Oracle Exadata Database Machine Quarter Rack to Oracle Exadata Database Machine Half Rack upgrade consists of installing the following a Two Oracle Database servers Four Exadata Storage Servers a One Sun Datacenter InfiniBand Switch 36 switch for Oracle Exadata Database Machine X2 2 with Sun Fire X4170 M2 Oracle Database Server only m Associated cables and hardware A Oracle Exadata Database Machine Half Rack to Oracle Exadata Database Machine Full Rack upgrade consists of installing the following a Four Oracle Database servers Seven Exadata Storage Servers m Associated cables and hardware Notes
125. llcli e list iormplan detail gt 1orm_detail txt HugePages memory configuration on the database servers using the following command dcli g dbs_group l root cat proc meminfo grep HugePages gt hugepages txt InfiniBand switch information using the following command ibswitches gt ibswitches txt Firmware version of the Sun Datacenter InfiniBand Switch 36 switches using the nm2version command on each switch The following network files from the first database server in the rack etc resolv conf etc ntp conf etc network etc sysconfig network scripts ifcfg 1 6 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Preparing to Extend Oracle Exadata Database Machine w Any users user identifiers groups and group identifiers created for cluster managed services that need to be created on the new servers such as Oracle GoldenGate etc passwd etc group Output of current cluster status using the following command crsctl stat res t gt crs_stat txt Patch information from the Grid Infrastructure and Oracle homes using the following commands The commands must be run as Grid Infrastructure home owner and the Oracle home owner u01 app oracle product 11 2 0 dbhome_1 OPatch opatch lsinventory oh GRID_HOME detail all_nodes gt opatch_grid txt u01 app oracle product 11 2 0 dbhome_1 OPatch opatch lsinventory oh ORACLE_HOME detail all_nodes gt
126. ly Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 36 Leaf Switch Connections for the Ninth Rack in the System Leaf Switch Connection R9 IB3 to Rack 1 R9 IB3 P8B to R1 U1 P11A R9 IB3 to Rack 2 R9 IB3 P9A to R2 U1 P11A R9 IB3 to Rack 3 R9 IB3 P9B to R3 U1 P11A R9 IB3 to Rack 4 R9 IB3 P10A to R4 U1 P11A R9 IB3 to Rack 5 R9 IB3 P10B to R5 U1 P11A R9 IB3 to Rack 6 R9 IB3 P11A to R6 U1 P11A R9 IB3 to Rack 7 R9 IB3 P11B to R7 U1 P11A R9 IB3 to Rack 8 R9 IB3 P9A to R8 U1 P11A R9 IB2 to Rack 1 R9 IB2 P8B to R1 U1 P11B R9 IB2 to Rack 2 R9 IB2 P9A to R2 U1 P11B R9 IB2 to Rack 3 R9 IB2 P9B to R3 U1 P11B R9 IB2 to Rack 4 R9 IB2 P10A to R4 U1 P11B R9 IB2 to Rack 5 R9 IB2 P10B to R5 U1 P11B R9 IB2 to Rack 6 R9 IB2 P11A to R6 U1 P11B R9 IB2 to Rack 7 R9 IB2 P11B to R7 U1 P11B R9 IB2 to Rack 8 R9 IB2 P8A to R8 U1 P11B 2 34 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Ten Rack Cabling Ten Rack Cabling Table 2 37 shows the cable connections for the switches when cabling ten racks together The cabling between racks 1 through 8 do not change Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropri
127. mmand setssh Linux sh s c N h path_to_file dbs_group n N Extending Oracle Exadata Database Machine 1 47 Configuring the New Hardware In the preceding command path_to_file is the directory path for the dbs_group file The file contains the names for the existing and new servers Notes a For Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers systems use the setssh sh command to configure user equivalence tmay be necessary to temporarily change the permissions on the setssh Linux sh file to 755 for this step Change the permissions back to the original settings after completing this step 8 Add the known hosts using InfiniBand using the following command This requires all database servers are accessible by way of their InfiniBand interfaces S opt oracle SupportTools onecommand setssh Linux sh s c N h root group_files dbs_ib_ group n N 9 Verify equivalence using the following commands S deli g dbs_group 1 oracle date S dcli g dbs_ib group 1 oracle date If there is a Grid Infrastructure user then also run the preceding commands for that user substituting the Grid Infrastructure user name for the oracle user Task 5 Starting the Cluster The following procedure describes how to start the cluster if it was stopped earlier for cabling an additional rack Notes a Oracle recommends you start one server and let it come up fully before starting Oracle Clusterwa
128. ne the grid disks The disk groups are created after the cluster has been extended on to the new nodes a If the existing Exadata Cells have High Performance HP disks and you are adding Exadata Cells with High Capacity HC disks or the existing Exadata Cells have HC disks and you are adding Exadata Cell HP disks then you must place the new disks in new disk groups It is not permitted to mix HP and HC disks within the same disk group 12 Go to Task 4 Setting User Equivalence to continue the hardware configuration Task 4 Setting User Equivalence User equivalence can be configured to include all servers once the servers are online This procedure must be done before running the post cabling utilities The following procedure describes how to set user equivalence 1 Log in to each new server manually using SSH to verify that each server can accept logins and that the passwords are correct 2 Modify the dbs_group and cell_group files on all servers to include all servers as follows a Create the new directories on the first existing database server using the following commands mkdir root new_group_files mkdir root old_group_files mkdir root group_files b Copy the group files for the new servers to the root new_group_files directory c Copy the group files for the existing servers to the root old_group_files directory d Copy the group files for the existing servers to the root group_files directory
129. nerate the configuration files Prepare the servers as follows starting with the first database server of the new servers a Configure the servers as described in Preparing the Servers in chapter 2 of Oracle Exadata Storage Server Software User s Guide Note Oracle Exadata Deployment Assistant checks the performance level of Exadata Storage Servers so it is not necessary to check them using the CellCLI CALIBRATE command at this time b Create the cell disks and grid disks as described in Configuring Cells Cell Disks and Grid Disks with CellCLI in chapter 2 of Oracle Exadata Storage Server Software User s Guide c Create the flash cache and flash log as described in Creating Flash Cache and Flash Grid Disks in chapter 2 of Oracle Exadata Storage Server Software User s Guide Note When creating the flash cache enable write back flash cache Ensure the InfiniBand and bonded client Ethernet interface names are the same on the new database servers as on the existing database servers When using the same earlier style bonding names such as BONDO for the new database servers then update the opt oracle cellos cell conf file to reflect the correct bond names Extending Oracle Exadata Database Machine 1 43 Configuring the New Hardware 10 11 12 13 14 15 16 Note If the existing servers use BONDIBO as the InfiniBand bonding name then this step can be skipped
130. neteen to 36 racks requires two external Sun Datacenter InfiniBand Switch 36 switches These switches are the external spine switches Divide the racks into groups each having two to eight racks Each group should include related racks such as racks that run a common database or are part of a common cluster Connect the internal spine switch in each rack to the two external spine switches Connecting the racks can be done online Bandwidth to and from each group is 80 Gbs multiplied by the number of racks in the group Perform the following tasks before cabling racks together 2 6 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Understanding Multi Rack Cabling 1 Determine the number of racks that will be cabled together 2 Count the spare cables from the kit and existing inter switch cables Note Oracle Exadata Database Machine X4 2 racks and Oracle Exadata Database Machine X3 8 Full Racks with Exadata Storage Server X4 2L Servers do not include spare cables or a third Sun Datacenter InfiniBand Switch 36 switch To extend Oracle Exadata Database Machine X4 2 racks and Oracle Exadata Database Machine X3 8 Full Racks with Exadata Storage Server X4 2L Servers order cables and a Sun Datacenter InfiniBand Switch 36 switch For Oracle Exadata Racks earlier than Oracle Exadata Database Machine X4 2 no additional InfiniBand cables need to be purchased when connecting up to
131. ngth R4 IB3 within Rack 4 R4 IB3 P8A to R4 U1 P3A 3 meters R4 IB3 P8B to R4 U1 P4A R4 IB3 to Rack 1 R4 IB3 P9A to R1 U1 P5A 10 meters R4 IB3 P9B to R1 U1 P6A R4 IB3 to Rack 2 R4 IB3 P10A to R2 U1 P7A 5 meters R4 IB3 P10B to R2 U1 P8A R4 IB3 to Rack 3 R4 IB3 P11A to R3 U1 P9A 5 meters R4 IB3 P11B to R3 U1 P10A R4 IB2 within Rack 4 R4 IB2 P8A to R4 U1 P3B 3 meters R4 IB2 P8B to R4 U1 P4B R4 IB2 to Rack 1 R4 IB2 P9A to R1 U1 P5B 10 meters R4 IB2 P9B to R1 U1 P6B R4 IB2 to Rack 2 R4 IB2 P10A to R2 U1 P7B 5 meters R4 IB2 P10B to R2 U1 P8B R4 IB2 to Rack 3 R4 IB2 P11A to R3 U1 P9B 5 meters R4 IB2 P11B to R3 U1 P10B Five Rack Cabling Table 2 10 shows the cable connections for the first spine switch R1 U1 when cabling five racks together 2 16 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Five Rack Cabling Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and O
132. ngth R7 IB3 to Rack 4 R7 IB3 P10B to R4 U1 P8A 10 meters R7 IB3 to Rack 5 R7 IB3 P11A to R5 U1 P9A 5 meters R7 IB3 to Rack 6 R7 IB3 P11B to R6 U1 P10A 5 meters R7 IB3 to Rack 8 R7 IB3 P8B to R8 U1 P4A 5 meters R7 IB2 within Rack 7 R7 IB2 P8A to R7 U1 P3B 3 meters R7 IB2 to Rack 1 R7 IB2 P9A to R1 U1 P5B 10 meters R7 IB2 to Rack 2 R7 IB2 P9B to R2 U1 P6B 10 meters R7 IB2 to Rack 3 R7 IB2 P10A to R3 U1 P7B 10 meters R7 IB2 to Rack 4 R7 IB2 P10B to R4 U1 P8B 10 meters R7 IB2 to Rack 5 R7 IB2 P11A to R5 U1 P9B 5 meters R7 IB2 to Rack 6 R7 IB2 P11B to R6 U1 P10B 5 meters R7 IB2 to Rack 8 R7 IB2 P8B to R8 U1 P4B 5 meters Table 2 35 shows the cable connections for the eighth spine switch R8 U1 when cabling eight full racks together Table 2 35 Leaf Switch Connections for the Eighth Rack in a Eight Rack System Leaf Switch Connection Cable Length R8 IB3 within Rack 8 R8 IB3 P8A to R8 U1 P3A 3 meters R8 IB3 to Rack 1 R8 IB3 P8B to R1 U1 P4A 10 meters R8 IB3 to Rack 2 R8 IB3 P9A to R2 U1 P5A 10 meters R8 IB3 to Rack 3 R8 IB3 P9B to R3 U1 P6A 10 meters R8 IB3 to Rack 4 R8 IB3 P10A to R4 U1 P7A 10 meters R8 IB3 to Rack 5 R8 IB3 P10B to R5 U1 P8A 10 meters R8 IB3 to Rack 6 R8 IB3 P11A to R6 U1 P9A 5 meters R8 IB3 to Rack 7 R8 IB3 P11B to R7 U1 P10A 5 meters R8 IB2 within Rack 8 R8 IB2 P8A to R8 U1 P3B 3 meters R8 IB2 to Rack 1 R8 IB2 P8B to R1 U1 P4B 10 meters R8 IB2 to Rack 2 R8 IB2 P9A to R2 U1 P5B 10 met
133. nodes rsp 1 54 add db nodes rsp 1 56 H health check utility 1 58 InfiniBand commands nm2version 2 8 installation safety guidelines 1 3 installing Sun Datacenter InfiniBand Switch 36 switch 1 26 IP addresses cellip ora 1 50 L leaf switches 2 2 N nm2version command 2 8 O Oracle Exadata Database Machine HealthCheck utility 1 58 S spare cables 2 7 spine switches 2 1 U utilities Index 1 Exachk 1 6 health check 1 58 oifcfg 1 42 Index 2
134. o R1 U1 P4A R1 IB3 P9A to R1 U1 P5A R1 IB3 P9B to R1 U1 P6A Cable Length 3 meters R1 IB3 to Rack 2 R1 IB3 P10A to R2 U1 P7A R1 IB3 P10B to R2 U1 P8A R1 IB3 P11A to R2 U1 P9A R1 IB3 P11B to R2 U1 P10A 5 meters R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B R1 IB2 P8B to R1 U1 P4B R1 IB2 P9A to R1 U1 P5B R1 IB2 P9B to R1 U1 P6B 3 meters R1 IB2 to Rack 2 R1 IB2 P10A to R2 U1 P7B R1 IB2 P10B to R2 U1 P8B R1 IB2 P11A to R2 U1 P9B R1 IB2 P11B to R2 U1 P10B 5 meters Table 2 2 shows the cable connections for the second spine switch R2 U1 when cabling two full racks together Table 2 2 Leaf Switch Connections for the Second Rack in a Two Rack System Leaf Switch R2 IB3 within Rack 2 R2 IB3 to Rack 1 Connection R2 IB3 P8A to R2 U1 P3A R2 IB3 P8B to R2 U1 P4A R2 IB3 P9A to R2 U1 P5A R2 IB3 P9B to R2 U1 P6A R2 IB3 P10A to R1 U1 P7A R2 IB3 P10B to R1 U1 P8A R2 IB3 P11A to R1 U1 P9A R2 IB3 P11B to R1 U1 P10A Cable Length 3 meters 5 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B R2 IB2 P8B to R2 U1 P4B R2 IB2 P9A to R2 U1 P5B R2 I1B2 P9B to R2 U1 P6B 3 meters R2 IB2 to Rack 1 R2 IB2 P10A to R1 U1 P7B R2 IB2 P10B to R1 U1 P8B R2 IB2 P11A to R1 U1 P9B R2 IB2 P11B to R1 U1 P10B 5 meters Multi Rack Cabling Tables 2 11 Three Rack Cabling Three Rack Cabling 2 12 Table 2 3 shows the cable connections for the first spine switch R1 U1
135. on one of the spine switches After cabling the racks together proceed to Configuring the New Hardware to configure the racks Extending Oracle Exadata Database Machine 1 39 Extending the Hardware Cabling Several Racks Together The following procedure describes how to cable several racks together This procedure assumes that the racks are adjacent to each other In the procedure the existing racks are R1 R2 Rn the new rack is Rn 1 and the Subnet Manager Master is running on R1 IB1 1 Set the priority of the current active Subnet Manager Master to 10 on the spine switch as follows a Log in to any InfiniBand switch on the active system b Use the getmaster command to determine that the Subnet Manager Master is running on the spine switch If it is not then follow the procedure in Oracle Exadata Database Machine Installation and Configuraton Guide Log in to the spine switch o p Use the disablesm command to stop Subnet Manager e Use the setsmpriority 10 command to set the priority to 10 f Usethe enablesm command to restart Subnet Manager g Repeat step b to ensure the Subnet Manager Master is running on the spine switch 2 Ensure the new rack is near the existing rack The InfiniBand cables must be able to reach the servers in each rack 3 Completely shut down the new rack Rn 1 4 Cable the leaf switch in the new rack according to the appropriate table in Chapter 2 Multi Rack Cabling Tabl
136. ookup ctx acc amp id trs if you are hearing impaired Related Documentation In addition to the Oracle Exadata Database Machine documentation set the following guides contain hardware information for Oracle Exadata Database Machine a Oracle Exadata Database Machine System Overview a Oracle Exadata Database Machine Security Guide a Oracle Exadata Database Machine Installation and Configuraton Guide a Oracle Exadata Database Machine Safety and Compliance Guide a Oracle Exadata Database Machine Maintenance Guide Sun Fire X4170 X4270 and X4275 Servers Service Manual at http docs oracle com cd E19477 01 820 5830 13 index html Sun Fire X4170 M2 Server Service Manual at http docs oracle com cd E19762 01 E22369 02 index html Sun Fire X4270 M2 Server Service Manual at http docs oracle com cd E19245 01 E21671 index html Sun Fire X4800 Server Service Manual at http docs oracle com cd E19140 01 htm1 821 0282 index html Sun Server X2 8 Service Manual at http docs oracle com cd E20815 01 html1 E20819 index html Sun Server X3 2 Service Manual at http docs oracle com cd E22368_ 01 html E27242 gentextid 13047 html scrolltoc Sun Server X3 2L Service Manual at http docs oracle com cd E23393_ 01 html E27229 gentextid 14806 html scrolltoc Sun Server X4 2 Service Manual at http docs oracle com cd E36975_01 index html Sun Server X4 2L Service Manual at http docs oracle com cd E36974_01 index html Sun Server X4 8 Oracl
137. pine switch has 16 inter switch connections distributed over all leaf switches The leaf switches are not directly interconnected with other leaf switches and the spine switches are not directly interconnected with the other spine switches When cabling 9 to 18 racks the inter switch connections between the leaf switches are removed in racks 9 through 18 and the spine switches are disconnected from the InfiniBand fabric in racks 9 through 18 The 8 links from every leaf switch in racks 9 through 18 are connected to the spine switches in racks 1 to 8 as shown in the following graphic The connections between racks 1 to 8 do not change when cabling racks 9 to 18 2 4 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Understanding Multi Rack Cabling Rack 1 Rack 2 Rack 8 Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 II I Spine Switch Spine Switch Spine Switch i dl DS Rack 9 Rack 18 Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 Spine Switch Spine Switch Note Cable lengths are specified for racks 1 through 8 Cable lengths for racks 9 through 18 vary depending on the layout of the racks Preparing to Cable Racks Together Racks can be added together to increase system capacity and performance When cabling racks together note the following m Oracle recommends that the names for the servers include the rack unit number This helps identify the server during di
138. r Velcro straps 6 Connector B Connector C Connector D Slide rail latching bracket used with connector D Rear slide bar Cable covers 10 Cable covers 3 Attach the CMA to the server 4 Route the CAT5e and power cables through the wire clip 1 32 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware 5 Bend the CAT5e and power cables to enter the CMA while adhering to the bend radius minimums See Also Reviewing the Cable Management Arm Guidelines on page 1 5 for the bend radius minimums 6 Secure the CAT5e and power cables under the cable clasps 7 Route the cables through the CMA and secure them with hook and loop straps at equal intervals 8 Connect the InfiniBand or TwinAx cables with the initial bend resting on the CMA The TwinAx cables are for client access to the database servers Extending Oracle Exadata Database Machine 1 33 Extending the Hardware 9 Secure the InfiniBand or TwinAx cables with hook and loop straps at equal intervals 10 11 12 13 14 15 16 17 18 Route the fiber core cables Rest the InfiniBand cables over the green clasp on the CMA Attach the red ILOM cables to the database server Attach the network cables to the Oracle Database server Attach the InfiniBand cables from Oracle Database server to the Sun Datacenter InfiniBand Switch 36 switches See Also Floor and Underfloor Deliv
139. r core cable 10 x diameter of the cable or 1 22 inch 31 75 mm for a 0 125 cable Install the cables with the best longevity rate first Obtaining Current Configuration Information The current configuration information is used to plan patching requirements configure new IP addresses and so on The following information should be collected as described before extending the rack The Exachk report for the current rack Refer to My Oracle Support note 1070954 1 for information about the Exachk utility Image history information using the following command dcli g all_group 1 root imagehistory gt imagehistory txt Current IP addresses defined for all Exadata Storage Servers and database servers using the following command cli g all_group 1 root ifconfig gt ifconfig_all txt Information about the configuration of the cells cell disks flash logs and IORM plans using the following commands dcli g cell_group 1 root cellcli e list cell detail gt cell detall txt deli g cell_group 1 root cellcli e list physicaldisk detail gt physicaldisk _detail txt dcli g cell_ group 1 root cellcli e list griddisk attributes name offset size status asmmodestatus asmdeactivationoutcome gt griddisk txt dcli g cell_group 1 root cellcli e list flashcache detail gt fc_detall txt dcli g cell_group 1 root cellcli e list flashlog detail gt fl detail txt dcli g cell_group 1 root ce
140. racle Enterprise Manager Grid Control to configure them The servers are configured using the Oracle Exadata Deployment Assistant or manually Configuring Servers Using Oracle Exadata Deployment Assistant Use the following procedure to configure the servers using Oracle Exadata Deployment Assistant 1 42 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware Note In order to configure the servers with Oracle Exadata Deployment Assistant the new server information must be entered in Oracle Exadata Deployment Assistant and configuration files generated Refer to Preparing the Network Configuration on page 1 7 for additional information Download the latest release of Oracle Exadata Deployment Assistant listed in My Oracle Support note 888828 1 Enter the new server information in Oracle Exadata Deployment Assistant Do not include information for the existing rack Notes When extending an existing rack that does not have Sun Server X4 2 Oracle Database Servers with Sun Server X4 2 Oracle Database Servers be sure to deselect the active bonding option for the InfiniBand network so the new database servers are configured with active passive bonded interfaces a When extending an existing Oracle Exadata Database Machine X4 2 with active active bonding select the active bonding option to configure the new database servers for active active bonding Ge
141. racle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively The cable lengths shown in the tables assume the racks are adjacent to each other If the racks are not adjacent or use overhead cabling trays then they may require longer cables lengths Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 10 Leaf Switch Connections for the First Rack in a Five Rack System Leaf Switch Connection Cable Length R1 IB3 within Rack 1 R1 IB3 P8A to R1 U1 P3A 3 meters R1 IB3 P8B to R1 U1 P4A R1 IB3 to Rack 2 R1 IB3 P9A to R2 U1 P5A 5 meters R1 IB3 P9B to R2 U1 P6A R1 IB3 to Rack 3 R1 IB3 P10A to R3 U1 P7A 5 meters R1 IB3 P10B to R3 U1 P8A R1 IB3 to Rack 4 R1 IB3 P11A to R4 U1 P9A 10 meters R1 IB3 to Rack 5 R1 IB3 P11B to R5 U1 P10A 10 meters R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 P8B to R1 U1 P4B R1 IB2 to Rack 2 R1 IB2 P9A to R2 U1 P5B 5 meters R1 IB2 P9B to R2 U1 P6B R1 IB2 to Rack 3 R1 IB2 P10A to R3 U1 P7B 5 meters R1 IB2 P10B to R3 U1 P8B R1 IB2 to Rack 4 R1 IB2 P11A to R4 U1 P9B 10 meters R1 IB2 to Rack 5 R1 IB2 P11B to R5
142. re on the rest of the servers Itis not necessary to stop a cluster when extending a Oracle Exadata Database Machine Half Rack to Oracle Exadata Database Machine Full Rack or Oracle Exadata Database Machine Quarter Rack to Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack 1 Log in as the root user on the original cluster 2 Start one server of the cluster using the following command GRID_HOME grid bin crsctl start cluster 3 Check the status of the server using the following command GRID_HOME grid bin crsctl stat res t Run the preceding command until it shows the first server has started 4 Start the other servers in the cluster using the following command GRID_HOME grid bin crsctl start cluster all 5 Check the status of the servers using the following command 1 48 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Configuring the New Hardware GRID_HOME grid bin crsctl stat res t It may take several minutes for all servers to start and join the cluster Task 6 Adding Grid Disks to Oracle ASM Disk Groups Grid disks can be added to Oracle ASM disk groups before or after the new servers are added to the grid infrastructure The advantage of adding the grid disks before adding the new servers is that the rebalance operation can start earlier The advantage of adding the grid disks after is that the rebalance operation can be done on the new serv
143. reate griddisk hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s Size datasize create griddisk hostname s Size datasize create griddisk hostname s size recosize create griddisk hostname s S1ze recosize create griddisk hostname s size recosize create griddisk X hostname s size recosize create griddisk hostname s size recosize create griddisk hostname s Size recosize create griddisk DBFS DG CD 06_ hostname s celldisk CD_06_ hostname s size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk DBFS DG _CD_07_ hostname s celldisk CD_07_ hostname s size dbfssize dcli g cell group 1 celladmin cellcli e create griddisk N DBFS_DG_CD_08_ hostname sN celldisk CD_08_N hostname sN size dbfssize dcli g cell group 1 celladmin cellcli e create griddisk DBFS_DG_CD_09_ hostname s celldisk CD_09_ hostname s size dbfssize dcli g cell group 1 celladmin cellcli e create griddisk DBFS DG _CD_10_ hostname sN celldisk CD_10_ hostname sN size dbfssize dcli g cell_group 1 celladmin cellcli e create griddisk DBFS DG _CD_ 11 _ hostname sN celldisk CD_11 hostname s size dbfssize Extending Oracle Exadata Database Machine 1 15 Extending the Hardware Adding Grid Disks to Oracle ASM Disk Groups in Oracle Exadata Database M
144. rmat a20 SQL gt SELECT att name attribute upper att value value dg name diskgroup FROM VSASM_ATTRIBUTE att VSASM DISKGROUP DG WHERE DG group_number att group_number AND Extending Oracle Exadata Database Machine 1 23 Extending the Hardware att name LIKE tappliance mode ORDER BY DG group_number Validate the number of Oracle ASM disks using the following command SQL gt SELECT g name d failgroup d mode_status count FROM vSasm_diskgroup g vSasm_disk d WHERE d group_number g group_number GROUP BY g name d failgroup d mode_status Extending a Quarter Rack or Half Rack Extending Oracle Exadata Database Machine from quarter rack to half rack or half rack to full rack consists of adding new hardware to the rack The following sections describe how to extend Oracle Exadata Database Machine with new servers Removing the Doors Adding a Sun Datacenter InfiniBand Switch 36 Switch Optional Adding New Servers Cabling Database Servers Cabling Exadata Storage Servers Closing the Rack Note It is possible to extend the hardware while the machine is online and with no downtime However extreme care should be taken In addition patch application to existing switches and servers should be done before extending the hardware Removing the Doors The following procedure describes how to remove the doors on Oracle Exadata Database Machine 1 Remove the Oracle Exadata Database Machine fron
145. rs R4 IB3 to Rack 7 R4 IB3 P10A to R7 U1 P7A 10 meters R4 IB2 within Rack 4 R4 IB2 P8A to R4 U1 P3B 3 meters R4 IB2 P8B to R4 U1 P4B R4 IB2 to Rack 1 R4 IB2 P10B to R1 U1 P8B 10 meters R4 IB2 to Rack 2 R4 IB2 P11A to R2 U1 P9B 5 meters R4 IB2 to Rack 3 R4 IB2 P11B to R3 U1 P10B 5 meters R4 IB2 to Rack 5 R4 IB2 P9A to R5 U1 P5B 5 meters R4 IB2 to Rack 6 R4 IB2 P9B to R6 U1 P6B 5 meters R4 IB2 to Rack 7 R4 IB2 P10A to R7 U1 P7B 10 meters Table 2 25 shows the cable connections for the fifth spine switch R5 U1 when cabling seven full racks together Table 2 25 Leaf Switch Connections for the Fifth Rack in a Seven Rack System Leaf Switch R5 IB3 within Rack 5 Connection R5 IB3 P8A to R5 U1 P3A R5 IB3 P8B to R5 U1 P4A Cable Length 3 meters 2 26 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Seven Rack Cabling Table 2 25 Cont Leaf Switch Connections for the Fifth Rack in a Seven Rack System Leaf Switch Connection Cable Length R5 IB3 to Rack 1 R5 IB3 P10A to R1 U1 P7A 10 meters R5 IB3 to Rack 2 R5 IB3 P10B to R2 U1 P8A 10 meters R5 IB3 to Rack 3 R5 IB3 P11A to R3 U1 P9A 5 meters R5 IB3 to Rack 4 R5 IB3 P11B to R4 U1 P10A 5 meters R5 IB3 to Rack 6 R5 IB3 P9A to R6 U1 P5A 5 meters R5 IB3 to Rack 7 R5 IB3 P9B to R7 U1 P6A 5 meters R5 IB2 within Rack 5 R5 IB2 P8A to R5 U1 P3B 3 meters R5 IB2 P8B to R5 U1 P4B R5 IB2 to Rack 1 R5 I
146. rs R6 IB2 P9B to R1 U1 P6B R6 IB2 to Rack 2 R6 IB3 P10A to R2 U1 P7B 10 meters R6 IB2 to Rack 3 R6 IB2 P10B to R3 U1 P8B 10 meters R6 IB2 to Rack 4 R6 IB2 P11A to R4 U1 P9B 5 meters Multi Rack Cabling Tables 2 23 Seven Rack Cabling Table 2 20 Cont Leaf Switch Connections for the Sixth Rack in a Six Rack System R6 IB2 to Rack 5 R6 IB2 P11B to R5 U1 P10B Seven Rack Cabling Table 2 21 shows the cable connections for the first spine switch R1 U1 when cabling seven racks together Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as I
147. rt the database servers in a rolling manner using the following command reboot Note Ensure the output for restore_status and config_sync_ status are shown as Ok before activating the storage server cores and disks Getting the status from the BIOS after restarting may take several minutes Extending Oracle Exadata Database Machine 1 19 Extending the Hardware Activating Storage Server Cores and Disks in Oracle Exadata Database Machine X3 2 Eighth Rack The following procedure describes how to activate the storage server cores and disks 1 Log in as the root user on the first database server 2 Activate the cores on the storage server group using the following command The command uses the dcli utility and runs the command as the celladmin user deli g cell group 1 celladmin cellcli e alter cell eighthRack false 3 Create the cell disks using the following command deli g cell group 1 celladmin cellcli e create celldisk all 4 Recreate the flash log using the following commands deli g cell group 1 celladmin cellcli e drop flashlog all force deli g cell group 1 celladmin cellcli e create flashlog all 5 Expand the flash cache using the following command deli g cell group 1 celladmin cellcli e alter flashcache all Creating Grid Disks in Oracle Exadata Database Machine X3 2 Eighth Rack Grid disk creation must follow a specific order to ensure the proper offset The order o
148. s celldisk FD_06_ celladmin cellcli e sN celldisk FD_07_ celladmin cellcli e s celldisk FD_04_ celladmin cellcli e s celldisk FD_05_ celladmin cellcli e s celldisk FD_06_ celladmin cellcli e s celldisk FD_07_ create griddisk hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s size datasize create griddisk hostname s size recosize create griddisk N hostname s size recosize create griddisk hostname s size recosize create griddisk hostname s size recosize dcli g cell_group 1 celladmin cellcli e create griddisk DBFS DG FD 04 hostname s celldisk FD_04_ hostname s dcli g cell group 1 celladmin cellcli e create griddisk DBFS_DG_FD_05_ hostname sN celldisk FD_05_ hostname s dcli g cell_group 1 celladmin cellcli e create griddisk DBFS_DG_FD_06_ hostname sN celldisk FD_06_ hostname s dcli g cell group 1 celladmin cellcli e create griddisk DBFS_DG_FD_07_ hostname sN celldisk FD_07_ hostname s Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Size dbfssize s1ze dbfssize S1ze dbfssize Size dbfssize Extending the Hardware Table 1 1 Cont Commands to Create Disk Groups When Extending Oracle Exadata Database Machine X4 2 E
149. s the cable connections for the switches when cabling seventeen racks together The cabling between racks 1 through 8 do not change Notes x The spine switch is in U1 also referred to as IB1 for all racks Inthe following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 44 Leaf Switch Connections for the Seventeenth Rack in the System Leaf Switch Connection R17 IB3 to Rack 1 R17 IB3 P8B to R1 U
150. se owner user cd SORACLE HOME oui bin S addNode sh silent responseFile path to add db nodes rsp Ensure the SORACLE_HOME oui oraparam ini file has the memory settings that match the parameters set in the Grid Infrastructure home Run the root sh script on each server when prompted as the root user using the deli utility S dcli g new db _nodes 1 root N u01 app oracle product 11 2 0 dbhome_1 root sh In the preceding command new_db_nodes is the file with the list of new database servers Verify the ORACLE HOME directories have been added to the new servers using the following command deli g root all_group 1 root du sm N u01 app oracle product 11 2 0 dbhome_1 Task 10 Adding Database Instance to the New Servers Before adding the database instances to the new servers check the following Maximum file size If any data files have reached their maximum file size then the addInstance command may fail with an ORA 740 error Oracle recommends you check that none of the files listed in DBA_DATA_FILES have reached their maximum size Files that have reached their maximum should be corrected Online redo logs If the online redo logs are kept in the directory specified by the DB_RECOVERY_FILE_DEST parameter then ensure the space allocated is sufficient for the additional redo logs for the new instances being added If necessary then increase the size for the DB_RECOVERY_FILE_DEST_SIZE parameter Total number of inst
151. son to watch the cables and CMA Cabling Database Servers After the new database servers are installed they need to be cabled to the existing equipment The following procedure describes how to cable the new equipment in the rack The images shown in the procedure are of a Sun Fire X4170 M2 Oracle Database Server Notes The existing cable connections in the rack do not change The blue cables connect to Oracle Database servers and the black cables connect to Exadata Storage Servers These network cables are for the NETO Ethernet interface port Attach and route the management cables on the CMA and rear panel one server at a time Do not slide out more than one server at a time Start from the bottom of the rack and work upward Route the cables through the CMA with the dongle on the top and power cables on the bottom Longer hook and loop straps are needed when cabling three CAT5e cables or two TwinAx cables See Also Oracle Exadata Database Machine System Overview for cabling tables 1 Connect the CAT5e cables AC power cables and USB to their respective ports on the rear of the server Ensure the flat side of the dongle is flush against the CMA inner rail Extending Oracle Exadata Database Machine 1 31 Extending the Hardware 2 Adjust the green cable management arm CMA brackets Description of the CMA callouts in the preceding image 1 Sr N ee pe e N Connector A Front slide ba
152. t and rear doors as follows a Unlock the front and rear doors The key is in the shipping kit b Open the doors c Detach the grounding straps connected to the doors by pressing down on the tabs of the grounding strap s quick release connectors and pull the straps from the frame d Lift the doors up and off their hinges 1 24 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Extending the Hardware Figure 1 2 Removing the Rack Doors Description of callouts in Figure 1 2 1 Detaching the grounding cable 2 Top rear hinge 3 Bottom rear hinge 4 Top front hinge 5 Bottom front hinge 2 Remove the filler panels where the servers will be installed using a No 2 screwdriver to remove the M6 screws The number of screws depends on the type of filler panel Save the screws for future use Extending Oracle Exadata Database Machine 1 25 Extending the Hardware Figure 1 3 Removing the Filler Panels Note If you are replacing the filler panels then do not remove the Duosert cage nuts from the RETMA Radio Electronics Television Manufacturers Association rail holes Adding a Sun Datacenter InfiniBand Switch 36 Switch Optional This procedure is necessary as follows Upgrading a rack with Sun Fire X4170 Oracle Database Servers to Oracle Exadata Database Machine Half Rack or Oracle Exadata Database Machine Full Rack a Extending an Oracle Exadata Database Machine
153. tarting from the bottom of the rack 2 Remove and discard the trough which attaches the cable harness when no server is installed in the unit 3 Remove and discard the solid filler Task 2 Installing the Rack Assembly The following procedure describes how to install the rack assembly 1 Position a mounting bracket against the chassis so that the slide rail lock is at the server front and the five keyhole openings on the mounting bracket are aligned with the five locating pins on the side of the chassis 2 Orient the slide rail assembly so that the ball bearing track is forward and locked in place 3 Starting on either side of the rack align the rear of the slide rail assembly against the inside of the rear rack rail and push until the assembly locks into place with an audible click 4 Align the front of the slide rail assembly against the outside of the front rack rail and push until the assembly locks into place and you hear the click 5 Repeat steps 2 to 4 on the other side on the rack Extending Oracle Exadata Database Machine 1 29 Extending the Hardware Task 3 Installing the Server WARNING Installing a server requires a minimum of two people or a lift because of the weight of each server Attempting this procedure alone can result in equipment damage personal injury or both Always load equipment into the rack from the bottom up so that the rack does not become top heavy and tip over Ext
154. tch 36 switches are needed when cabling racks together If your rack does not include a third switch then order the expansion switch kit These switches attach to standard Quad Small Form factor Pluggable QSFP connectors at the end of the InfiniBand cables Up to eighteen racks can be cabled together without external InfiniBand switches The procedures in this section assume the racks are adjacent to each other If they are not then longer cables may be required for the connections The switch at rack unit 1 U1 is referred to as the spine switch Multi Rack Cabling Tables 2 1 Understanding Multi Rack Cabling 2 2 The following switches are referred to as leaf switches a The switches at rack unit 20 U20 and rack unit 24 U24 in Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks a The switches at rack unit 21 U21 and rack unit 23 U23 in Oracle Exadata Database Machine X5 8 Full Racks Oracle Exadata Database Machine X4 8 Full Racks Oracle Exadata Database Machine X3 8 Full Racks and Oracle Exadata Database Machine X2 8 Full Racks a The switches at rack unit 20 U20 and 22 U22 in Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 racks and Oracle Exadata Storage Expansion Rack X3 2s Note Oracle Exadata Database Machine X4 2 racks and Oracle Exadata Database Machine X3 8 Full Racks with Exadata Storage Server X4 2L Servers do not include spare cables
155. ters R3 IB3 to Rack 2 R3 IB3 P11B to R2 U1 P10A 5 meters R3 IB3 to Rack 4 R3 IB3 P9A to R4 U1 P5A 5 meters R3 IB3 to Rack 5 R3 IB3 P9B to R5 U1 P6A 5 meters Multi Rack Cabling Tables 2 25 Seven Rack Cabling Table 2 23 Cont Leaf Switch Connections for the Third Rack in a Seven Rack System Leaf Switch Connection Cable Length R3 IB3 to Rack 6 R3 IB3 P10A to R6 U1 P7A 10 meters R3 IB3 to Rack 7 R3 IB3 P10B to R7 U1 P8A 10 meters R3 IB2 within Rack 3 R3 IB2 P8A to R3 U1 P3B 3 meters R3 IB2 P8B to R3 U1 P4B R3 IB2 to Rack 1 R3 IB2 P11A to R1 U1 P9B 5 meters R3 IB2 to Rack 2 R3 IB2 P11B to R2 U1 P10B 5 meters R3 IB2 to Rack 4 R3 IB2 P9A to R4 U1 P5B 5 meters R3 IB2 to Rack 5 R3 IB2 P9B to R5 U1 P6B 5 meters R3 IB2 to Rack 6 R3 IB2 P10A to R6 U1 P7B 10 meters R3 IB2 to Rack 7 R3 IB2 P10B to R7 U1 P8B 10 meters Table 2 24 shows the cable connections for the fourth spine switch R4 U1 when cabling seven full racks together Table 2 24 Leaf Switch Connections for the Fourth Rack in a Seven Rack System Leaf Switch Connection Cable Length R4 IB3 within Rack 4 R4 IB3 P8A to R4 U1 P3A 3 meters R4 IB3 P8B to R4 U1 P4A R4 IB3 to Rack 1 R4 IB3 P10B to R1 U1 P8A 10 meters R4 IB3 to Rack 2 R4 IB3 P11A to R2 U1 P9A 5 meters R4 IB3 to Rack 3 R4 IB3 P11B to R3 U1 P10A 5 meters R4 IB3 to Rack 5 R4 IB3 P9A to R5 U1 P5A 5 meters R4 IB3 to Rack 6 R4 IB3 P9B to R6 U1 P6A 5 mete
156. tes for the switches to become completely operational To check the switch log in to the switch and run the ibswitches command The output should show n 1 3 switches for IB1 IB2 and IB3 in racks R1 R2 Rn 1 Ensure the Subnet Manager Master is running on R1 IB1 by running the getmaster command from any switch Power on all servers in Rn 1 Log in to spine switch R1 IB1 and lower its priority to 8 as follows a Use the disablesm command to stop Subnet Manager b Use the setsmpriority 8 command to set the priority to 8 c Use the enablesm command to restart Subnet Manager Ensure Subnet Manager Master is running on one of the spine switches using the getmaster command from any switch Ensure Subnet Manager is running on every spine switch using the following command from any switch ibdiagnet r Each spine switch should show as running in the Summary Fabric SM state priority section of the output If a spins switch is not running then log in to the switch and enable Subnet Manager using the enablesm command If there are now four or more racks then log in to the leaf switches in each rack and disable Subnet Manager using the disablesm command Configuring the New Hardware This section contains the following tasks needed to configure the new hardware Task 1 Changing the Interface Names Task 2 Setting Up New Servers Task 3 Setting Up a New Rack Task 4 Setting User Equivalence Task 5 Starting the
157. the root user on one of the existing database servers to configure user equivalence for all servers using the following command Oracle recommends using the first database server opt oracle SupportTools onecommand setssh Linux sh s c N h N path_to_file all_group n N In the preceding command path_to_file is the directory path for the all_group file containing the names for the existing and new servers Note For Oracle Exadata Database Machine X2 2 with X4170 and X4275 servers systems use the setssh sh command to configure user equivalence The command line options for the setssh sh command differ from the setssh Linux sh command Run setssh sh without parameters to see the proper syntax Add the known hosts using InfiniBand using the following command This requires all database servers are accessible by way of their InfiniBand interfaces opt oracle SupportTools onecommand setssh Linux sh s c N h path_to_file all_ib_ group n N p password Verify equivalence using the following commands deli g all _group 1 root date deli g all_ib group 1 root date Run the setssh Linux sh script as the oracle user on one of the existing database servers to configure user equivalence for all servers using the following command Oracle recommends using the first database server If there are separate owners for the Grid Infrastructure then run a similar command for each owner S opt oracle SupportTools oneco
158. the services on the new nodes 1 Ensure the etc oracle cell network config ora files are correct and consistent on all database servers The cellip ora file all database server should include the older and newer database servers and storage servers 2 Ensure the ORACLE BASE and diag destination directories have been created on the Grid Infrastructure destination home The following is an example deli g root new_group_files dbs_group 1 root mkdir p u01 app 11 2 0 grid u01 app oraInventory u01 app grid diag deli g root new_group_files dbs_group 1 root chown R grid oinstall N u01 app 11 2 0 u01 app oraInventory u01 app grid dcli g root new_group_files dbs_group 1 root chmod R 770 u01 app oraInventory deli g root new _group_files dbs_group 1 root chmod R 755 FuOlfenp rl 2 0 u0l appi112 0 grid 3 Ensure the inventory directory and Grid Infrastructure home directories have been created and have the proper permissions The directories should be owned by the Grid Infrastructure owner and the OINSTALL group The inventory directory should have 770 permission and the Grid Infrastructure home directories should have 755 4 Create users and groups on the new nodes with the same user identifiers and group identifiers as on the existing nodes Note If Oracle Exadata Deployment Assistant was used earlier then these users and groups should have been created Check that they do exist and
159. ti Rack Cabling Tables 2 27 Eight Rack Cabling Table 2 27 Leaf Switch Connections for the Seventh Rack in a Seven Rack System Leaf Switch Connection Cable Length R7 IB3 within Rack 7 R7 IB3 P8A to R7 U1 P3A 3 meters R7 IB3 P8B to R7 U1 P4A R7 IB3 to Rack 1 R7 IB3 P9A to R1 U1 P5A 10 meters R7 IB3 to Rack 2 R7 IB3 P9B to R2 U1 P6A 10 meters R7 IB3 to Rack 3 R7 IB3 P10A to R3 U1 P7A 10 meters R7 IB3 to Rack 4 R7 IB3 P10B to R4 U1 P8A 10 meters R7 IB3 to Rack 5 R7 IB3 P11A to R5 U1 P9A 5 meters R7 IB3 to Rack 6 R7 IB3 P11B to R6 U1 P10A 5 meters R7 IB2 within Rack 7 R7 IB2 P8A to R7 U1 P3B 3 meters R7 IB2 P8B to R7 U1 P4B R7 IB2 to Rack 1 R7 IB2 P9A to R1 U1 P5B 10 meters R7 IB2 to Rack 2 R7 IB2 P9B to R2 U1 P6B 10 meters R7 IB2 to Rack 3 R7 IB2 P10A to R3 U1 P7B 10 meters R7 IB2 to Rack 4 R7 IB2 P10B to R4 U1 P8B 10 meters R7 IB2 to Rack 5 R7 IB2 P11A to R5 U1 P9B 5 meters R7 IB2 to Rack 6 R7 IB2 P11B to R6 U1 P10B 5 meters Eight Rack Cabling Table 2 28 shows the cable connections for the first spine switch R1 U1 when cabling eight racks together 2 28 Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Eight Rack Cabling Notes The spine switch is in U1 also referred to as IB1 for all racks In the following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracl
160. tion Cable Length R1 IB3 P8B to R1 U1 P4A R1 IB3 P9A to R1 U1 P5A R1 IB3 to Rack 2 R1 IB3 P9B to R2 U1 P6A 5 meters R1 IB3 to Rack 3 R1 IB3 P11A to R3 U1 P9A 5 meters R1 IB3 P10A to R2 U1 P7A R1 IB3 P10B to R2 U1 P8A R1 IB3 P11B to R3 U1 P10A R1 IB2 within Rack 1 R1 IB2 P8A to R1 U1 P3B 3 meters R1 IB2 P8B to R1 U1 P4B R1 IB2 P9A to R1 U1 P5B R1 IB2 to Rack 2 R1 IB2 P9B to R2 U1 P6B 5 meters R1 IB2 P10A to R2 U1 P7B R1 IB2 P10B to R2 U1 P8B Oracle Exadata Database Machine Extending and Multi Rack Cabling Guide Three Rack Cabling Table 2 3 Cont Leaf Switch Connections for the First Rack in a Three Rack System Leaf Switch R1 IB2 to Rack 3 Connection R1 IB2 P11A to R3 U1 P9B R1 IB2 P11B to R3 U1 P10B Cable Length 5 meters Table 2 4 shows the cable connections for the second spine switch R2 U1 when cabling three racks together Table 2 4 Leaf Switch Connections for the Second Rack in a Three Rack System Leaf Switch R2 IB3 within Rack 2 Connection R2 IB3 P8A to R2 U1 P3A R2 IB3 P8B to R2 U1 P4A R2 IB3 P9A to R2 U1 P5A Cable Length 3 meters R2 IB3 to Rack 1 R2 IB3 P11A to R1 U1 P9A R2 IB3 P11B to R1 U1 P10A 5 meters R2 IB3 to Rack 3 R2 IB3 P9B to R3 U1 P6A R2 IB3 P10A to R3 U1 P7A R2 IB3 P10B to R3 U1 P8A 5 meters R2 IB2 within Rack 2 R2 IB2 P8A to R2 U1 P3B R2 IB2 P8B to R2 U1 P4B R2 IB2 P9A to R2 U1 P5B 3 meters
161. to R8 U1 P13B Table 2 39 shows the cable connections for the switches when cabling twelve racks together The cabling between racks 1 through 8 do not change Notes a The spine switch is in U1 also referred to as IB1 for all racks Inthe following tables the leaf switches are referred to as IB2 and IB3 Use the rack unit that is appropriate for your system In Oracle Exadata Database Machine X2 2 racks and Oracle Exadata Storage Expansion Racks with Exadata Storage Server with Sun Fire X4270 M2 Servers the leaf switches are in U20 and U24 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X5 8 Full Rack Oracle Exadata Database Machine X4 8 Full Rack Oracle Exadata Database Machine X3 8 Full Rack and Oracle Exadata Database Machine X2 8 Full Rack racks the leaf switches are in U21 and U23 also referred to as IB2 and IB3 respectively a In Oracle Exadata Database Machine X4 2 racks Oracle Exadata Database Machine X3 2 Full Racks and Oracle Exadata Storage Expansion Rack X3 2 racks the leaf switches are in U20 and U22 also referred to as IB2 and IB3 respectively Cable lengths for racks 9 through 18 vary depending on the layout of the racks Up to 100 meters is supported Only optical cables are supported for lengths greater than 5 meters Table 2 39 Leaf Switch Connections for the Twelfth Rack in the System Leaf Switch Connection R12 IB3 to Rack
162. uams Leaf Switch 2 DE mer Leaf Switch 2 The following graphic shows the cable connections from Oracle Exadata Quarter Rack to Oracle Exadata Half Rack or Oracle Exadata Full Rack The leaf switches within each rack maintain their existing seven connections The leaf switches interconnect between the racks with two links each using the ports reserved for external connectivity Quarter Rack Half or Full Rack Leaf Switch 1 Leaf Switch 1 Ml LS ee m J RD Lo BS Be hee Bee a Spine Switch 1 7 Sates Leaf Switch 2 Multi Rack Cabling Tables 2 9 Two Rack Cabling The following graphic shows the cable connections from Oracle Exadata Quarter Rack to two or more racks The racks that connect to Oracle Exadata Quarter Rack must be all Oracle Exadata Half Racks or Oracle Exadata Full Racks interconnected using a fat tree topology Each leaf switch in Oracle Exadata Quarter Rack connects to the spine switches in the other half racks or full racks with two links each If there are more than four racks then use one link instead of two Quarter Rack Full Half Rack 1 Full Half Rack n Spine Switch Spine Switch Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 Leaf Switch 1 Leaf Switch 2 Two Rack Cabling Table 2 1 shows the cable connections for the first spine switch R1 U1 when cabling two racks together Notes The spine switch is in U1 also referred
163. xadata Database Machine X5 2 m Cabling Tables for Oracle Exadata Database Machine X4 2 Cabling Tables for Oracle Exadata Database Machine X4 8 Full Rack m Cabling Tables for Oracle Exadata Database Machine X3 2 and Oracle Exadata Database Machine X3 8 Full Rack m Cabling Tables for Oracle Exadata Database Machine X2 2 and Oracle Exadata Database Machine X2 8 Full Rack m Cabling Tables for Oracle Exadata Storage Expansion Rack Attach a CMA to the server Extending Oracle Exadata Database Machine 1 35 Extending the Hardware 2 Insert the cables into their ports through the hook and loop straps then route the cables into the CMA in this order a Power b Ethernet c InfiniBand 3 Route the cables through the CMA and secure them with hook and loop straps on both sides of each bend in the CMA 4 Close the crossbar covers to secure the cables in the straightaway 5 Verify operation of the slide rails and the CMA for each server Note Oracle recommends that two people do this step one person to move the server in and out of the rack and another person to watch the cables and the CMA a Slowly pull the server out of the rack until the slide rails reach their stops b Inspect the attached cables for any binding or kinks c Verify that the CMA extends fully from the slide rails 6 Push the server back into the rack a Release the two sets of slide rail stops 1 36 Oracle Exadata Database M
164. xtending the Hardware Adding Grid Disks to Oracle ASM Disk Groups in Oracle Exadata Database Machine X4 2 Eighth Rack or Later Validating New Oracle Exadata Database Machine X3 2 Half Rack or Later Configuration Note In the following procedures the disk group names and sizes are examples The values should be changed in the commands to match the actual system The procedures assume user equivalence exists between the root user on the first database server and all other database servers and to the celladmin user on all storage cells The text files cell_group and db_group should be created to contain lists of cell host names and database server host names respectively Reviewing and Validating Current Configuration of Oracle Exadata Database Machine X4 2 Eighth Rack or Later The following procedure describes how to review and validate the current configuration 1 2 3 Log in as the root user on the first database server Review the current configuration of the database servers using the following command deli g db_group 1 root dbmcli e list dbserver attributes coreCount The following is an example of the output from the command for Oracle Exadata Database Machine X5 2 Eighth Rack dm01db01 18 dm01db02 18 Note The number of active cores in Oracle Exadata Database Machine X5 2 Eighth Rack database server is 18 The number of active cores in Oracle Exadata Database Machine X4 2 Eighth R

Oracle Exadata Database Machine Extending and Multi

Contents

Download Pdf Manuals

Related Search

Related Contents