Home

Intel SGI Altix 450 User's Manual

1. Figure A 3 Pin Number Locations for External SAS SATA Port Table A 9 SAS SATA Pin Assignments Pin Number Signal Name Pin Number Signal Name 1 Gnd 8 Gnd 2 2tx_h 9 3tx_h 3 2tx_1 10 3tx_l 4 Gnd 11 Gnd 5 2rx_1 12 3rx_1 6 2rx_h 13 3rx_h 7 Gnd 14 Gnd 162 007 4857 002 T O Port Specifications USB Type A Connector Figure A 4 shows the USB type A connector provided on the base I O that supports general USB applications and optional keyboard and mouse configurations Table A 10 lists the pin assignments for the USB type A connector TTT 1 2 3 4 Figure A 4 Pin Number Locations for USB Type A Connector Table A 10 Pin Assignments for USB Type A Connector Signal Color Pin Number VCC Red 1 Data White 2 Data Green 3 Ground Black 4 007 4857 002 163 Appendix B Safety Information and Regulatory Specifications This appendix provides safety information and regulatory specifications for your system in the following sections Safety Information 007 4857 002 Safety Information on page 165 Regulatory Specifications on page 167 Read and follow these instructions carefully 1 Follow all warnings and instructions marked on the product and noted in the documentation included with this product Unplug this product before cleaning Do not use liquid cleaners or aerosol cleaners Use a damp cloth for cleaning Do not use this product near water Do not place this prod
2. If your system does not boot correctly see Troubleshooting Chart in Chapter 7 for troubleshooting procedures 108 007 4857 002 Installing or Removing Internal Parts Installing or Removing Internal Parts 007 4857 002 A Caution The components inside the system are extremely sensitive to static electricity Always wear a wrist strap when you work with parts inside your system To use the wrist strap follow these steps 1 Unroll the first two folds of the band 2 Wrap the exposed adhesive side firmly around your wrist unroll the rest of the band and then peel the liner from the copper foil at the opposite end 3 Attach the copper foil to an exposed electrical ground such as a metal part of the chassis Caution Do not attempt to install or remove components that are not listed in Table 6 1 Components not listed must be installed or removed by a qualified SGI field engineer Table 6 1 lists the customer replaceable components and the page on which you can find the instructions for installing or removing the component Table 6 1 Customer replaceable Components and Maintenance Procedures Component Procedure L1 controller panel Remove and Replace the System Control Display Panel on page 110 IRU power supplies Removing or Adding IRU Power Supplies on page 113 IRU fans blowers Removing and Replacing IRU Fans Blowers on page 114 Replac
3. 007 4857 002 103 5 Rack Information Figure 5 2 Short 20U Altix Rack Rear View Technical Specifications 104 Table 5 1 lists the technical specifications of the Altix 450 tall rack Table 5 1 Tall Altix Rack Technical Specifications Characteristic Specification Height 79 5 in 201 9 cm Width 25 8 in 65 5 cm Depth 43 45 in 110 4 cm Weight full 1500 Ib 682 kg Table 5 2 lists the technical specifications of the Altix 450 short rack Table 5 2 Short Altix Rack Technical Specifications Characteristic Specification Height 42 8 in 108 7 cm Width 25 8 in 65 5 cm 007 4857 002 Technical Specifications Table 5 2 Short Altix Rack Technical Specifications continued Characteristic Specification Depth 40 9 in 103 9 cm Weight full 750 Ib 340 kg 007 4857 002 105 Chapter 6 Maintenance and Upgrade Procedures This chapter provides information about installing or removing components from your SGI system as follows e Maintenance Precautions and Procedures on page 107 e Installing or Removing Internal Parts on page 109 e Adding or Replacing PCI PCI X Cards on page 128 e Installing or Replacing a Disk Drive in the IA Blade on page 138 Maintenance Precautions and Procedures This section describes how to open the system for maintenance and upgrade protect the components from static damage and return the system to operation
4. This section provides the contents of electromagnetic emissions notices from various countries This equipment complies with Part 15 of the FCC Rules Operation is subject to the following two conditions e This device may not cause harmful interference e This device must accept any interference received including interference that may cause undesired operation Note This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case you will be required to correct the interference at your own expense If this equipment does cause harmful interference to radio or television reception which can be determined by turning the equipment off and on you are encouraged to try to correct the interference by using one or more of the following methods e Reorient or relocate the receiving antenna e Increase the separation between the equipment and receiver e Connect the equipment to an outlet on a circuit different from that to
5. ar 29 25 1068 mm 743 mm Pallet jack position Figure 1 4 Dimensions of Short Rack Shipping Crate Inspecting the Shipping Crate After the system is unloaded from the truck follow these steps before you unpack it 1 Ensure that the crates and cartons arrive unopened 2 Inspect the shipping crate for signs of external damage such as dents holes crushed corners and water marks 3 Ensure that the tilt watch has not been tripped 4 Ifthe crate is damaged file a damage claim with the carrier immediately In addition notify your local Customer Support Center CSC for any missing incorrect or damaged items For CSC contact information see http www sgi com support supportcenters html 14 007 4857 002 Installing a Rack Transporting the Shipping Crate 007 4857 002 A Use a pallet jack with forks that are 48 in 122 cm long or longer to transport the shipping crate to the designated location See Figure 1 3 and Figure 1 4 for the crate dimensions and location to position the pallet jack If the crate does not fit through all access doors you may need to partially disassemble the crate Caution If the system shipping or storage environment is significantly colder than the environment in which it will be installed 40 F 22 C or greater disparity leave the rack in its shipping crate for at least 24 hours at room temperature before you start the installation This acclimation prevents damage
6. xi List of Figures Figure 2 4 Figure 3 1 Figure 3 2 Figure 3 3 Figure 3 4 Figure 3 5 Figure 4 1 Figure 4 2 Figure 4 3 Figure 5 1 Figure 5 2 Figure 6 1 Figure 6 2 Figure 6 3 Figure 6 4 Figure 6 5 Figure 6 6 Figure 6 7 Figure 6 8 Figure 6 9 Figure 6 10 Figure 6 11 Figure 6 12 Figure 6 13 Figure 6 14 Figure 6 15 Figure 6 16 Figure 6 17 Figure 6 18 Figure 6 19 Figure 6 20 Figure 6 21 Figure 6 22 xii 2D Graphics Card Location in IA2 Blade SGI Altix 450 System Short Rack SGI Altix 450 Server System Tall Rack Blade IRU and Rack Components Functional Block Diagram of the Individual Rack Unit IRU Altix 450 IRU System Components Example SGI Altix 450 System Control Network Example L1 Front Panel tel te gh ge r a Be he Ethernet Switch System Controller Block Diagram Example Tall 42U Altix Rack and Front Lock Short 20U Altix Rack Rear View Removing the L1 Controller Panel Replacing the L1 Controller Panel Removing an IRU Power Supply Replacing an IRU Power Supply IRU Fan Assembly Blowers Removing a Fan From the IRU Replacing an IRU Fan Removing a Compute Memory Blade From the IRU Replacing a Compute Memory Blade in the IRU DIMM Group Locations Diagram DIMM Memory Slot Location Example DVD Removal from IA Blade i DVD Drive Replacement in the IA Blade Comparison of PCI PCI X Connector with PCI BEE Connectors Removing a Card Carrier From the Three Slot Blade
7. 2 2 KIX Conventions 4 ogee GL we WA 2 ee ew se ee ae ET Product Suppott io Bo pus soea Become oh CR ee Se ye oe GR OR oe Re ee ge XK Reader Comments a a a a aa XX 1 System Installation System Installation Overview Safety Precautions 1 2 3 Hazard Statements 2 4 ESD Precautions 4 5 Safety Measures 007 4857 002 v Contents Installing a Rack Preinstallation Activities Site Plan Verification Tools Required Power Receptacle Verification Unloading and Moving System Equipment Unloading the Equipment from the Truck Inspecting the Shipping Crate Transporting the Shipping Crate Removing a Short Rack from the Shipping Crate Unpacking and Installing the Tall Rack Positioning and Leveling Racks Unpacking and Inspecting Modules Installing the IRUs in a Rack Determining Space Requirements f Determining Where to Attach Rails in a Rack Rackmounting IRUs with Shelf Rails Removing a Rackmounted IRU Module NUMaAlink Cabling the System IRUs Cabling the System IRUs to a Power Source Connecting an IRU Module to a Power Source No Rack Connecting System Modules to a Rack Power Source Connecting the System Console Operating Your System Powering the System On and Off Preparing to Power On Powering On With Optional System Console and L1 L2 Controller Powering On Manually or With L1 Console Powering Off at the System Console Powering Off Manually L1 Front Panel C
8. CMN Number on page 167 e CE Notice and Manufacturer s Declaration of Conformity on page 167 e Electromagnetic Emissions on page 168 e Shielded Cables on page 170 e Electrostatic Discharge on page 170 e Laser Compliance Statements on page 171 e Lithium Battery Statements on page 172 This SGI system conforms to several national and international specifications and European Directives listed on the Manufacturer s Declaration of Conformity The CE mark insignia displayed on each device is an indication of conformity to the European requirements Caution This product has several governmental and third party approvals licenses and permits fy Do not modify this product in any way that is not expressly approved by SGI If you do you may lose these approvals and your governmental agency authority to operate this device CMN Number The model number or CMN number for the system is on the system label which may be on the IRU or mounted inside the rear door on the base of the rack CE Notice and Manufacturer s Declaration of Conformity The CE symbol indicates compliance of the device to directives of the European Community A Declaration of Conformity in accordance with the standards has been made and is available from SGI upon request 007 4857 002 167 B Safety Information and Regulatory Specifications Electromagnetic Emissions FCC Notice USA Only 168
9. Figure 6 8 119 007 4857 002 6 Maintenance and Upgrade Procedures Replacing a Compute Memory Blade in the IRU Figure 6 9 007 4857 002 120 Replacing IRU Components Memory DIMM Group Placement and Guidelines 007 4857 002 Memory is contained on cards that are referred to as DIMMs dual inline memory modules Each individual memory or compute memory blade can contain four eight or twelve DIMMs installed in DIMM slots located on the blade s node board These twelve DIMM slots are organized into three groups of four DIMMs each as shown in Figure 6 10 DIMMs are installed one per DIMM slot and must be installed in groups of four Figure 6 11 shows an example of the physical DIMM locations within the processor blade assembly Follow these guidelines when installing DIMM groups Memory is increased or decreased in four DIMM group increments only Each of the four DIMMs that make up a DIMM group must be the same memory size and speed however different blades can differ in total memory size The first four DIMMs group 0 must be in place for the blade s memory to operate properly Mixing compute memory blades with different capacity DIMM groups within an IRU is permitted When installing 4 GB DIMMs check with your SGI service representative Space limitations may require that a maximum of eight 4 GB DIMMs be installed on each blade 121 6 Maintenance and Upgrade Procedures
10. Secondary is PhysDisk 1 Bus 0 Target 1 RAID actions menu select an option 1 99 or e p w or 0 to quit 99 Resetting port RAID actions menu select an option 1 99 or e p w or 0 to quit 0 Main menu select an option 1 99 or e p w or 0 to quit 0 Port Name Chip Vendor Type Rev MPT Rev Firmware Rev IOC 1 01 00 01 0 LSI Logic SAS1068 BO 105 01100000 0 Select a device 1 1 or 0 to quit 0 Sas Pun0 Lun0 LSILOGICLogical Volume 3000 007 4857 002 Installing Optional Components fs0 gt reset EFI Boot Manager ver 1 10 Partition 0 CBlades 2 RBlades 0 TOBlades d Please select a boot opti EFI Shell netboot gt gt gt Boot option maintenance menu Loading netboot Running LoadFile CLIENT MAC ADDR 08 00 69 CLIENT IP 137 38 82 91 GATEWAY IP 137 38 82 252 TSize Running LoadFile Starting netboot ELILO Quietly populating dev s 14 62 Enabled Disabled Nodes 3 0 CPUs 4 0 Mem GB 6 0 on 14 E3 0C MASK 255 255 255 0 DHCP IP 137 38 228 4 137 38 82 254 da2 and dev sdal with ia64 propack51 sles10 Jan19 reboot Note The initial resynchronization of the mirrored disk depending on the size of the disk may take up to 45 minutes or longer RAID actions menu 007 4857 002 select an option 1 99 or e p w or 0 to quit 1 65 2 Operation Procedures 1 volume Volume 0 Volume Volume Volume Volume P
11. Specification Dimensions for a single short 20U rack system including doors and side panels Shipping dimensions Weight of full rack maximum Shipping weight maximum Access requirements Front Rear Side Individual Rack Unit IRU enclosure specifications Height 41 8 in 106 2 cm Width 25 8 in 65 5 cm Depth 40 9 in 103 9 cm Height 47 5 in 120 7 cm Width 31 25 in 79 4 cm Depth 49 5 in 125 7 cm 750 Ib 340 kg 862 Ib 391 kg 48 in 121 9 cm 48 in 121 9 cm None Dimensions 8 68 in high x 17 5 in wide x 17 in deep 22 cm high x 44 45 cm wide x 43 2 cm deep Weight 130 Ibs 59 1 kg 155 A Technical Specifications and Pinouts Environmental Specifications 156 Table A 4 lists the environmental specifications of the 42U rack system Table A 5 provides the environmental information for the 20U system Table A 4 Environmental Specifications 42U Rack Feature Specification Temperature tolerance operating Temperature tolerance non operating Relative humidity Heat dissipation Altix 450 42U rack Cooling requirement Air flow intake front exhaust rear Maximum altitude Acoustical noise level 5 C 41 F to 35 C 95 F up to 1500 m 5000 ft 5 C 41 F to 30 C 86 F 1500 m to 3000 m 5000 ft to 10 000 ft 40 C 40 F to 60 C 140 F 10 to 90 operating no condensation 8 to 95
12. e Initiation of system resets e Read write storage for identification and configuration information e Provides console diagnostic and scan interface The L1 controller in each of the enclosures is a complete and fully functional system controller All the blades are interconnected by NUMAIink and each shares its system control information with all other system controllers Console Mode from L1 Output from the system is visible and all input is directed to the system console Note The console mode from L1 mode is supported only if the system console L1 port is connected directly to the console system laptop PC etc If you see a prompt of the following form the L1 is ready to accept commands 001c01 L1 gt Common operations are discussed in the following sections e Viewing System Configuration from an IRU s Perspective on page 86 e Command Targeting on page 86 85 4 System Control e Viewing Information Warnings and Error Messages on page 94 Viewing System Configuration from an IRU s Perspective Command Targeting 86 An L1 has limited knowledge of the system topology depending on the system s configuration Typically an L1 has information only about L1s that are directly NUMAlink connected In large configurations with more than one L1 the L1 may have knowledge of only a portion of the L1s in the system These configurations require the use of the L2 see L2 Ope
13. subject to the provisions of its applicable license agreement as specified in a 48 CFR 12 212 of the FAR or if acquired for Department of Defense units b 48 CFR 227 7202 of the DoD FAR Supplement or sections succeeding thereto Contractor manufacturer is SGI 1140 East Arques Avenue Sunnyvale CA 94085 TRADEMARKS AND ATTRIBUTIONS SGI the SGI logo and Altix are registered trademarks and NUMAlink and NUMAflex are trademarks of SGI in the United States and or other countries worldwide Intel and Itanium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries UNIX is a registered trademark in the United States and other countries licensed exclusively through X Open Company Ltd Linux is a registered trademark of Linus Torvalds All other trademarks mentioned herein are the property of their respective owners 007 4857 002 Record of Revision Version Description 001 August 2006 First release 002 March 2007 Engineering updates to cover RAID 1 DVD R W 2D graphics and IA2 Contents Contents ListofFigur s s b p 8 8 amp Be Be 8 oe ee we we ew be eo oe a Xi List of Tables s 26 i gt Ge dn al UR Re OK dk owl Oe ea ee Ge Gh OEY About This Guide XVII Audience a lt 2 Sho Side ot ee ke Se RU Ot er EE ee ee a eI OR ee SEV ImportantInformation 2 ee XVI Chapter Descriptions 2 2 XVII Related Publications
14. Extracting the Carrier Metal Filler Plate Adjusting the Carrier Guide Bar Mounting Card in Carrier Mounting Half height PCI Card into Carrier Moving Carrier Guide Bar to Secure Half height PCI Card Installing the Bracket to Secure a Half height PCI Card Installing a Card in a Slot 59 68 69 70 71 TI 80 82 83 103 104 111 112 113 114 115 116 117 119 120 122 123 125 126 127 129 130 131 132 133 133 134 135 007 4857 002 List of Figures 007 4857 002 Figure 6 23 Figure 6 24 Figure 6 25 Figure 6 26 Figure 6 27 Figure 6 28 Figure 7 1 Figure 7 2 Figure A 1 Figure A 2 Figure A 3 Figure A 4 Figure B 1 Figure B 2 Figure B 3 Seating the Card in the Slot Removing the Card Access Panel From the IA Blade Replacing the IA Blade PCI Access Door Location of Disk Drive Bays Removing a Disk Drive Replacing a Disk Drive am a Compute Blade Status LED Locations Full Support Sequence Ethernet Port Serial Port Connector Pin Number Locations for External SAS SATA Port Pin Number Locations for USB Type A Connector VCCI Notice Japan Only Chinese Class A Regulatory Notice Korean Class A Regulatory Notice 135 136 137 138 139 141 149 150 159 160 162 163 169 169 169 xiii List of Tables 007 4857 002 List of Tables Table 1 1 Installation Tools Table 1 2 IRU Space Requirements Table 5 1 Tall Altix Rack Technica
15. The following are ten common subchannels associated with console communications e Subchannel OA specifies Blade 0 CPU A e Subchannel OC specifies Blade 0 CPU C e Subchannel 1A specifies Blade 1 CPU A e Subchannel 1C specifies Blade 1 CPU C e Subchannel 2A specifies Blade 2 CPU A e Subchannel 2C specifies Blade 2 CPU C e Subchannel 3A specifies Blade 3 CPU A 007 4857 002 L2 Operation 007 4857 002 e Subchannel 3C specifies Blade 3 CPU C e Subchannel console0 Blade 0 console subchannel e Subchannel console Blade 1 console subchannel The select command output console input 001c01 console0 shows that the L2 will send console input to IRU 001c01 blade 0 and the console subchannel will be used To change the IRU that will be the system console use the select lt rack gt lt slot gt command where lt rack gt is the rack and lt slot gt is the slot where the IRU is located L2 gt select 1 1 console input 001c01 console console output no filtered console detection L2 detected To change the subchannel used on the selected IRU use the select subchannel lt OA OC 1A 1C gt command Use the select subchannel console to select the current console as the subchannel of the IRU to be the system console For example to select blade 1 CPU A as the subchannel of the IRU to be the system console enter the following L2 gt select subchannel 1A console input 001c01 console CPUIA console ou
16. The following topics are covered e Preparing the System for Maintenance or Upgrade on page 108 e Returning the System to Operation on page 108 e Installing or Removing Internal Parts on page 109 007 4857 002 107 6 Maintenance and Upgrade Procedures Preparing the System for Maintenance or Upgrade To prepare the system for maintenance follow these steps 1 If you are logged on to the system log out Follow standard procedures for gracefully halting the operating system At your system console switch from console mode to system control mode by entering the following command gt Ctrl T The system will respond with the system control prompt L2 gt From the system control prompt L2 gt power off the system with the following command L2 gt power down Locate the power distribution unit s PDUs in the front of the rack and turn off the circuit breaker switches on each PDU Returning the System to Operation When you finish installing or removing components return the system to operation as follows 1 2s Turn each of the PDU circuit breaker switches to the on position At your system console enter the following command L2 gt power up Verify that the LEDs on the L1 controller s turn on and illuminate green and that your controllers display that the system is powered on for each segment of the procedure which indicates that the power on procedure is proceeding properly
17. This microprocessor runs an embedded version of the Linux operating system The system controller runs off standby power and is running as long as the enclosure is connected to an active power source There are two primary applications that run on the system controller The L1 or Level 1 system controller is an application that provides control and monitoring functionality for each individual rack unit enclosure IRU and communication to other L1s in adjacent enclosures connected via NUMALIink 4 cables The L1 is always resident The console mode Level 2 L2 system controller is an application that provides control over multiple L1s and communication to other L2s The L2 is resident and active when the enclosure is connected by an Ethernet connection to a Local Area Network LAN The system controller network provides the following functionality e Powering the entire system on and off e Powering individual IRUs on and off e Monitoring the environmental state of the system e Viewing the system s status and error message information generated and displayed by the SGI system s L1 controller e Enter L1 controller commands to monitor or change particular system functions You can for example monitor the speed of fans for a particular individual rack unit IRU enclosure See the SGI LI and L2 Controller Software User s Guide for a complete list of commands e Provides access to the system OS console allowing you to run diagnosti
18. photograph it for reference After you remove the contents keep the damaged container and the packing materials Remove the IRU s from the packaging container and ensure that all accessories are included Inspect the IRU s and accessories for damage If the contents appear damaged file a damage claim with the carrier immediately In addition notify your local SGI Customer Support Center CSC for any missing incorrect or damaged items For CSC contact information see http www sgi com support supportcenters html Note When using the IRU as a stand alone unit set on a desk or table top you must secure the IRU and its fan module together with connection plates RUs mounted in a rack are held in place by the screws at the front and rear of the unit 007 4857 002 Unpacking and Inspecting Modules Firmly connect the base module of the IRU to the fan module using these steps 1 Align the rear connectors of the fan module with the rear of the IRU base module and carefully slide it in until it fully joins together 2 Align the right side joining plate screw holes with the eight holes in the two adjacent modules See Figure 1 10 for an example 3 Insert and tighten the eight retention screws that hold the connection plate in place 4 Repeat the process for mounting the joining plate on the left side of the IRU Figure 1 10 IRU Connection Plate for Desktop Systems 007 4857 002 23 1 System Installation Ins
19. w Enable logging A Ne Main menu select an option 1 99 or e p w or 0 to quit 8 Important After this command is entered the load device cannot be accessed again to open files until 1siutil has been exited and restarted SAS1068 s links are 3 0 G 3 0 G down down down down down down B_T__ L Type Vendor Product Rev SASAddress PhyNum 0 O 0 Disk SGI ST3146854SS X422 5000c5000002cb45 0 0 1 OO Disk SGI ST3146854SS X422 5000c500000121cd 1 Main menu select an option 1 99 or e p w or 0 to quit 0 Port Name Chip Vendor Type Rev MPT Rev Firmware Rev IOC 1 01 00 01 0 LSI Logic SAS1068 BO 105 01100000 0 Select a device i 1 or 0 to quart 1 Installing Optional Components 007 4857 002 1 Identify firmware BIOS and or FCode 2 Download firmware update the FLASH 4 Download erase BIOS and or FCode update the FLASH 8 Scan for devices 10 Change IOC settings interrupt coalescing 13 Change SAS IO Unit settings 16 Display attached devices 18 Change WWID 20 Diagnostics 21 RAID actions 22 Reset bus 23 Reset target 24 Clear ACA 39 Force firmware download boot 45 Concatenate SAS firmware and NVDATA files 60 Show non default settings 61 Restore default settings 97 Reset SAS phy 98 Reset SAS link 99 Reset port e Enabl xpert mode in menus p Enable paged mode in menus w Enable logging Main menu select an option 1 99 or e p w or 0 to quit 21 1 Show volumes
20. 4 445 cm Because IRUs occupy multiple standard units IRU locations within a rack are identified by the bottom unit U in which the IRU resides For example in a 42U rack an IRU positioned in U01 through UOS5 is identified as U01 Each rack is numbered with a three digit number sequentially beginning with 001 A rack contains IRU enclosures optional mass storage enclosures and potentially other options In a single compute rack system the rack number is always 001 77 3 System Overview Optional System Components Availability of optional components for the SGI 450 systems may vary based on new product introductions or end of life components Some options are listed in this manual others may be introduced after this document goes to production status Check with your SGI sales or support representative for the most current information on available product options not discussed in this manual 78 007 4857 002 Chapter 4 System Control This chapter describes the interaction and functions of system controllers in the following sections e Levels of System Control on page 80 e System Controller Interaction on page 81 e L1 Controller on page 81 e Console Hardware Requirements on page 84 e Operating the L1 on page 84 e L2 Operation on page 89 e Upgrading L1 Firmware on page 99 The control system for the SGI Altix 450 series servers manages power control and sequencing pr
21. 64 bit processor cores in a single rack Because it is modular the DSM combines the advantages of lower entry cost with the ability to scale processors memory and I O independently The system architecture for the Altix 450 system is a fourth generation NUMAflex DSM architecture known as NUMAlink 4 In the NUMAlink 4 architecture all processors and memory are tied together into a single logical system with special crossbar switches routers This combination of processors memory and crossbar switches constitute the interconnect fabric called NUMAlink There are two internal router switches on each 5U IRU enclosure The basic expansion building block for the NUMAlink interconnect is the processor node each processor node consists of a Super Hub SHub ASIC and one or two 64 bit processors with three levels of on chip secondary caches The Intel 64 bit processors are connected to the SHub ASIC via a single high speed front side bus The SHub ASIC is the heart of the processor and memory node blade technology This specialized ASIC acts as a crossbar between the processors local SDRAM memory and the network 007 4857 002 System Architecture 007 4857 002 interface The SHub ASIC memory interface enables any processor in the system to access the memory of all processors in the system Another component of the NUMAIink 4 architecture is the router ASIC The router ASIC is a custom designed 8 port crossbar ASIC Using the router ASICs
22. Adding or Replacing PCI PCI X Cards 7 Insert the carrier mounted PCI card into the vacant slot using the slot guide as shown in Figure 6 22 Figure 6 22 Installing a Card in a Slot 8 Push in horizontally on the carrier seating bar as shown in Figure 6 23 to seat the card securely in the slot Figure 6 23 Seating the Card in the Slot 9 Install the blade into the IRU 10 Power on the IRU or restart the system 11 Run the 1s pci PCI hardware inventory command to verify the installation This command lists PCI hardware that the operating system discovered during the boot operation 007 4857 002 135 6 Maintenance and Upgrade Procedures Add or Remove a PCI Card in Base IA Blades To add or replace a PCI card in the base IA blade base I O follow these steps 1 Sh OY et oe Power off the IRU by following the power off instructions in the Powering the System On and Off on page 51 if you are removing the IA blade Extract the blade you are going to install the new PCI card in See the information in Replacing IRU Components on page 110 if you have not already removed the blade Note that in the case of extracting an IA blade that holds the system disk you must shut down the operating system After removing the IA blade from the IRU chassis place it on a stable flat surface with left side facing upward see Figure 6 24 Using a Tx20 Torx driver remove the front screw securing the PCI ca
23. B image At boot time the boot image validates the A and B image and if it is not instructed otherwise it executes the newer of the two images Because the L1 is running one of the two images the image not in use is the image that will be overwritten when the firmware is upgraded You need to re boot any L1 update either by power cycling the IRU or by using the L1 command reboot_11 Typically you will upgrade the firmware through the network connection from the console to the LI gt usr cpu firmware sysco flashse 12 10 1 1 1 p usr cpu firmware sysco 11 bin all This updates all the IRUs in the system The p at the end of the first line instructs the firmware to flash the proms in parallel 99 4 System Control 100 You can update individual IRUs by replacing all with a rack and slot number gt usr cpu firmware sysco flashsec 12 10 1 1 1 usr cpu firmware sysco 11 bin 1 16 This updates only the IRU in rack 1 slot 16 007 4857 002 Chapter 5 Overview 007 4857 002 Rack Information This chapter describes the physical characteristics of the tall 42U and short 20U Altix racks in the following sections e Overview on page 101 e Altix 42U and 20U Rack Features on page 102 e Technical Specifications on page 104 At the time this document was published only the tall 42U and short 20U Altix racks shown in Figure 5 1 and Figure 5 2 were tested and approved for use with Alti
24. IRU are plugged in View the L1 display see Table 7 2 if an error message is present If the L1 controller is not running contact your SSE Ensure the IA IA2 base I O blade that houses the system disk s is properly seated in the IRU Contact your SSE View the L1 display of the failing IRU see Table 7 2 for a description of the error message View the L1 display of the failing IRU see Table 7 2 fora description of the error message Ensure that the NUMAlink cable is seated properly on both ends Reseat the PCI card Check to make sure the blade is seated fully in the IRU Reseat the PCI card Check to make sure the blade is seated properly in the IRU If the fault LED remains on replace the PCI card Replace the disk drive 007 4857 002 L1 Controller Error Messages L1 Controller Error Messages 007 4857 002 Table 7 2 lists error messages that the L1 controller generates and displays on the L1 display This display is located on the front of the IRU Note In Table 7 2 a voltage warning occurs when a supplied level of voltage is below or above the nominal normal voltage by 10 percent A voltage fault occurs when a supplied level is below or above the nominal voltage by 20 percent Table 7 2 L1 Controller Messages L1 System Controller Message Message Meaning and Action Needed Internal voltage messages ATTN lt power VRM description gt high fault limit 30 second power off sequen
25. L1 Controller Functions The following list summarizes the control and monitoring functions that the L1 controller performs e Controls voltage margining within the IRUs e Controls and monitors IRU fan speeds e Reads system identification ID PROMs e Monitors voltage levels and reports failures e Monitors and controls warning LEDs on the IRU e Monitors the On Off power switch e Monitors the reset switch and the nonmaskable interrupt NMI switch e Reports the population of the PCI cards and the power levels of the PCI slots in installed I O blades e Powers on the PCI slots and their associated LEDs L1 Front Panel Display Figure 4 2 shows the L1 controller front panel features L1 controller display Failure LED Service required LED Reset Power button with LED Figure 4 2 L1 Front Panel 82 007 4857 002 Ethernet Switch The front panel display contains the following items e 2x 12 character liquid crystal display LCD The display identifies the IRU shows system status warns of required service and identifies a failed component e Power On Off button insert paper clip to actuate and power on LED e Service required LED e Failure LED e Reset switch insert paper clip to actuate e Non maskable interrupt NMI switch insert paper clip to actuate Note The reset and NMI switch functions on any IRU in a NUMAlink connected system will affect all the IRUs in the system Ethernet Switc
26. and the module is down e Power button Normally you power on the system from the system console Alternatively you can press this button to power on the module e Reset button Actuate this button to reset the internal processors and ASICs The reset will cause a memory loss To perform a reset without losing memory see the NMI button information that follows e NMI button Actuate the NMI non maskable interrupt button to reset the internal processor s and ASICs and write register data and memory to the var adm crash file Use a paperclip or other small object to access the pin hole actuators on the front panel L1 controller display Failure LED Service required LED Reset Power button with LED Figure 1 22 L1 Front Panel Functions 44 007 4857 002 Operating Your System Note If your system includes an optional storage module see the user s guide that comes with the product for operating instructions Operating the L1 Controller 007 4857 002 The L1 controller operates in one of the two following modes e L1 mode The L1 prompt 001c01 L1 gt is visible and all input is directed to the L1 command processor e Console mode from L1 Output from the system is visible and all input is directed to the system The L1 controller is ready to accept commands when you see a prompt of the following form 001c01 L1 gt See the SGI LI and L2 Controller Software User s Guide 007 3938 00x for a
27. be if the LAN cable is plugged in and the cable is connected to an active LAN At the L1 prompt type 001c01 L1 gt 12 L2 Controller is running 001c01 L1 gt If the L2 is not running type 001c01 L1 gt init 4 This switches the system controller to run level 4 and forces the L2 to be started whether or not the LAN is plugged in There is a space between the and init Verify the L2 is running again as above To set the IP address on the L2 type 001c01 L1 gt 12 ip a b c d 255 255 255 0 a b c 255 Verifying that the system serial number is set on the L2 001c01 L1 gt 12 serial To set the L2 system serial number 001c01 L1 gt 12 serial set lt serial number gt Verifying that msys is enabled this allows multiple L2s in a system to exist peacefully with other L2s from another system on the same subnet 001c01 L1 gt 12 msys If msys is off turn it on 001c01 L1 gt 12 msys on Reboot the system controller to make the IP address change take effect 001c01 L1 gt reboot_11 E Once this is done for all target IRUs and Dense routers connect them to the network using an optional Ethernet switch if necessary The rackidon the L2 cannot be set with the L2 rackid command instead it will be inherited from the local L1 As an example the L2 running on the system controller in 1r41 will have a rack id of 141 rack 100 slot of the local L1 007 4857 002 L2
28. connect either a local or remote workstation to monitor the system via Ethernet These console connections enable you to view the status and error messages generated by the L1 L2 controllers in your Altix 450 rack You can also use these consoles to input L1 L2 commands to manage and monitor your system e A DVI VGA video card allows you to interact directly with the system though not directly with the system s L1 L2 interface See 2D Graphics Video Interface on page 58 for more information SLES 10 or later is required for this card RHEL 5 is not supported L1 controller panel Oeeeeeeeee Ollie i o 33 Port O EHe H iag 3 3 a D e xel i g Port C S A lt Port G H f Eii Sm o f F 5 Power Compute memory blades supplies Router IA blade System control Router board board board Figure 2 3 IRU L1 Controller Display Location Example 007 4857 002 57 2 Operation Procedures 2D Graphics Video Interface 58 If your system was ordered in January 2007 or later it may come equipped with an optional 2D graphics board interface supported with SLES 10 or later system software This low profile PCI interface card is installed in the base I O blade in the Altix system see Figure 2 4 on page 59 Note that this blade is commonly referred to as the IA2 blade One 2D card is supported per system or partition no audio function is supported The 2
29. default destination L2 gt destination 001c01 127 0 0 1 001c01 127 0 0 1 L2 gt 0 2 0 0 The following command resets the default destination to all racks and all slots L2 gt destination reset default destination reset to all racks and slots L2 gt Current Destination The current destination is a range of racks and slots for a given command For example the following command sends the command lt L1 command gt to all IRUs in racks 2 3 4 and 7 93 4 System Control L2 gt r 2 4 7 lt L1 command gt This is a one time destination Command Interpretation Some L2 commands are the same as the L1 commands In many cases this is intentional because the L2 provides sequencing that is necessary for a command to function correctly When L1 and L2 commands are similar you can ensure that an L1 command is entered for the IRUs current destination by preceding the command lt L1 command gt with the 11 command L2 gt r 2 4 7 11 lt L1 command gt This is a one time destination Viewing Information Warnings and Error Messages All information warnings and error messages generated by any of the system controllers are in the following form 001c01 ERROR invalid arguments for ver command try help ver The general format includes an IRU identification and the type of message followed by the message A message may be the result of an invalid command as shown in the example or th
30. detailed list of L1 commands To enter console mode press Ctr1 D at the L1 prompt as follows 001c01 L1 gt Ctrl D entering console mode 001c01 console lt CTRL T gt to escape to L1 lt system output appears here gt To return to L1 mode press Ctr1 T as follows Ctrl T escaping to L1 system controller 001c01 L1 gt At this point you can enter any L1 command When the command completes execution the L1 returns to console mode re entering console mode 001c01 console lt CTRL T gt to escape to L1 To permanently engage the L1 mode press Ctr1 T and then enter the 11 command as follows Ctrl T 45 1 System Installation 46 escaping to L1 system controller 001c01 L1 gt 11 L1 command processor engaged 001c01 L1 gt lt CTRL D gt for console mode 007 4857 002 Chapter 2 Precautions ESD Precaution 007 4857 002 Operation Procedures This chapter explains how to operate your new system in the following sections e Precautions on page 47 e System Controller Network on page 49 e Powering the System On and Off on page 51 e Using Embedded Support Partner ESP on page 56 e Monitoring Your Server on page 57 e Installing Optional Components on page 59 Before operating your system familiarize yourself with the safety information in the following sections e ESD Precaution on page 47 e Safety Precautions on page 48 Caution
31. follow these steps 1 Grasp the rear of each rack and roll the rack to its designated location 2 Remove the ESD bags 3 Adjust the leveling bolts of a rack as shown in Figure 1 8 until the rack is level Figure 1 8 Leveling Bolts 4 If you are installing additional racks repeat until all of the racks are level 5 Ensure that the circuit breakers on the power distribution units are in the OFF O position Then connect the power cords to grounded power outlets Plugging in the power cords grounds the racks 6 It is recommended you secure all racks with optional seismic tie downs if you are installing the system in an earthquake zone Note Tall and short racks have four threaded holes that are located at the bottom of the rack see Figure 1 9 Use these holes to secure the seismic tie downs Note that SGI does not supply the seismic tie downs 21 1 System Installation Top view of short rack Top view of tall rack e Seismic tie down attachment points O Casters m Leveling pads Figure 1 9 Seismic Tie down Attachment Points Unpacking and Inspecting Modules 22 This section describes how to unpack and inspect individual IRUs Follow these instructions if you plan to install individual modules in a rack or on a table top Before unpacking your IRUs inspect the packaging container for evidence of mishandling during transit If the packaging container is damaged
32. from the IRU power supply 2 Press down on the retention latch at the top of the power supply see Figure 6 3 3 Using the power supply handle pull the power supply straight out of the IRU Figure 6 3 Removing an IRU Power Supply 007 4857 002 113 6 Maintenance and Upgrade Procedures Install the new power supply with the retention latch at the top of the supply Slide the power supply into the chassis until the retention latch engages Reconnect the power cord to the power supply SY Oo On eS Reattach the L1 front panel using the information in Remove and Replace the System Control Display Panel on page 110 Figure 6 4 Replacing an IRU Power Supply Removing and Replacing IRU Fans Blowers 114 The IRU fans blowers are located at the rear of the unit see Figure 6 5 You will need to access the rack from the back to remove and replace a fan The IRU s system controller issues a warning message when a fan is not running properly This means the RPM level of the fan is not within tolerance When an IRU fan fails the following things happen 1 The L1 display on the IRU with the failed fan shows a warning message ATTN Fan number warning 2 The console will show the same warning indicating the rack and IRU position 001c01 L2 gt Fan number warning limit reached 0 RPM 007 4857 002 Replacing IRU Components 3 A line will be added to the L1 system controller s log file indicating th
33. gt Left bus Y DIMM 2 DIMM location 5 Left bus Y DIMM 1 DIMM location 4 E Left bus Y DIMM 0 DIMM location 3 e Left bus X DIMM 2 DIMM location 2 Left bus X DIMM 1 DIMM location 1 Left bus X DIMM 0 DIMM location 0 Group 2 Memory NUMAlink 4 SAW osc Group 1 Group 0 Front side bus FSB NUMAlink 4 Memory L gt Right bus X DIMM 0 DIMM location 0 _ Right bus X DIMM 1 DIMM location 1 a Right bus X DIMM 2 DIMM location 2 L gt Right bus Y DIMM 0 DIMM location 3 _ Right bus Y DIMM 1 DIMM location 4 lt m gt Right bus Y DIMM 2 DIMM location 5 Figure 6 10 DIMM Group Locations Diagram 122 007 4857 002 Replacing IRU Components DIMM slots Figure 6 11 DIMM Memory Slot Location Example Remove and Replace the DVD Drive 007 4857 002 Use the following steps to remove the DVD drive from an JA blade 1 2 a N A 2 Power off the IRU use instructions in Powering the System On and Off on page 51 Extract the IA blade you are going to install the new DVD drive in See the information in Replacing IRU Components on page 110 After removing the IA blade from the IRU chassis place it on a stable flat surface with the left side facing upward see Figure 6 12 Use a Phillips type screwdriver to remove the DVD access panel screw see Figure 6 1
34. of PCI Express Operation Adding or Replacing PCI PCI X Cards Installing PCI Cards in the Three Slot Blade Add or Remove a PCI Card in Base IA Blades Installing or Replacing a Disk Drive in the IA Blade Removing a System Disk Drive Installing a System Disk Drive Troubleshooting and Diagnostics Troubleshooting Chart L1 Controller Error Messages LED Status Indicators IRU Power Supply LEDs IRU NUMAlink Router Port LEDs Compute Memory Blade LEDs SGI Electronic Support Technical Specifications and Pinouts System level Specifications 102 104 107 107 108 108 109 110 110 113 114 118 118 121 123 127 128 129 136 138 139 140 143 144 145 147 147 147 148 149 153 153 Contents Physical Specifications Environmental Specifications Power Specifications T O Port Specifications Ethernet Port Serial Ports Be hes ey ot External SAS SATA Port Connector USB Type A Connector Safety Information and Regulatory Specifications Safety Information Regulatory Specifications CMN Number CE Notice and Manufacturer s Declaration of Conformity Electromagnetic Emissions FCC Notice USA Only Industry Canada Notice Canada Only VCCI Notice Japan Only Chinese Class A Regulatory Notice Korean Class A Regulatory Notice Shielded Cables Electrostatic Discharge Laser Compliance Statements Lithium Battery Statements Index 154 156 157 158 159 160 1
35. press Ct r1 D at the L2 prompt and observe the response L2 gt Ctrl D entering system console mode 001c01 console0 lt CTRL_T gt to escape to L2 lt system output appears here gt To return to L2 mode from console mode press Ct r1 T Ctr1 T escaping to L2 system controller L2 gt At this point you can enter any L2 or L1 command When the command completes the L2 returns to console mode Re entering system console mode 001c01 console0 lt CTRL_T gt to escape to L2 To permanently engage the L2 mode press Ct r1 T and then enter the 12 command 95 4 System Control Console Selection 96 Ctr1 T escaping to L2 system controller L2 gt 12 L2 command processor engaged lt CTRL_D gt for console mode L2 gt When in console mode the L2 communicates with the IRU set with the select command to be the system console or global master All input from the console is directed to that IRU You can set and view the system console with the select command The L2 chooses an IRU as the default console in the following order of priority e The IRU in the lowest numbered rack and slot which has previously produced console output The IRU in the lowest numbered rack and slot The select command by itself shows the current console mode settings L2 gt select known system consoles non partitioned 001c01 L2 detected current system console console input 001c01 CPU OA console output not filtered
36. rack s power distribution unit PDU and the wall power plug receptacle is secure For each individual IRU that you want to power on make sure that the power cables are plugged into all the IRU power supplies correctly as shown in Figure 2 1 Setting the circuit breakers on the PDUs to the On position will apply power to the IRU and will start the system controller s in the IRUs Note that the system controller in each IRU stays powered on as long as there is power coming into the unit CTO ODEPY eeeceececcces oo d AN p gt N Jo ee C2 o D SS 8 4 m m m AA YEE Figure 2 1 IRU Power Supply Cable Location Example 3 52 If you plan to power on a server that includes optional mass storage enclosures make sure that the power switch on the rear of each PSU cooling module one or two per enclosure is in the 1 on position Make sure that all PDU circuit breaker switches see the example in Figure 2 2 on page 53 are turned on to provide power to the server system when it is switched on 007 4857 002 Powering the System On and Off Power distribution Ss PDU eS 2 N On off switches Figure 2 2 Single Phase Rack PDU Circuit Breaker Switches 007 4857 002 53 2 Operation Procedures Powering On and Off at the Console The power on and off procedure at a console varies with your server setup as follows e Ifyou have a co
37. section Warning Before installing operating or servicing any part of this product read the Safety Information on page 165 System I O is primarily accomplished through standard and optional blades in the SGI Altix 450 IRUs The PCI PCI X and PCIe based I O sub systems are industry standard for connecting peripherals storage and graphics to a processor blade These are the primary configurable I O system interfaces for the Altix 450 series systems e The IA TA2 blade base I O used in the Altix 450 system includes two half height PCI PCI X slots The two option cards install adjacent to the system disk and DVD drives that are also resident in the IA blade The IA2 blade differs from the original IA base I O in that it supports RAID 1 disk mirroring and DVD R W capabilities e The optional three slot PCI PCI X double wide blade holds three sled mounted PCI PCI X cards for easy insertion and extraction from the system It is only used in A450 systems that have more than one IRU e The optional single wide two slot PCI X blade supports two PCI or PCI X option cards e The optional single wide PCI express PCIe blade supports two optional PCI express cards e The optional double wide PCI express PCIe blade is used only in Altix 450 systems with two or more IRUs It houses two PCIe slots and two PCI X slots Note that use of two PCIe 3D graphics cards in this blade precludes using the two PCI X slots due to space limitati
38. the rear of the crate and secure the ramp in place using the two bolts removed from the pallet s rear gate 6 Using two people carefully pull the rack off the crate and roll it down the ramp Warning The maximum weight of the short rack is 750 Ibs 340 kg Use caution when you roll the rack down the ramp 007 4857 002 Installing a Rack Rear mounting bracket Front mounting bracket Figure 1 5 Removing a Short Rack from the Shipping Crate 007 4857 002 17 1 System Installation Unpacking and Installing the Tall Rack 18 Warning In its maximum configuration a 42U tall rack system weighs approximately 1 450 Ib 658 kg Use caution when you unpack and move this rack A pallet jack is needed to move the rack prior to unpacking Ensure that the rack remains on a level surface and that the rack weight remains evenly distributed across the four casters To unpack a tall rack you will need the following tools e Extension 6 inch 15 3 cm 3 8 in 13mm drive e 13 mm standard 3 8 in drive socket e Ratchet reversible 3 8 in 13mm drive See Figure 1 6 as you follow these steps the numbered illustrations correspond to the numbered steps Ensure that the temperature of the rack is acclimated to the installation environment and that the system crate is stable and in an upright position 1 Use a sharp cutting tool to remove the bands that secure the cardboard crate then remove the outer packi
39. with a highly specialized backplane and NUMAlink 4 cables provides a high bandwidth extremely low latency interconnect between all processor I O and other option blades within the system Figure 3 4 shows a functional block diagram of the Altix 450 series IRU components Blade Stato ______ Left router board Right router board Blade Slot 1 i L D B Stab 0 ort Saas Sure a c Slab 1 ti J rtr NUMAIink G A C i p E B G E NUMAIink l Slot 3 NUMAlink Ay H A BLY i A H NUMAlink Blade Slot 4 DC power System control bus Power Supply Power Supply ee e ee ee ee es ee ae Ss ee es aU Figure 3 4 Functional Block Diagram of the Individual Rack Unit IRU 71 3 System Overview System Features The main features of the Altix 450 series server systems are introduced in the following sections e Modularity and Scalability on page 72 e Distributed Shared Memory DSM on page 72 e Distributed Shared I O on page 73 e Reliability Availability and Serviceability RAS on page 74 Modularity and Scalability The Altix 450 series systems are modular systems The components are primarily housed in building blocks referred to as individual rack units IRUs Additional optional mass storage may be added to the rack al
40. 001c01 Viewing Information Warnings and Error Messages L2 Operation All information warnings and error messages generated by any of the system controllers are in the following form 001c01 ERROR invalid arguments for ver command try help ver The general format includes a IRU identification and the type of message followed by the message A message may be the result of an invalid command as shown in the example or the result of tasks running on the L1 such as the environmental monitor Each L1 has a log of local events Use the L1 command 1og to view events on any of the LIs As mentioned in System Controller Interaction on page 81 the system controller in an Altix 450 system can provide L2 functionality An Ethernet cable can be plugged into the RJ45 connector on the IRU enclosure Connecting the IRU to an active LAN via the L2 host connector will cause the system controller to spawn an L2 This connection provides network access to the system controller through the L2 Configuring an L2 s IP Address 007 4857 002 This section refers to setting the IP address on the IRU enclosure when using an Ethernet connection Setting the IP address of the L2 on the target IRUs should be done before connecting the IRUs to the network as follows Connect a serial cable to the serial console port on the target IRU and get the L1 prompt 89 4 System Control 90 To see if the L2 is running it will
41. 1 system controller This connection is typically used for service purposes or for system controller and system console access in small systems where an ethernet connection is not available Caution An L1 L2 network connection must be kept on a private and physically secure network connection The System Controller does not provide software security authentication or safeguards against malicious or careless users The consequences of unauthorized access to the System Controllers are potential system unavailability system reconfiguration and detailed access to every aspect of the system The recommended configuration for the L1 L2 network connection is a separate private and physically secure network attachment to an equally separate and physically secure PC Any Linux system hosting L3 software must be administered to prevent unauthorized access Optional L3 software is a package that runs on a Linux based PC It is mainly used by service personnel as a diagnostic aid 007 4857 002 Communicating with the System Controller Communicating with the System Controller Once a connection to the console is established the user will be presented with either an L2 prompt ethernet connection known as L2 mode or and L1 prompt serial connection known as L1 mode From either of these prompts various system controller commands can be entered 001c01 L1 gt lt 11 command gt olympic 101 L2 gt lt 12 command gt To access the system con
42. 2 Lift the DVD access panel clear of the IA blade and set it aside Unscrew and remove the DVD drive assembly retention screw Gently but firmly disengage the DVD power and data cable connectors from the old DVD drive and slide the DVD unit out of the IA blade housing 123 6 Maintenance and Upgrade Procedures 124 10 11 12 13 Slide the new DVD drive assembly into the IA blade housing Connect the DVD power and data cable connectors to the replacement drive Screw in the DVD drive assembly retention screw Replace the DVD access panel and tighten the retention screw Return the IA blade to its IRU slot Power up the IRU and or system 007 4857 002 Replacing IRU Components DVD retention gt screw Power connector Data connector Figure 6 12 DVD Removal from IA Blade 007 4857 002 125 6 Maintenance and Upgrade Procedures Power connector Data connector Figure 6 13 DVD Drive Replacement in the IA Blade 126 007 4857 002 Overview of PCI Express Operation Overview of PCI Express Operation 007 4857 002 This section provides a brief overview of the new PCI Express PCIe technology that will be available as an option with your system PCI Express has both compatibility and differences with older PCI PCI X technology Check with your SGI sales or service representative for more detail on PCI Express options available with the SGI Altix 450 PCI Express is compat
43. 2 Troubleshooting and Diagnostics This chapter provides the following sections to help you troubleshoot your system e Troubleshooting Chart on page 144 e L1 Controller Error Messages on page 145 e SGI Electronic Support on page 149 143 7 Troubleshooting and Diagnostics Troubleshooting Chart 144 Table 7 1 lists recommended actions for problems that can occur To solve problems that are not listed in this table use the SGI Electronic Support system or contact your SGI system support engineer SSE For more information about the SGI Electronic Support system see the SGI Electronic Support on page 149 Table 7 1 Troubleshooting Chart Problem Description Recommended Action The system will not power on An individual IRU will not power on The system will not boot the operating system The Service Required LED illuminates on an IRU The Failure LED illuminates on an IRU The green or yellow LED of a NUMAlink port is not illuminated The PWR LED of a populated PCI slot is not illuminated The Fault LED of a populated PCI slot is illuminated on The amber LED of a disk drive is on Ensure that the power cords of the IRU are seated properly in the power receptacles Ensure that the PDU circuit breakers are on and properly connected to the wall source If the power cord is plugged in and the circuit breaker is on contact your SSE Ensure the power cables of the
44. 2 Show physical disks 3 Get volume state 23 Replace physical disk 30 Create volume 31 Delete volume 32 Change volume settings 50 Create hot spare 99 Reset port e Enabl xpert mode in menus p Enable paged mode in menus w Enable logging RAID actions menu select an option 1 99 or e p w or 0 to quit 1 0 volumes are active 0 physical disks are active RAID actions menu select an option 1 99 or e p w or 0 to quit 30 63 2 Operation Procedures 64 B rE L Type Vendor Product Rev Disk Blocks Disk MB Ls 0 0 0O Disk SGI ST3146854SS X422 286749488 140014 2 O iL 0 Disk SGI ST3146854SS X422 286749488 140014 To create a volume select two or more of the available targets Select a target 1 2 or RETURN to quit 1 Select a target 1 2 or RETURN to quit 2 2 physical disks were created Select volume type O Mirroring 1 Striping default is 0 Select volume size 1 to 139898 MB default is 139898 Enable write caching Yes or No default is No Zero the first and last blocks of the volume Yes or No default is No Volume was created RAID actions menu select an option 1 99 or e p w or 0 to quit 1 1 volume is active 2 physical disks are active Volume 0 is Bus 0 Target 0 Type IM Integrated Mirroring Volume State degraded enabled resync in progress Volume Settings write caching disabled auto configure Volume Size 139898 MB Stripe Size 0 KB 2 Members Primary is PhysDisk 0 Bus 0 Target 56
45. 62 163 165 165 167 167 167 168 168 169 169 169 169 170 170 171 172 173 007 4857 002 List of Figures 007 4857 002 List of Figures Figure 1 1 30 amp Single phase Power Receptacle for North American Sites Figure 1 2 32 amp Single phase Power Plug for International Sites Figure 1 3 Dimensions of Tall Rack Shipping Crate Figure 1 4 Dimensions of Short Rack Shipping Crate Figure 1 5 Removing a Short Rack from the Shipping Crate Figure 1 6 Removing a Tall Rack from the Shipping Crate Figure 1 7 Reattaching the Tall Rack Doors Figure 1 8 Leveling Bolts Figure 1 9 Seismic Tie down Attachment Points Figure 1 10 IRU Connection Plate for Desktop Systems Figure 1 11 Mounting Hole Pattern of Rack Rails Figure 1 12 Installing the Shelf Rails in the Rack Figure 1 13 Securing the Module to the Front of the Rack Figure 1 14 NUMAlink Cabling an IRU to an Additional IRU Figure 1 15 Power Supply Connectors on the IRU Module Figure 1 16 Connecting an Unracked IRU System to a Power Source Figure 1 17 Connecting Multiple Modules to a PDU Figure 1 18 Optional Three Phase PDU Example Figure 1 19 Location of the L1 Console Port Figure 1 20 PDU Circuit Breakers Example Figure 1 21 Location of the Power Button Figure 1 22 L1 Front Panel Functions og 28 Figure 2 1 IRU Power Supply Cable Location Example Figure 2 2 Single Phase Rack PDU Circuit Breaker Switches Figure 2 3 IRU L1 Controller Display Location Example
46. 8 75 kW max per short rack Hold up time 16 ms Total harmonic distortion Less than 10 at full load 007 4857 002 157 A Technical Specifications and Pinouts I O Port Specifications This section contains specifications and port pinout information for the base I O ports of your system as follows e Ethernet Port on page 159 e Serial Ports on page 160 e External SAS SATA Port Connector on page 162 e USB Type A Connector on page 163 158 007 4857 002 T O Port Specifications Ethernet Port The system auto selects the Ethernet port speed and type duplex vs half duplex when the server is booted based on what it is connected to Figure A 1 shows the Ethernet port Pin 4 Pin 5 Pin 3 Pin 6 Pin 2 Pin 7 Pin 1 Pin 8 Figure A 1 Ethernet Port Table A 7 shows the cable pinout assignments for the Ethernet port operating in 10 100 Base T mode and also operating in 1000Base T mode Table A 7 Ethernet Pinouts Ethernet 10 100Base T Pinouts Gigabit Ethernet Pinouts Pins Assignment Pins Assignment 1 Transmit 1 Transmit Receive 0 2 Transmit 2 Transmit Receive 0 3 Receive 3 Transmit Receive 1 4 NU 4 Transmit Receive 2 5 NU 5 Transmit Receive 2 6 Receive 6 Transmit Receive 7 NU 7 Transmit Receive 3 8 NU 8 Transmit Receive 3 NU Not used 007 4857 002 159 A Technical Specifications and Pinouts Serial Ports 160 The I
47. Before you install an Altix 450 system you should familiarize yourself with the safety precautions discussed in the following subsections Hazard Statements on page 4 ESD Precautions on page 4 Safety Measures on page 5 1 System Installation Hazard Statements ESD Precautions ZN During the installation of your Altix 450 system be alert for hazard advisory statements with icons which signify the following Caution indicates a potentially hazardous situation that if not avoided can result in minor or moderate injury A caution statement also alerts you to unsafe practices that can result in equipment damage and or data corruption A caution message is accompanied by an icon as shown in the following example N Caution Warning indicates a potentially hazardous situation that if not avoided could result in death or serious injury A warning message is accompanied by icon as shown in the following example Warning Danger indicates an imminently hazardous situation that if not avoided will result in death or serious injury A danger message is accompanied by the same icon as a warning Observe electrostatic discharge ESD precautions during the entire installation process to eliminate possible ESD damage to the equipment Wear an SGI approved wrist strap when you handle an ESD sensitive device Connect the wrist strap cord directly to earth ground Caution Observe all ESD precau
48. D video card has the following features 64 MB DDR graphics memory DMS 59 display connector for two display support no synchronization between monitors Simultaneous display to DVI and VGA monitors using an adapter Analog resolution up to 2048 x 1536 per display DVI resolution up to 1600 x 1200 per display Note A single USB keyboard mouse is supported by the IA2 base I O blade The 2D video interface card can be used for all basic interaction with your Altix system You should note that it does not provide an interface to the L1 L2 controller and L1 L2 administrative commands cannot be issued through the 2D video interface 007 4857 002 Installing Optional Components 800000008 2D graphics card Figure 2 4 2D Graphics Card Location in IA2 base I O Blade Installing Optional Components 007 4857 002 Besides adding a system console you can add or replace the following hardware items on your Altix 450 series server e Peripheral component interface PCI cards into your system I O blades e Disk drives in your IA IA2 blade base I O The sections that follow discuss these activities in more detail 59 2 Operation Procedures I O Blade Overview 60 Warning You can add or replace only the items listed in this section For your safety and for the protection of your server system contact your SGI system support engineer SSE to install any hardware items not listed in this
49. L T gt to escape to L2 001c01 L1 gt To return to L2 mode press Ct r1 T 001c01 L1 gt Ctr1 T escaping to L2 system controller lt CTRL T gt to send escape to L1 L2 gt At this point you can enter any L2 command Once the command is executed the L2 returns to L1 mode re entering L1 mode 001c01 lt CTRL T gt to escape to L2 001c01 L1 gt To permanently engage the L2 mode press Ct r1 T and enter the 12 command 001c01 L1 gt Ctr1 T escaping to L2 system controller lt CTRL T gt to send escape to Ll L2 gt 12 L2 command processor engaged lt CTRL T gt for console mode L2 gt 007 4857 002 Upgrading L1 Firmware Upgrading L1 Firmware The L1 firmware is currently distributed as part of the snxsc_firmware package To determine which version of the package is installed on your system console enter the following command gt rpm q snxsc_firmware If the package is installed the full package name including the revision is returned snxsc_firmware 1 18 3 1 The L1 firmware binary and the utilities used to update it are stored in usr cpu firmware sysco Upgrading L1 Firmware 007 4857 002 Note that an Ethernet connection or console option is required to execute the commands described in this section See Console Hardware Requirements on page 84 and Figure 4 1 on page 80 for descriptions of the hardware connections The L1 firmware consists of three parts e Boot image e A image e
50. Observe all ESD precautions Failure to do so can result in damage to the equipment Wear an SGI approved wrist strap when you handle an ESD sensitive device to eliminate possible ESD damage to equipment Connect the wrist strap cord directly to earth ground 47 2 Operation Procedures Safety Precautions Warning Before operating or servicing any part of this product read the Safety Information on page 165 Danger Keep fingers and conductive tools away from high voltage areas Failure to follow these precautions will result in serious injury or death The high voltage areas of the system are indicated with high voltage warning labels Caution Power off the system only after the system software has been shut down in an orderly manner If you power off the system before you halt the operating system data may be corrupted Warning Ifa lithium battery is installed in your system as a soldered part only qualified SGI service personnel should replace this lithium battery For a battery of another type replace it only with the same type or an equivalent type recommended by the battery manufacturer or an explosion could occur Discard used batteries according to the manufacturer s instructions Oo tk 9 48 007 4857 002 System Controller Network System Controller Network All Altix 450 enclosures contain an embedded microprocessor board and display assembly known as the system controller
51. Operation Once the L2 is running you can telnet to the L2 or use an optional SGIconsole After the connection to the L2 controller is established the following prompt appears indicating that the L2 is ready to accept commands olympic 101 L2 gt Common operations are discussed in the subsections that follow Viewing System Configuration 007 4857 002 You can use the L2 s config command to view the current system configuration from an IRU level olympic 101 L2 gt config L2 127 0 0 1 001 LOCAL ER 127 0 0 1 0 0 O001c31 Dl 2h 0404 T2031 001621 Ld 1257 00 12022 0O01er1 EL 127 0 0 1 0 3 001c01 L2 gt As shown above config produces a list of IRUs and their locations in the system and the system controller address of each IRU This is similar to the output from using the config command on the L1 with the addition of the L2 IP address L1 connection and L1 index The structure of the IRU address is as follows a b c d x y where a b c d is the IP address of the L2 In the example above the IP address is 127 0 0 1 x connection number is only 0 for Altix 450 y is the L1 index the index identifies an IRU by its rack type and slot 001c01 The structure of the IRU location is as follows rrrbss p where rir is the rack number 91 4 System Control b is the enclosure type SS is the slot location of the enclosure p is the partition of the enclosure not present if the system is
52. Powered Down appears on the L1 display when the IRU is powered off The L1 controller is still powered on To completely power down an L1 controller unplug the IRU s power cords from the PDU or two front power supplies 007 4857 002 Operating Your System Powering Off Manually To power off your system manually follow these steps N Caution If you power off the system before you halt the operating system you can lose data 1 Shut down the operating system by entering the following command init 0 2 Press the power buttons or power switches on each of the IRUs that you want to power off You may power off the IRUs in any order e To power off the optional storage units press the power button s on its rear panel to the OFF O position e To power off the IRUs press the power button with the LED on the front panel of each unit see Figure 1 21 on page 41 007 4857 002 43 1 System Installation L1 Front Panel Controls The L1 front panel of the Altix 450 IRU provides the following control features as shown in Figure 1 22 e Status LEDs The front panel has the following LEDs e Power button LED This LED illuminates green when the internal components are on e Service required LED This LED illuminates yellow to indicate that an item is not functioning properly for example a fan is off but the IRU is operating e Failure LED This LED illuminates red to indicate that a failure has occurred
53. RU modules have 9 pin serial interface connectors These ports are for console interface and are capable of transferring data at rates as high as 230 kbps Other features of the ports include the following e Programmable data parity and stop bits e Programmable baud rate and modem control Figure A 2 shows a serial port example Pin 3 Transmit Pin 2 Data TXD Receive Data Pin 4 Pin 1 RXD Data Terminal Data Carrier Ready DTR Detect DCD not used not used Data Set Ready DSR not used Ringing Indicator RI Pin 7 not used Request to ie to Send RT Send RTS CTS Figure A 2 Serial Port Connector 007 4857 002 T O Port Specifications 007 4857 002 Table A 8 shows pinout assignments for the 9 pin male DB 9 connector Table A 8 Serial Port Pinout Pin Assignment Description 1 DCD Data carrier detect 2 RXD Receive data 3 TXD Transmit data 4 DTR Data terminal ready 5 GND Signal ground 6 DSR Data set ready 7 RTS Request to send 8 CTS Clear to send 9 RI Ring indicator 161 A Technical Specifications and Pinouts External SAS SATA Port Connector Figure A 3 shows the connector pin locations for the external single channel Serial Attached SCSI SAS and Serial Advanced Technology Attachment SATA connector This combined technology connector is used to support optional external storage Table A 9 lists the pin assignments for the 14 pin SAS SATA connector
54. SGI Altix 450 System User s Guide Document Number 007 4857 002 CONTRIBUTORS Written by Mark Schwenden Illustrated by Chrystie Danzer Production by Mark Schwenden Engineering contributions by Rich Altmaier Steve Bowen Michael T Brown Dick Brownell David Collins Thomas Crapisi Steve Dean Hartmut Gottwald Paul Wiley Gary Meyer Galen Flunker Paul Kinyon Mark Koneazny William Kellerman Paul Pedersen Gregory Thorson and Gary Spilde COPYRIGHT 2007 SGI All rights reserved provided portions may be copyright in third parties as indicated elsewhere herein No permission is granted to copy distribute or create derivative works from the contents of this electronic documentation in any manner in whole or in part without the prior written permission of Silicon Graphics Inc LIMITED RIGHTS LEGEND The software described in this document is commercial computer software provided with restricted rights except as to included open free source as specified in the FAR 52 227 19 and or the DFAR 227 7202 or successive sections Use beyond license provisions is a violation of worldwide intellectual property laws treaties and conventions This document is provided with limited rights as defined in 52 227 14 The electronic software version of this document was developed at private expense if acquired under an agreement with the USA government or any contractor thereto it is acquired as commercial computer software
55. U Power Supply LEDs Each power supply installed in an IRU has a single bi color green amber status LED The LED will either light green or amber yellow or flash green or yellow to indicate the status of the individual supply See Table 7 3 for a complete list Table 7 3 Power Supply LED States Power supply status Green LED Amber LED No AC power to the supply Off Off Power supply has failed Off On Power supply problem warning Off Blinking AC available to supply standby Blinking Off but IRU is off Power supply on IRU on On Off IRU NUMAlink Router Port LEDs 007 4857 002 Each IRU supports a total of four external NUMAIink connectors located on the front of the unit Each of these connectors has two status LEDs one green and one amber The amber LED illuminates to indicate that both the Altix 450 IRU NUMAlIlink connector and the module to which it is connected are powered on e The green LED illuminates when a link has been established between the Altix 450 NUMAlink connector and the module to which it is connected If both LEDs are dark check the connections at both ends of the NUMAlink cable to ensure they are firmly seated Check the power on status of both units the cable is connected with 147 7 Troubleshooting and Diagnostics Compute Memory Blade LEDs Each compute memory blade installed in an IRU has a total of eight LED indicators arranged in tworows of fourand behind the perforated of
56. Us is also covered Chapter 2 Operation Procedures provides instructions for operation of the system including powering on and powering off your system Chapter 3 System Overview provides environmental and technical information needed to properly set up and configure the Altix 450 series system Chapter 4 System Control describes the function of the L1 and L2 controllers and provides instructions for operating the controllers Chapter 5 Rack Information describes the rack sizes and features Chapter 6 Maintenance and Upgrade Procedures provides instructions for installing or removing the customer replaceable components of your system Chapter 7 Troubleshooting and Diagnostics provides recommended actions if problems occur on your system Appendix A Technical Specifications and Pinouts provides physical environmental and power specifications for your system Also included are the pinouts for the non proprietary connectors Appendix B Safety Information and Regulatory Specifications lists all regulatory information related to use of the Altix 450 system in the United States and other countries It also provides a list of safety instructions to follow when installing operating or servicing the product 007 4857 002 Related Publications Related Publications 007 4857 002 The following SGI documents are relevant to the Altix 450 series system SGI LI and L2 Contro
57. Where to Attach Rails in a Rack The base IRU and any additional IRUs each require five units 5U of vertical space within the rack one unit is equivalent to 1 75 inches 44 5 mm To determine where you should install the unit s shelf rails in the rack you must count mounting holes Each U contains three mounting holes therefore in the 5U of space that the module occupies there are 15 mounting holes The bottom hole of the 5U space is hole 1 The top mounting hole in the 5U space is hole 15 See Figure 1 11 Note An IRU in the rack is identified by the lowest U number that it occupies For example in Figure 1 11 the IRU s bottom is in U1 the first unit within the rack 7 16th hole R gt 5U Figure 1 11 Mounting Hole Pattern of Rack Rails To determine how many mounting holes you must count use the following formula 3 x the lowest U number that the module will occupy 2 For example when you want to install the module in locations U6 through U11 count 16 mounting holes 3 x 6 2 starting from the bottom of the rack The 16th hole is the first mounting hole of U6 007 4857 002 25 1 System Installation Rackmounting IRUs with Shelf Rails This section describes how to rackmount an IRU using shelf rails The shelf rails which are shipped with the module support the module within the rack To attach the shelf rails to the rack follow these steps 1 Locate the slot location in
58. Z line Figure 1 2 32 amp Single phase Power Plug for International Sites 11 1 System Installation Unloading and Moving System Equipment Your Altix 450 system arrives at the site in a pallet mounted cardboard shipping crate For a short rack system the documentation carton and the accessories carton are packed with the system The documentation carton contains the system manuals as well as warranty and licensing information The accessories carton contains the I O peripheral and system cables and any additional connectors or tools that are required for a specific configuration Optional storage or monitors are shipped in a separate carton For a tall rack system the system documentation accessories and other optional hardware arrive in separate cartons If you have purchased optional installation support from SGI service skip the following sections and go on to the next chapter This section describes how to unload and transport the rack system to its designated location as follows e Unloading the Equipment from the Truck on page 12 e Inspecting the Shipping Crate on page 14 e Transporting the Shipping Crate on page 15 Unloading the Equipment from the Truck 12 If your loading dock is the same height as the transportation vehicle use a pallet jack to unload the system from the transportation vehicle The pallet jack should have 48 in tines or forks Follow any instructions that are print
59. ack from toppling over Install all equipment in the lowest available position in the rack Use optional seismic tiedowns where appropriate Always site the unit on a level surface 1 System Installation Installing a Rack This section describes how to install a rack that already has your system IRUs rackmounted and cabled together The following information is provided e Preinstallation Activities on page 6 e Unloading and Moving System Equipment on page 12 e Removing a Short Rack from the Shipping Crate on page 16 e Unpacking and Installing the Tall Rack on page 18 e Positioning and Leveling Racks on page 21 After you have completed installing your rack proceed to Cabling the System IRUs to a Power Source on page 31 to continue your installation If your system components were not shipped in a rack skip this section and proceed to Unpacking and Inspecting Modules on page 22 to begin your installation Preinstallation Activities Site Plan Verification Perform the following preinstallation activities e Perform site verifications see Site Plan Verification on page 6 e Gather appropriate tools to complete the installation see Tools Required on page 7 e Ensure that the correct power receptacle is installed and properly wired see Power Receptacle Verification on page 8 Note You can perform the preinstallation activities days or weeks bef
60. an page style Important system configuration files and commands are documented on man pages These are found online on the internal system disk or on a DVD and are displayed using the man command For example to display the man page for the xscsidisktest command type the following on a command line man xscsidisktest xix About This Guide References in the documentation to these pages include the name of the command and the section number in which the command is found For additional information about displaying man pages using the man command see man 1 In addition the apropos command locates man pages based on keywords For example to display a list of man pages that describe disks type the following on a command line apropos disk For information about setting up and using apropos see apropos 1 XX 007 4857 002 Conventions Conventions The following conventions are used throughout this document Convention Meaning Command This fixed space font denotes literal items such as commands files routines path names signals messages and programming language structures variable The italic typeface denotes variable entries and words or concepts being defined Italic typeface is also used for book titles user input This bold fixed space font denotes literal items that the user enters in interactive sessions Output is shown in nonbold fixed space font Brackets enclose optional portions of a
61. and with the blade designator 001c01 L1 gt b1 power down The command above issues a power down command to the blade in blade slot 1 of the IRU in rack 001 U position 01 Console Mode from L1 007 4857 002 In console mode output from the system boot process or OS is visible and all input is directed to the system To enter console mode press Ct r1 D at the L1 prompt 001c01 L1 gt Ctrl D entering console mode 001c01 console lt CTRL T gt to escape to L1 lt system output appears here gt To return to L1 mode press Ct r1 T Ctr1 T escaping to L1 system controller 001c01 L1 gt While in L1 mode you can enter any L1 command Once the command is executed the L1 returns to console mode re entering console mode 001c01 console lt CTRL T gt to escape to L1 To permanently engage the L1 mode press Ct r1 T and then enter the 11 command Ctr1 T escaping to L1 system controller 001c01 L1 gt 11 L1 command processor engaged lt CTRL D gt for console mode 001c01 L1 gt 87 4 System Control L1 Console Selection 88 If the system contains more than one IRU and a serial connection is utilized for the console the serial cable must be connected to the IRU that is located in the lowest rack and slot position The select command shows the current console mode settings 001c01 L1 gt select console input 001c01 console0d console output not filtered The following are common subchannels associated wi
62. ase module and carefully slide it in until it fully connects together 10 Fasten the fan module to the rear rails of the rack with the screws provided p 10 32 x 1 2 inch screws Figure 1 13 Securing the Module to the Front of the Rack 007 4857 002 27 1 System Installation Removing a Rackmounted IRU Module To remove an IRU that is already mounted in a rack follow these steps 1 Power off the module For instructions on how to power off the module see Powering the System On and Off on page 38 2 Disconnect all of the cables at the front of the module 3 Removing the system blades and power supplies from the IRU is recommended to decrease the overall weight of the unit Be sure to place any system blades in a static free area Warning Components may be hot To avoid injury allow the components to cool for approximately five minutes before you proceed with these instructions 4 Remove the screws that secure the module to the front rails of the rack 5 Remove the screws that secure the IRU at the rear of the rack Caution Always use at least two people when installing or removing a rackmounted Altix 450 IRU 6 Use a minimum of two people to slide the unit out the front of the rack Both lifters need to place one hand securely under the IRU as it slides out to support the weight of the unit as it clears the rack 7 Place the IRU base module on a flat stable surface 8 Remov
63. blade Supports three PCI PCI X 133 MHz 64 bit option cards This three slot blade features card carriers that allow you to slide PCI PCI X boards directly into and out of the unit Single wide PCIe expansion blade The single wide PCI blade supports one or two PCI Express option cards Note that when used with a 3D PCIe graphics card only one can be inserted due to space constraints Double wide PClIe PCI X expansion blade This blade supports two PCI Express option cards and two PCI PCI X option cards Note that when this blade is used with two 3D PCIe graphics cards the PCI X slots cannot be used due to space constraints IA IA2 blade Base I O blade Double wide I O blade that supports all base system I O functions including one or two disk drives a DVD drive two low profile PCI X card slots two ethernet ports one SAS SATAITI port and four USB ports Optional RAID 1 functionality and DVD R W is available with the IA2 version of the base I O blade only Figure 3 5 on page 77 shows the Altix 450 IRU system components 76 007 4857 002 System Components L1 controller panel Power Compute memory blades supplies Router A blade System control Router board board board Figure 3 5 Altix 450 IRU System Components Example Bay Unit Numbering Rack Numbering 007 4857 002 Bays in the racks are numbered using standard units A standard unit SU or unit U is equal to 1 75 inches
64. ce for the IRU reached x xxV ATTN lt power VRM description gt low fault limit 30 second power off sequence for the IRU reached x xxV ATTN lt power VRM description gt high warning A higher than nominal voltage condition is detected limit reached x xxV ATTN lt power VRM description gt low warning A lower than nominal voltage condition is detected limit reached x xxV ATTN lt power VRM description gt level stabilized A monitored voltage level has returned to within x xxV acceptable limits Fan messages ATTN FAN lt fan description gt fault limit reached A fan has reached its maximum RPM level The xx RPM ambient temperature may be too high Check to see if a fan has failed ATTN FAN lt fan description gt warning limit A fan has increased its RPM level Check the reached xx RPM ambient temperature Check to see if the fan stabilizes ATTN FAN lt fan description gt stabilized xx An increased fan RPM level has returned to normal RPM 145 7 Troubleshooting and Diagnostics 146 Table 7 2 L1 Controller Messages continued L1 System Controller Message Message Meaning and Action Needed Temperature messages low alt ATTN lt temp sensor description gt advisory temperature reached xxC xxF ATTN lt temp sensor description gt critical temperature reached xxC xxF ATTN lt temp sensor description gt fault temperature reached xxC xxF Temperature messages hi
65. command or directive line Ellipses indicate that a preceding element can be repeated man page x Man page section identifiers appear in parentheses after man page names GUI element This font denotes the names of graphical user interface GUI elements such as windows screens dialog boxes menus toolbars icons buttons boxes fields and lists Product Support SGI provides a comprehensive product support and maintenance program for its products as follows e Ifyou are in North America contact the Technical Assistance Center at 1 800 800 4SGI or contact your authorized service provider e Ifyou are outside North America contact the SGI subsidiary or authorized distributor in your country 007 4857 002 xxi About This Guide Reader Comments If you have comments about the technical accuracy content or organization of this document contact SGI Be sure to include the title and document number of the manual with your comments Online the document number is located in the front matter of the manual In printed manuals the document number is located at the bottom of each page You can contact SGI in any of the following ways e Send e mail to the following address techpubs sgi com e Contact your customer service representative and ask that an incident be filed in the SGI incident tracking system e Send mail to the following address Technical Publications SGI 1140 East Arques Avenue M S 50 1 946 Sunnyval
66. cs and boot the system 007 4857 002 49 2 Operation Procedures Console Connections A console is defined as a connection to the system that provides access to the system controller network A console can be a personal computer Ethernet connection or a dumb terminal serial connection The Altix 450 series supports multiple types of console connections e An ethernet connection on the system control board of an IRU an RJ45 connection is typically labeled L2 host e A serial connection to the serial console port on the system control board a DB9 connector typically labeled Console e A 2D video card installed in the base I O blade not available with RHEL 5 this low profile PCI video interface card requires use of a USB keyboard and mouse Connecting to the System Controller 50 The ethernet connection is generally the preferred method of accessing the system console Depending on the size of the system there will be one or more ethernet connections utilized When an enclosure is connected to the LAN the system controller will start the ethernet interface with either the pre assigned IP address or will attempt to acquire an IP address via DHCP and will then spawn the L2 application See the section on setting up an IP address for further details Once the system controller is connected to the LAN it can be accessed via a simple telnet session The serial connection is used to communicate directly with the L
67. d be 02 or higher 001c02 L1 gt 001c01 L1 gt brick slot 02 brick slot set to 02 See the SGI LI and L2 Controller Software User s Guide 007 3938 00x for more information on L1 commands 007 4857 002 29 1 System Installation 30 Figure 1 14 NUMAlink Cabling an IRU to an Additional IRU To connect two Altix 450 compute modules to each other with NUMAlIink cables follow these steps 1 Power off both compute modules see Powering the System On and Off on page 38 2 Connect the NUMAIink cables to the NUMAlink connectors on both IRU modules 3 Connect a system console to the master IRU the one with the system disk as described in Connecting the System Console on page 36 4 Power on both IRUs as described in Powering the System On and Off on page 38 007 4857 002 Cabling the System IRUs to a Power Source The console output should indicate that the processors and memory on both IRU modules have been discovered If the secondary IRU module has not been discovered use the reset L1 command to perform a reset of the system Note Use the network L1 command to set the network mode See the SGI L1 and L2 Controller Software User s Guide 007 3938 00x for more information on L1 commands Cabling the System IRUs to a Power Source This section describes how to make the following connections to a power source e Connecting an IRU Module to a Power Source No Rack on page 31
68. dundant and can be hot swapped Fans run at multiple speeds in the IRUs Speed increases automatically when temperature increases or when a single fan fails e System monitoring System controllers monitor the internal power and temperature of the IRUs and can automatically shut down an enclosure to prevent overheating Memory L2 cache L3 cache and all external bus transfers are protected by single bit error correction and double bit error detection SECDED The NUMAlink interconnect network is protected by cyclic redundancy check CRC The L1 primary cache is protected by parity Each IRU and each blade node installed has failure LEDs that indicate the failed part LEDs are readable at the front of the IRU or via the system controllers Systems support the optional Embedded Support Partner ESP a tool that monitors the system when a condition occurs that may cause a failure ESP notifies the appropriate SGI personnel Systems support remote console and maintenance activities 007 4857 002 System Features 007 4857 002 Power on and boot Automatic testing occurs after you power on the system These power on self tests or POSTs are also referred to as power on diagnostics or PODs Processors and memory are automatically de allocated when a self test failure occurs Boot times are minimized Further RAS features Optional RAID 1 in base I O IA2 blade check with your SGI sa
69. e Connecting System Modules to a Rack Power Source on page 34 Connecting an IRU Module to a Power Source No Rack If you are operating your system on a table top connect the IRU s to an approved electric receptacle See Figure 1 15 for the location of the IRU s power connectors 007 4857 002 31 1 System Installation 32 Figure 1 15 Power Supply Connectors on the IRU Module Figure 1 16 shows the connecting of a dual IRU system located on either a tabletop or desktop to a power source 007 4857 002 Cabling the System IRUs to a Power Source Power distribution unit PDU Power source Figure 1 16 Connecting an Unracked IRU System to a Power Source 007 4857 002 33 1 System Installation Connecting System Modules to a Rack Power Source If your Altix 450 system is shipped from SGI in a rack there will be at least one power distribution unit PDU installed in the rack Note that each single phase PDU can support a maximum of two Altix 450 IRUs Plug both power cords from each IRU into the PDU s installed in your short or tall rack Figure 1 17 shows an example of cabling the system modules that reside in a rack to a PDU Power distribution unit PDU Power source Figure 1 17 Connecting Multiple Modules to a PDU An optional three phase PDU used in 42U system racks only has 18 outlet connections of which up to 16 may be needed to power eight IRUs Each thre
70. e California 94085 SGI values your comments and will respond to them promptly xxii 007 4857 002 Chapter 1 007 4857 002 System Installation This chapter describes how to unpack and install your Altix 450 system Specifically the following topics are covered e System Installation Overview on page 2 e Safety Precautions on page 3 e Installing a Rack on page 6 e Unpacking and Inspecting Modules on page 22 e Installing the IRUs in a Rack on page 24 e NUMAlink Cabling the System IRUs on page 29 e Cabling the System IRUs to a Power Source on page 31 If you have arranged to have your system installed and set up by SGI professional services personnel you can skip this chapter and go on to Chapter 2 Operation Procedures 1 System Installation System Installation Overview In this chapter different installation instructions are provided for different types of system installations Follow the instructions for your type of installation as follows Note It is assumed that you will be connecting a system console to your system in each installation case e Ifyou have ordered a server system that is already rackmounted which means that the modules that compose your system are also already cabled together follow the instructions in these sections Safety Precautions on page 3 Installing a Rack on page 6 Cabling the System IRUs to a Power Source
71. e result of tasks running on the L1 such as the environmental monitor Each L1 has a log of local events Use the L1 command 1log to view events on any of the L1s Powering On Powering Off and Resetting the System From the L2 94 You can power on and power off the system with the power command This command is interpreted by the L2 because the IRUs must be powered on in a specific order L2 gt power up L2 gt The power command may require several seconds to several minutes to complete In the example above all racks and slots in the default destination are affected Any errors or warnings are reported as described above in Viewing Information Warnings and Error Messages 007 4857 002 L2 Operation To power on or power off a specific IRU specify a current destination L2 gt r 2 s 5 power up L2 gt To power on or power off all IRUs in a partition enter the following L2 gt partition lt partition number gt lt power up or power down gt To reset the system enter the following L2 gt reset L2 gt This command restarts the system by resetting all registers to their default settings and rebooting the system controllers Resetting a running system will cause the operating system to reboot and all data in memory will be lost Console Mode from the L2 007 4857 002 In console mode all output from the system is visible and all input is directed to the system To enter console mode from L2
72. e 6 27 Removing a Disk Drive 007 4857 002 139 6 Maintenance and Upgrade Procedures Installing a System Disk Drive Before you install a disk drive you must power off the IRU that houses the IA blade base I O where the disk s are installed see the instructions in the Powering the System On and Off on page 51 Note that you should always back up all files before replacement of the disk Be sure to replace any Altix 450 hard disk with an SGI approved drive Caution To protect the system from ESD damage SGI recommends that you use a grounding wrist strap while installing the disk drive To install a disk drive see Figure 6 28 and follow these steps 1 Access the two disk drive bays see panel 1 in Figure 6 28 Note If you are installing only one disk drive install it in the left side housing nearest the DVD drive 2 Position the drive assembly so that it engages the bay guide rails and then gently push the drive into the blade Do not use the locking handle to push the drive into the bay 3 Swing the locking handle towards the chassis until the locking handle engages the latch After you finish installing the disk drive power on the IRU or system according to the instructions in the Powering the System On and Off on page 51 140 007 4857 002 Installing or Replacing a Disk Drive in the IA Blade Figure 6 28 Replacing a Disk Drive 007 4857 002 141 Chapter 7 007 4857 00
73. e fan warning 4 If ESP is used a warning message will be sent to it also see SGI Electronic Support on page 149 The system controller monitors the temperature within each IRU If the temperature increases due to the failed fan the remaining fans will run at a higher RPM to compensate for the failed fan The system will continue running until a scheduled maintenance occurs The fan numbers for the IRU as viewed from the rear are shown in Figure 6 5 Fan 0 Fan 1 Fan 2 Figure 6 5 IRU Fan Assembly Blowers Note that a fan can be replaced while the IRU is operating You will need a number 1 phillips head screw driver to complete the procedure Use the following steps and illustrations to replace an IRU fan 1 Undo the captive screw located in the middle of the blower assembly handle The handle has a notch for the screw access see Figure 6 6 2 Grasp the blower assembly handle and pull the assembly straight out 007 4857 002 115 6 Maintenance and Upgrade Procedures Figure 6 6 Removing a Fan From the IRU 3 Slide a new blower assembly completely into the open slot see Figure 6 7 4 Tighten the blower assembly screw to secure the new fan 116 007 4857 002 Replacing IRU Components Figure 6 7 Replacing an IRU Fan 007 4857 002 117 6 Maintenance and Upgrade Procedures Removing or Replacing a System Blade From the IRU The blades installed in your SGI Altix 450 can generally be re
74. e phase rack system houses a single PDU 34 007 4857 002 Cabling the System IRUs to a Power Source Figure 1 18 shows an example of the optional three phase PDU available in tall racks only Power distribution unit PDU Figure 1 18 Optional Three Phase PDU Example 007 4857 002 35 1 System Installation Connecting the System Console 36 This section explains how to attach and establish a connection between a system console and an Altix 450 system The system console enables you to perform the following activities Start a Linux console session to set up networking and other configuration parameters Monitor your system by reading the status and error message information that the L1 controller generates Enter L1 controller commands to monitor or change particular system functions You can for example monitor the speed of fans for a particular module See the SGI LI and L2 Controller Software User s Guide 007 3938 00x for descriptions of the L1 controller commands that you can use Power on or power off your system To attach a PC to your server system connect a null modem serial cable between the PC and the Console port of the IRU see Figure 1 19 To establish a connection between your system and the system console PC follow these steps 1 Ensure that the L1 controller of the base compute module is powered on When the base compute module is connected to a power source the L1 cont
75. e the IRU s fan module from the rear of the rack and place it nearby 28 007 4857 002 NUMAlink Cabling the System IRUs NUMAIlink Cabling the System IRUs Some configurations of an Altix 450 system will consist of either two or more IRUs This section describes how to cable together these multiple units If your system was shipped with all the cables pre installed go on to the next section Note If your system is made up of a single module you can skip this section and proceed to Cabling the System IRUs to a Power Source on page 31 for instructions to connect your system to power The primary IRU module housing the system disk connects to additional Altix 450 IRUs via the module s front NUMAIink connectors Figure 1 14 shows an example Note Before you connect two Altix 450 IRUs to each other you must assign unique system ID numbers to the compute modules as follows 1 Designate one of the IRU modules as the master server and the other IRU chassis as the slave 2 Connect a console to the master IRU see Connecting the System Console on page 36 and then power on the IRU see Powering the System On and Off on page 38 3 Use the lt slotnumber gt L1 command to set the secondary IRU s ID number to a higher number than the primary IRU s ID number For example if the master IRU unit s L1 prompt indicates that its ID number is 01 001c01 L1 gt then the subsidiary IRU ID number shoul
76. ed on the packing crates If the loading dock is not the same height as the vehicle you must provide a forklift or another approved method to unload the system You can use a platform or ramp to obtain the desired level as long as the ramp incline does not exceed a ratio of one unit vertical to six units horizontal For more information on site requirements contact SGI site planning by e mail site sgi com Warning Use two or more people to prevent computer equipment from rolling off the transportation vehicle Failure to do so could result in serious damage to the computer equipment 007 4857 002 Installing a Rack 007 4857 002 If your site does not have a loading dock arrange for a forklift to unload the system from the transportation vehicle Ensure that two or three people are available to help unload the equipment Move all crates slowly and carefully Figure 1 3 shows the lift openings and dimensions of a tall rack shipping crate This figure also shows where to position the pallet jack Side view End view 81 5 2070 1 mm i a ee J ath 1320 8 mm 1066 8 mm Pallet jack position Figure 1 3 Dimensions of Tall Rack Shipping Crate 13 1 System Installation Figure 1 4 shows the lift openings and dimensions of a short rack shipping crate This figure also shows where to position the pallet jack Side view End view 48 75 1238 mm rar 42 75
77. eplacing a card gently remove the existing card from the carrier 4 Ifyou are adding a card extract the carrier metal filler plate by pushing down on it as shown in Figure 6 16 This filler plate covers the area where your card connectors will protrude out to the rear panel of the blade Figure 6 15 Removing a Card Carrier From the Three Slot Blade 007 4857 002 129 6 Maintenance and Upgrade Procedures Figure 6 16 Extracting the Carrier Metal Filler Plate 130 007 4857 002 Adding or Replacing PCI PCI X Cards Note If you are adding or replacing a half height PCI card skip ahead to step 6 5 If the full height card that you want to install does not fit in the carrier follow these steps a Loosen the three screws on the guide bar with a Phillips screwdriver if your carrier uses T8 Torx screws you will need a Torx screw driver b Adjust the carrier guide bar also known as the alignment rail up or down as necessary see Figure 6 17 Carrier guide bar Figure 6 17 Adjusting the Carrier Guide Bar 007 4857 002 131 6 Maintenance and Upgrade Procedures c Mount the card in the carrier see Figure 6 18 and ensure the following occurs The card connectors protrude through the bulkhead where the metal filler plate was located The edge connectors fit between the bottom guides of the carrier The top of the card fits under the clips on the guide bar d Tighten the t
78. er on or power off a system with multiple IRUs from the L1 prompt you will need to prepend the command with an asterisk as follows oolc01 L1 gt power up or 54 007 4857 002 Powering the System On and Off 007 4857 002 oolc01 L1 gt power down indicates all From the L1 prompt display the system configuration information by entering the following command 001c01 L1 gt config 0 001c01 LOC 2 002r01 L1H 4 002r05 L3H 001c01 L1 gt In L1 mode you can obtain only limited information about the system configuration An IRU has information about its internal blades and also when other IRUs are NUMAlIink attached to the IRU information about those IRUs Note Verify that the power LED on the L1 display turns on and lights green and that your controllers display that the system is powered on for each segment of the procedure which indicates that the power on procedure is proceeding properly If you have a problem while powering on and an error message appears on the L1 controller or the system console see your online log files and the information in L1 Controller Error Messages on page 145 to learn what the error message indicates and how to best resolve the problem 55 2 Operation Procedures Powering On and Off From the L2 Commands issued at the L2 prompt are typically sent to all NUMAIink connected IRU enclosures in the system To power on a system system 101 L2 gt power up T
79. est when a processor accesses local memory Distributed Shared I O Like DSM I O devices are distributed among the blade nodes within the IRUs each base I O blade node has two NUMAlink ports and are accessible by all compute nodes within the SSI through the NUMAlIink interconnect fabric ccNUMA Architecture Cache Coherency 007 4857 002 As the name implies the cache coherent non uniform memory access CCNUMA architecture has two parts cache coherency and nonuniform memory access which are discussed in the sections that follow The Altix 450 systems use caches to reduce memory latency Although data exists in local or remote memory copies of the data can exist in various processor caches throughout the system Cache coherency keeps the cached copies consistent To keep the copies consistent the ccCNUMA architecture uses directory based coherence protocol In directory based coherence protocol each block of memory 128 bytes has an entry in a table that is referred to as a directory Like the blocks of memory that they represent the directories are distributed among the compute memory blade nodes A block of memory is also referred to as a cache line Each directory entry indicates the state of the memory block that it represents For example when the block is not cached it is in an unowned state When only one processor has a copy of the memory block it is in an exclusive state And when more than one processor has a copy of
80. f other software interfaces has not been verified and cannot be supported Important The example process that follows presumes there are two disks identical in size and speed installed in your Altix system IA2 base I O blade Also note that this example of re imaging a disk is done via network access The 1siutil file efi version resides at boot efi Launch the configuration utility with the command fs0 gt lsiutil64 efi You should see an interactive interface similar to the following LSI Logic MPT Configuration Utility Version 1 xx November 7 200X 1 MPT Port found Port Name Chip Vendor Type Rev MPT Rev Firmware Rev IOC 1 01 00 01 0 LSI Logic SAS1068 BO 105 01100000 0 The SAS1068 is the chip that controls the device that the 1siutil utility was loaded from 61 Select a device tI or 0 to quant T Identify firmware BIOS and or FCode Download firmware update the FLASH Download erase BIOS and or FCode update the FLASH 8 Scan for devices 10 Change IOC settings interrupt coalescing 13 Change SAS IO Unit settings 16 Display attached devices 18 Change WWID 20 Diagnostics 21 RAID actions 22 Reset bus 23 Reset target 24 Clear ACA 39 Force firmware download boot 45 Concatenate SAS firmware and NVDATA files 60 Show non default settings 61 Restore default settings 97 Reset SAS phy 98 Reset SAS link 99 Reset port e Enabl xpert mode in menus p Enable paged mode in menus
81. g PCI PCI X Cards 128 Warning Before installing operating or servicing any part of this product read the Safety Information on page 165 This section provides instructions for adding or replacing a PCI PCI X card in the I O blade or PCI expansion blades installed in your system s Individual Rack Units IRUs To maximize the operating efficiency of your cards be sure to read all the introductory matter before beginning the installation There are different types of blades used in the IRU that hold PCI PCI X boards Note the following guidelines e The IA IA2 blade base I O blade containing the system disk and DVD drive will only accommodate half height PCI PCI X cards e The three slot double wide PCI blade supports only PCI PCI X cards that mount and install with an SGI sliding card carrier 007 4857 002 Adding or Replacing PCI PCI X Cards Caution To protect the PCI cards from ESD damage SGI recommends that you use a grounding wrist strap while installing a PCI card Installing PCI Cards in the Three Slot Blade Use the following instructions and illustrations to install or replace a PCI PCI X card in the three slot blade 1 Follow the instructions in Replacing IRU Components on page 110 and extract the three slot blade from the IRU 2 To extract the card carrier from a slot in the blade pull the handle on the selected card carrier straight out as shown in Figure 6 15 3 If you are r
82. gh alt ATTN lt temp sensor description gt advisory temperature reached xxC xxF ATTN lt temp sensor description gt critical temperature reached xxC xxF ATTN lt temp sensor description gt fault temperature reached xxC xxF Temperature stable message ATTN lt temp sensor description gt stabilized xxC xxF Power off messages Auto power down in xx seconds IRU appears to have been powered down The ambient temperature at the IRU s air inlet has exceeded 30 C The ambient temperature at the IRU s air inlet has exceeded 35 C The ambient temperature at the IRU s air inlet has exceeded 40 C The ambient temperature at the IRU s air inlet has exceeded 27 C The ambient temperature at the IRU s air inlet has exceeded 31 C The ambient temperature at the IRU s air inlet has exceeded 35 C The ambient temperature at the IRU s air inlet has returned to an acceptable level The L1 controller has registered a fault and is shutting down The message displays every five seconds until shutdown The L1 controller has registered a fault and has shut down 007 4857 002 LED Status Indicators LED Status Indicators There are a number of LEDs on the front of the IRUs that can help you detect identify and potentially correct functional interruptions in the system The following subsections describe these LEDs and ways to use them to understand potential problem areas IR
83. h 007 4857 002 Use of an Ethernet switch is the preferred method of interconnecting large systems with multiple L2s and remote support hardware to multiple systems An optional Ethernet switch provides multiple additional Ethernet connectors Figure 4 3 shows sample connections between an Ethernet switch a remote workstation an IRU and other SGI systems Ethernet switch yvy To optional To Ethernet connectors To IRU remote workstation on other systems Figure 4 3 Ethernet Switch System Controller Block Diagram Example 83 4 System Control Console Hardware Requirements Operating the L1 84 The console type and how these console types are connected to the Altix 450 servers is determined by what console option is chosen If you have an Altix 450 server with a dumb terminal you can connect the terminal via a serial cable to the DB 9 console port connector on the system control board of the IRU The terminal should be set to the following functional modes e Baud rate of 38 400 e 8 data bits e One stop bit no parity e Hardware flow control RTS CTS Note that the serial console is generally connected to the first bottom IRU in a rack configuration You can also connect to the RJ45 Ethernet connector on the system control board of the IRU and place it on a network This starts an L2 on the IRU s system controller You can use the telnet command to connect By default the L2 uses DHCP to obtai
84. her to specify an individual IRU You can use the aliases r and s to specify rack and slot respectively You can use the alias all or in both the lt rack list gt and the lt slot list gt or by themselves to specify all racks and all slots To send a command to all IRUs in a partition enter the following partition lt partition gt lt cmd gt Individual IRU and Dense routers can also be targeted with a short lt rack gt lt slot gt prefix As in 1 11 lt command gt 92 007 4857 002 L2 Operation 007 4857 002 To target individual blades in an IRU use the following syntax olympic 101 L2 gt 1 11 b1 power down Executing the above command will power down the blade in blade slot 1 of the IRU in rack 001 U position 11 Default Destination When the L2 starts the default destination is set to all racks and all slots You can determine the default destination by using the dest inat ion command L2 gt destination all racks all slots L2 gt The following command sets the destinations to rack 2 and 3 all slots L2 gt r 2 3 destination 2 default destination s set L2 gt The following example shows what IRUs are found in the default destination If you enter a command not understood by the L2 the command is sent to these IRUs Note In the current implementation if you add an IRU to either rack 2 or 3 it would not be automatically included in the default destination You would need to reset the
85. hes Power source Figure 1 20 PDU Circuit Breakers Example 007 4857 002 39 1 System Installation 40 Verify that all of the L1 controllers display L1 running If any L1 controllers are not running contact your SGI system support engineer SSE At the system console access the L2 controller by entering the following command gt stand sysco bin 12term From the L2 prompt display the system configuration by entering the following command L2 gt cfg This command lists the IRUs that the L2 controller detects in the system and their system controller addresses If a IRU that you want to power on does not appear in the list it will not power on Ensure that the IRU s L1 controller is running and that the IRU is cabled properly From the L2 prompt L2 gt power on an individual IRU by entering the following command If you want to power on the entire system proceed to Step 8 L2 gt r lt rack gt s lt slot gt pwr u For example to power on a IRU in rack 1 slot 18 enter the following command L2 gt r 1s 18 pwr u The slot number is the unit number of the IRU within the rack If you want to power on two selected IRUs of a rack at the same time you must enter the rack number followed by the slot numbers of the IRUs that you want to power on For example to power on the IRUs in slots 16 and 21 enter the following command L2 gt r 1 s 16 21 pwr u If you want to power on the enti
86. hree screws on the guide bar Skip step 6 and proceed to step 7 Bulkhead gt i Guides Figure 6 18 Mounting Card in Carrier 132 007 4857 002 Adding or Replacing PCI PCI X Cards 6 To install a half height PCI card in a carrier follow these steps a Remove the two screws from the carrier guide bar as shown in Figure 6 19 Note Take care not to lose the threaded inserts in the carrier guide bar b Loosen the third screw of the carrier guide bar c Place the half height PCI card in the carrier so that the connector edge of the card rests against the lower guide tab of the carrier see Figure 6 19 Loosen T8 Torx screw Remove T8 Torx screws Guide tab Connector edge of PCI card Figure 6 19 Mounting Half height PCI Card into Carrier d Move the carrier guide bar adjustment rail so that it holds the card firmly in place see Figure 6 20 Adjustment rail Figure 6 20 Moving Carrier Guide Bar to Secure Half height PCI Card 007 4857 002 133 6 Maintenance and Upgrade Procedures e Place the PCI card bracket so that the screw holes of the bracket align with the empty screw holes of the carrier guide bar as shown in Figure 6 21 f Secure the bracket to the carrier guide bar with two T8 Torx screws g Tighten the screw that you loosened in step 6b Figure 6 21 Installing the Bracket to Secure a Half height PCI Card 134 007 4857 002
87. ible with PCI PCI X in the following ways e Compatible software layers e Compatible device driver models e Same basic board form factors e PCIe controlled devices appear the same as PCI PCI X devices to most software PCI Express technology is different from PCI PCI X in the following ways e PCI Express uses a point to point serial interface vs a shared parallel bus interface used in older PCI PCI X technology e PCIe hardware connectors are not compatible with PCI PCI X see Figure 6 14 e Potential sustained throughput of x16 PCI Express is approximately four times that of the fastest PCI X throughputs PCI 2 0 32 bit Figure 6 14 Comparison of PCI PCI X Connector with PCI Express Connectors PCI Express technology uses two pairs of wires for each transmit and receive connection 4 wires total These four wires are generally referred to as a lane or x1 connection This is also called by 127 6 Maintenance and Upgrade Procedures 1 SGI Altix 450 technology uses an x16 connector 64 wires or by 16 in PCI Express card slots This technology will support PCIe boards that use connectors up to x16 in size Table 6 2 shows this concept Table 6 2 SGI PCIe Support Levels SGI x16 PCle Connector x1 PCIe cards Supported x2 PCIe cards Supported x4 PCIe cards Supported x8 PCIe cards Supported x16 PCIe cards Supported x32 PCIe cards Not supported Adding or Replacin
88. icks 54 console communications 96 subchannels 96 007 4857 002 console mode from L1 mode 87 from L2 emulator mode 95 console selection 96 controller L1 81 customer service xxi D disk drive installation 140 documentation available via the World Wide Web xix conventions xxi E embedded support partner ESP 152 system support 56 environmental specifications 156 error messages 145 ESP embedded support partner system support 56 F failed fan 114 fan numbers for the IRU 115 173 Index fan RPM level 114 front panel display L1 controller 82 front panel LEDs 44 information warning and error messages viewing in L2 mode 89 94 install disk drive 140 IRU block diagram 71 IRU slot numbering 77 K Knowledgebase 152 L L1 console mode 45 operation mode 45 L1 console selection 88 L1 controller 57 description 81 error messages 145 front panel display 82 front panel display items 83 functions 82 L1 firmware upgrade 99 L1 mode 45 command targeting 86 from L1 software 85 from L2 emulation mode 98 powering on at console 54 174 to console mode 87 viewing information warning and error messages 89 viewing system configuration 86 L1 software modes of operation 84 to LI mode 85 L2 emulation mode to L1 mode 98 viewing system configuration 91 L2 emulator mode powering off 94 powering on 94 resetting system 94 setting com
89. ing IRU blades Removing and Replacing a Compute Memory Blade on page 118 Add or replace DIMMs Memory DIMM Group Placement and Guidelines on page 121 Replace the DVD drive Remove and Replace the DVD Drive on page 123 PCI PCI X PCIe cards Adding or Replacing PCI PCI X Cards on page 128 Base I O blade disk drives Installing or Replacing a Disk Drive in the IA Blade on page 138 109 6 Maintenance and Upgrade Procedures Replacing IRU Components While the individual rack units IRUs are not considered end user replaceable a number of their internal components can be removed and replaced These include L1 controller panel IRU power supplies IRU cooling fans also called blowers IRU blades and their components Remove and Replace the System Control Display Panel To remove and replace either of the two power supplies in an IRU you must first remove the system control display panel Use the following steps to remove the system control display 1 110 Using a T10 torx driver loosen the lower screw that holds the display panel to the IRU chassis see Figure 6 1 on page 111 Lift the display assembly up about 1 3 inch 84 mm and pull it outward from the IRU chassis Don t unplug the ribbon cable unless you are replacing the system control panel Let the control panel gently dangle and go to the section Removing or Adding IRU Power Supplies on page 113 Replace the ap
90. ing On Manually or With L1 Console on page 41 Powering Off at the System Console on page 42 Powering Off Manually on page 43 To prepare to power on your system follow these steps 1 Confirm that all the cable connections between the IRUs and other system items are connected securely If necessary see NUMAlink Cabling the System IRUs on page 29 for information Confirm that all the power cables are plugged in securely to a power source If necessary see Cabling the System IRUs to a Power Source on page 31 for information Note You must manually power on any optional storage modules by placing the power switch es in the ON I position Note that you should turn any optional storage module on before starting the operating system 007 4857 002 Operating Your System Powering On With Optional System Console and L1 L2 Controller To power on and boot your system using the system console follow these steps 1 If not yet done so connect a system console to the server system as described in Connecting the System Console on page 36 2 Ifthe monitors and other peripherals are equipped with voltage select switches verify that they are set for the appropriate AC voltage and plug them in Note that they are normally plugged into sources outside the rack system 3 Turn on the on off circuit breaker switchs of the PDU as shown in Figure 1 20 Power distribution unit PDU On off switc
91. l Spedali Table 5 2 Short Altix Rack Technical Specifications po sy Table 6 1 Customer replaceable Components and Maintenance Procedures Table 6 2 SGI PCIe Support Levels Table 7 1 Troubleshooting Chart Table 7 2 L1 Controller Messages Table 7 3 Power Supply LED States Table A 1 Altix 450 System Configuration eines Table A 2 Altix 450 Tall Rack System Physical E Table A 3 Altix 450 Short Rack System Physical Specifications Table A 4 Environmental Specifications 42U Rack Table A 5 Environmental Specifications 20U Rack Table A 6 Power Specifications Table A 7 Ethernet Pinouts Table A 8 Serial Port Pinout Table A 9 SAS SATA Pin Assignments Table A 10 Pin Assignments for USB Type A Connector 24 104 104 109 128 145 147 153 154 155 156 156 157 159 161 162 163 XV Audience About This Guide This guide provides an overview of the installation architecture general operation and descriptions of the major components that compose the SGI Altix 450 family of systems It also provides the standard procedures for powering on and powering off the system basic troubleshooting and maintenance information and important safety and regulatory specifications This guide is written for owners installers system administrators and users of SGI Altix 450 computer systems It is written with the assumption that the reader has a good working knowledge of computers and computer systems Importan
92. les or service representative for availability Note that RAID 0 striping while supported by the IA2 hardware is not recommended as a RAS option Systems have a local field replaceable unit FRU analyzer All system faults are logged in files Memory can be scrubbed using error checking code ECC when a single bit error occurs 75 3 System Overview System Components The Altix 450 series system features the following major components 20U rack The short rack is a custom rack used with the Altix 450 system It holds up to 4 IRUs or a combination of IRUs and option modules such as mass storage 42U rack The tall rack is a custom rack used for both the compute and I O rack in the Altix 450 system Up to 8 IRUs can be installed in each rack There is also a 2U space reserved at the top for an option module Individual Rack Unit IRU This enclosure contains the compute memory blades IA blade base I O standard routers and optional I O blades for the Altix 450 series systems The enclosure is 5U high Compute Memory blade Holds up to two IA 64 processor sockets and 4 8 or 12 memory DIMMs Memory only blade This blade acts as a memory expansion node with no processor compute circuitry included on the blade This blade holds 4 8 or 12 memory DIMMs Single wide PCI X expansion blade This two slot PCI PCI X option blade supplies an individual PCI bus for each option card Double wide PCI X expansion
93. ller Software User s Guide P N 007 3938 00x This guide describes how to use the L1 and L2 controller commands at your system console to monitor and manage your SGI system SGI InfiniteStorage 120 Mass Storage Hardware Topics P N 007 4853 00x This document provides a hardware overview of the SGI non RAID 12 drive optional mass storage enclosure used with Altix 450 systems Reconfigurable Application Specific Computing User s Guide P N 007 47 18 00x This guide discusses the optional SGI reconfigurable application specific software computing RASC program that delivers scalable configurable computing elements for the SGI Altix family of servers and superclusters Man pages online Man pages locate and print the titled entries from the online reference manuals You can obtain SGI documentation release notes or man pages in the following ways See the SGI Technical Publications Library at http docs sgi com Various formats are available This library contains the most recent and most comprehensive set of online books release notes man pages and other information The release notes which contain the latest information about software and documentation in this release are in a file named README SGI in the root directory of the SGI ProPack for Linux Documentation CD You can also view man pages by typing man lt title gt on a command line SGI systems include a set of Linux man pages formatted in the standard UNIX m
94. longer than the other two pins on the North American plug Optional three phase power distribution is available in systems mounted in 42U tall racks For North American sites follow these steps to ensure that a single phase power receptacle is properly wired 1 Set the voltmeter to a high AC voltage range 2 Check the voltage between socket X and socket Y see Figure 1 1 The meter should read between 200 and 240 VAC 3 Check the voltage between socket X and the ground socket The meter should read approximately 120 VAC 4 Check the voltage between socket Y and the ground socket The meter should read approximately 120 VAC 5 Check the voltage between the ground socket and an earth ground location The meter should read 0 VAC 6 Change the voltmeter to a low resistance setting 7 Measure between the ground socket and an appropriate earth ground location and ensure that resistance is less than 1 ohm 8 Repeat steps 1 through 7 for any additional single phase power receptacles Caution If a voltage reading is incorrect or if the resistance measured in step 7 is more than 1 ohm contact a site approved electrician Do not proceed with the installation 8 007 4857 002 Installing a Rack lt Power cord connector lt Ground pole EN Ground Socket Y m socket Socket X A lt Receptacle Figure 1 1 30 amp Single phase Power Recep
95. m to service sooner if it is out of service In addition to the event monitoring and problem reporting SGI Electronic Support monitors both system configuration to help with asset management and system availability and performance to help with capacity planning 151 The following three components compose the integrated SGI Electronic Support system SGI Embedded Support Partner ESP is a set of tools and utilities that are embedded in the SGI Linux ProPack release ESP can monitor a single system or group of systems for system events software and hardware failures availability performance and configuration changes and then perform actions based on those events ESP can detect system conditions that indicate potential problems and then alert appropriate personnel by pager console messages or e mail plain text or encrypted You also can configure ESP to notify an SGI call center about problems ESP then sends e mail to SGI with information about the event SGI Knowledgebase is a database of solutions to problems and answers to questions that can be searched by sophisticated knowledge management tools You can log on to SGI Knowledgebase at any time to describe a problem or ask a question Knowledgebase searches thousands of possible causes problem descriptions fixes and how to instructions for the solutions that best match your description or question Supportfolio Online is a customer support resource that includes the latest i
96. mand targeting 92 to console mode 95 viewing information warning and error messages 94 laser compliance statements 171 LEDs front panel 44 leveling bolts of a rack 21 lithium battery warning statements 48 172 Isiutil configuration 61 major components 76 modes of operation L1 software 84 monitoring embedded support partner 56 IRUs with L1 controller 57 server 57 N non uniform memory access NUMA 74 numbering IRUs in a rack 77 007 4857 002 Index racks 77 P PCI and PCI X cards installation procedure PCI and PCI X card installation 128 PCIe slots 60 physical specifications Altix Tall Rack Physical Specifications 154 pinouts Ethernet connector 159 serial connector 160 power receptacle 8 11 power specifications 157 powering off L2 mode 94 powering on at system console 54 in L1 mode 54 L2 emulator mode 94 preparation 52 product support xxi R RAID 1 disk mirror 61 RAS features 74 replace a fan 114 Replacing 110 resetting system L2 mode 94 007 4857 002 S SAS SATA external connector pin assignments 162 server monitored at L1 controller 57 monitoring locations 57 SGI Knowledgebase See Knowledgebase Short rack shipping crate removal 17 site requirements 6 space requirements 24 subchannels associated with console communications 96 support embedded support partner ESP 56 Supportfolio Online 152 system architecture 70 sys
97. moved from or installed in an IRU without powering down the entire system However the operating system should be halted and the power to the IRU should be shut down in most cases Note that the IA blade holding your system disk s should never be removed without backing up and gracefully powering down the unit Removing and Replacing a Compute Memory Blade Use the following information and illustrations to remove and replace a compute memory blade from the host IRU 1 Open the front door of the rack and access the IRU blade slated for replacement 2 Move the blade s retention tab gently to the left to disengage the locking tab Be careful not to use too much force as the tab is breakable 3 Pull the blade s retention tab outward until it is fully extended see Figure 6 8 4 Slide the blade outward by pulling the bottom ring gently out 5 Remove the blade completely from the IRU and place it on an anti static work surface or in anti static packaging 6 Slide the new blade carefully into the IRU slot until seated in the slotting mechanism 7 Carefully push the retention tab back into place so that the blade is seated and locked down see Figure 6 9 for an example Note that pushing in the tab fully seats the blade moves it deeper into the IRU and locks it in place 8 Restart the IRU or system 118 007 4857 002 Replacing IRU Components Removing a Compute Memory Blade From the IRU
98. n an IP address To set a static IP see Configuring an L2 s IP Address on page 89 These console connections enable you to view the status and error messages generated by the system controllers on your system You can also use the console to input commands to manage and monitor your system s For more information on the L2 see L2 Operation on page 89 For more details on connecting a console to an Altix 450 series server see System Controller Network on page 49 For more information on monitoring your server see Monitoring Your Server on page 57 Each IRU in the Altix 450 system has an updated and enhanced system control implementation This updated system controller provides both L1 and L2 functionality The system controller utilizes an embedded version of the Linux operating system L1 functionality is provided by an 007 4857 002 Operating the L1 L1 Mode 007 4857 002 application that is always running on the system controller When the enclosure is connected to a LAN via the L2 host connector the system controller spawns an application that provides L2 functionality The L1 operates in one of these two modes which are discussed in the sections that follow L1 Mode The L1 prompt is visible and all input is directed to the L1 command processor The Altix 450 server L1 system control can perform the following e Managing power and sequencing control e Environmental monitoring and control functions
99. nformation about patch sets bug reports and software releases The complete SGI Electronic Support services are available to customers who have a valid SGI Warranty FullCare FullExpress or Mission Critical support contract To purchase a support contract that allows you to use the complete SGI Electronic Support services contact your SGI sales representative For more information about the various support contracts see the following Web page http www sgi com support customerservice html For more information about SGI Electronic Support see the following Web page http www sgi com support es Appendix A Technical Specifications and Pinouts This appendix contains technical specification information about your system as follows System level Specifications on page 153 Physical Specifications on page 154 Environmental Specifications on page 156 Power Specifications on page 157 I O Port Specifications on page 158 System level Specifications 007 4857 002 Table A 1 summarizes the Altix 450 system configuration ranges Note that while each compute memory board can house either one or two processor sockets each socket holds two processor cores Table A 1 Altix 450 System Configuration Ranges Category Minimum Maximum Processor cores 4 76 Individual Rack Units IRUs 1 per rack 4 per system image Compute memory blade DIMM 4 DIMMs per blade 12 DIMMs per blade capacity S
100. ng materials 2 Remove the cover bag and bungee cushion from the rack 3 Locate and remove the 8 bolts that secure the wooden frame pieces to the base 4 The rack s front and rear doors must be removed before you can remove the rack s shipping runners 5 Remove and discard the foam plugs from either end of the rack s base At this time you will need to place a pallet jack under the rack and lift it until the shipping runners are clear of the floor 6 Undo and remove the two bolts on the shipping runners at either end of the rack and set the runners aside 7 Use the pallet jack to lower the rack to the floor and use at least two people to roll the rack to the installation site Warning Personal injury and system damage could result if the rack becomes unbalanced or gains too much momentum when rolled Reattach the doors to the front and rear before you move the rack to its designated location see Figure 1 7 on page 20 007 4857 002 Installing a Rack Figure 1 6 Removing a Tall Rack from the Shipping Crate 007 4857 002 19 1 System Installation 20 Figure 1 7 Reattaching the Tall Rack Doors 007 4857 002 Installing a Rack Positioning and Leveling Racks 007 4857 002 Z Caution To avoid ESD damage to the electronic components be sure to position the racks before you remove the ESD bags that cover the rack assemblies To position and level your system racks
101. non operating no condensation 59 71 kBTU hr maximum based on 17 5 kW Ambient air cooling Approximately 1400 CFM normal operation 10 000 ft 3 049 m operating 40 000 ft 12 195 m non operating Less than 65 dBa maximum with doors Table A 5 Environmental Specifications 20U Rack Feature Specification Temperature tolerance operating Temperature tolerance non operating Relative humidity 5 C 41 F to 35 C 95 F up to 1500 m 5000 ft 5 C 41 F to 30 C 86 F 1500 m to 3000 m 5000 ft to 10 000 ft 40 C 40 F to 60 C 140 F 10 to 90 operating no condensation 8 to 95 non operating no condensation 007 4857 002 Power Specifications Table A 5 Environmental Specifications 20U Rack continued Feature Specification Heat dissipation Altix 450 29 86 kBTU hr maximum based on 8 75 kW Cooling requirement Ambient air cooling only Air flow intake front Approximately 750 CFM normal operation exhaust rear Maximum altitude 10 000 ft 3 049 m operating 40 000 ft 12 195 m non operating Acoustical noise level Less than 65 dBa maximum with doors Power Specifications Table A 6 shows the power specifications for the system Table A 6 Power Specifications Feature Specification Single phase power requirements Voltage 200 240V 180 254 VAC min max Frequency 50 60 Hz Power 17 5 kW max per tall rack
102. not partitioned In the example shown above 001c01 is an IRU in rack 001 and slot position 01 Setting Command Targeting If a command is not understood by the L2 system controller in general it is passed to the L1 system controllers The destination determines which L1s receive the command A destination specified by the following is a range of racks and slots rack lt rack list gt slot lt slot list gt The lt rack list gt specifies a list of racks This can be a list delimited by commas such that 2 4 7 specifies racks 2 4 and 7 You can use a dash to specify a range of racks such that 2 4 specifies racks 2 3 and 4 Both nomenclatures can be combined such that 2 4 7 specifies racks 2 3 4 and T You can specify the lt slot list gt using the same nomenclature The slot number sometimes referred to as a bay number is the unit position number located on the rack slightly above where the bottom of the IRU sits Each rack unit position number is located toward the top of the two lines that mark the unit position that the number represents For example the rack numbering for an IRU located in slot 10 would appear on the left front side of the rack The slot lt slot list gt is optional if not given then all slots in the specified rack s are implied You should avoid specifying a rack list and a slot list that includes multiple racks and slots such as rack 2 4 7 slot 1 8 11 13 Generally you specify a rack and slot toget
103. ns Warning Advarsel Lithiumbatteri Eksplosionsfare ved fejlagtig handtering Udskiftning ma kun ske med batteri af samme fabrikat og type L ver det brugte batteri tilbage til leverandgren Warning Advarsel Eksplosjonsfare ved feilaktig skifte av batteri Benytt samme batteritype eller en tilsvarende type anbefalt av apparatfabrikanten Brukte batterier kasseres i henhold til fabrikantens instruksjoner Warning Varning Explosionsfara vid felaktigt batteribyte Anvand samma batterityp eller en ekvivalent typ som rekommenderas av apparattillverkaren Kassera anvant batteri enligt fabrikantens instruktion Warning Varoitus Paristo voi r j ht jos se on virheellisesti asennettu Vaihda paristo ainoastaan laitevalmistajan suosittelemaan tyyppiin H vit k ytetty paristo valmistajan ohjeiden mukaisesti Warning Vorsicht Explosionsgefahr bei unsachgem en Austausch der Batterie Ersatz nur durch denselben oder einen vom Hersteller empfohlenem hnlichen Typ Entsorgung gebrauchter Batterien nach Angaben des Herstellers 007 4857 002 Index A Altix server illustration 69 Altix servers monitoring locations 57 battery statements 172 block diagram IRU 71 Cc cache coherency 73 circuit breaker switches 39 CMN number 167 command targeting L1 mode 86 L2 mode 92 connector pin assignment SAS SATA 162 USB type A 163 console L1 mode power on 54 powering on br
104. nsole connected to a server with a serial interface you can toggle between L1 and console mode This enables you to power on your server with L1 commands and view the activity by changing to the console mode For detailed instructions on using a system console using the L1 mode see Operating the L1 on page 84 e Ifyou have a system console connected to a server with an ethernet interface you can toggle between L2 and console mode and power on your server with L2 controller commands and monitor the power on activity by changing to the console mode Powering On and Off from the L1 Commands issued at the L1 prompt typically only affect the local IRU The following sections describe how to power on and power off your system in L1 mode The L1 controller display located on the front of each IRU should display L1 running once the power on procedure starts storage modules do not have L1s The prompt on your console screen shows the rack and slot number of the IRU to which you have connected your console Note If you have a problem while powering on and an error message appears on your console display see L1 Controller Error Messages on page 145 to learn what the error message indicates and how to resolve the problem If you want to power on the IRU 001c01 in our example indicated in the prompt enter the following command 001c01 L1 gt power up To power off 001c01 L1 gt power down If you are attempting to pow
105. o power off a system system 101 L2 gt power down Using Embedded Support Partner ESP 56 Embedded Support Partner ESP automatically detects system conditions that indicate potential future problems and then notifies the appropriate personnel This enables you and SGI system support engineers SSEs to proactively support systems and resolve issues before they develop into actual failures ESP enables users to monitor one or more systems at a site from a local or remote connection ESP can perform the following functions e Monitor the system configuration events performance and availability e Notify SSEs when specific events occur Generate reports ESP also supports the following e Remote support and on site troubleshooting e System group management which enables you to manage an entire group of systems from a single system For additional information on this and other available monitoring services see the section SGI Electronic Support in Chapter 7 007 4857 002 Monitoring Your Server Monitoring Your Server You can monitor your Altix 450 server from the following sources e On the L1 controller s display at the front of the IRU as shown in Figure 2 3 you can monitor system operational status For example you can monitor error messages that warn of power or temperature values that are out of tolerance e You can connect and monitor the system via an Ethernet port adapter You will need to
106. on page 31 Connecting the System Console on page 36 e Ifyou have ordered un racked IRU modules with the intention of rackmounting and cabling your own server system follow the instructions in these sections Safety Precautions on page 3 Installing a Rack on page 6 Installing the IRUs in a Rack on page 24 NUMAlIink Cabling the System IRUs on page 29 Cabling the System IRUs to a Power Source on page 31 Connecting the System Console on page 36 007 4857 002 Safety Precautions Safety Precautions 007 4857 002 If you have ordered either a single or dual IRU system that you intend to install on a table top follow the instructions in these sections Safety Precautions on page 3 Unpacking and Inspecting Modules on page 22 Installing the IRUs in a Rack on page 24 Cabling the System IRUs to a Power Source on page 31 Connecting the System Console on page 36 If you have your own rack and have ordered system modules with the intention of rackmounting and cabling your own system follow the instructions in these sections Safety Precautions on page 3 Unpacking and Inspecting Modules on page 22 Installing the IRUs in a Rack on page 24 NUMAlink Cabling the System IRUs on page 29 Cabling the System IRUs to a Power Source on page 31 Connecting the System Console on page 36
107. ong with additional IRUs You can add different types of blade options to a system IRU to achieve the desired system configuration You can easily configure systems around processing capability I O capability memory size or storage capacity You place individual blades that create the basic functionality compute memory I O and power into IRUs The air cooled IRU enclosure system has redundant hot swap fans and redundant hot swap power supplies at the IRU level Distributed Shared Memory DSM In the Altix 450 system memory is physically distributed both within and among the IRU enclosures compute memory I O blades however it is accessible to and shared by all NUMaAlinked devices within the single system image This is to say that all NUMAlinked components sharing a single Linux operating system operate and share the memory fabric of the system Note the following sub types of memory within a system e Ifa processor accesses memory that is connected to the same SHub ASIC on a compute node blade the memory is referred to as the node s local memory e If processors access memory located in other blade nodes within the IRU or other NUMAlinked IRUs the memory is referred to as remote memory e The total memory within the NUMAlinked system is referred to as global memory 72 007 4857 002 System Features Memory latency is the amount of time required for a processor to retrieve data from memory Memory latency is low
108. ons Not all blades may be available with your system configuration Check with your SGI sales or service representative for availability See Chapter 6 Maintenance and Upgrade Procedures for detailed instructions on installing or removing PCI cards and blades 007 4857 002 Installing Optional Components Adding or Removing Disk Drives The IA TA2 blade base I O blade within the IRU supports the system boot functions and contains one or two low profile disk drives See Installing or Replacing a Disk Drive in the IA Blade on page 138 for detailed instructions on installing or removing disk drives For information on removing or replacing disk drives in optional mass storage enclosures see the hardware document supplied with the storage option Reconfigure the System Disk Drive s for RAID 1 007 4857 002 System disk drive s ordered starting in March 2007 may come configured as either a non RAID jbod or as a RAID 1 disk mirror system Note that while RAID 0 striping is supported by the hardware SGI recommends using RAID 1 to ensure maximum reliability and data retention The type of disk configuration is ordered by the customer and configured at the SGI factory If a requirement exits to reconfigure the system disk s use the following information to make the changes needed The 1siutil configuration software is the only supported methodology for modifying the disks at the time this document was published Use o
109. ontrols Operating the L1 Controller vi 007 4857 002 Contents 007 4857 002 Operation Procedures Precautions ESD Precaution Safety Precautions System Controller Network Console Connections Connecting to the System Controller Communicating with the System Controller Powering the System On and Off Preparing to Power On Powering On and Off at the Console Powering On and Off from the L1 Powering On and Off From the L2 Using Embedded Support Partner ESP Monitoring Your Server 2D Graphics Video Interface Installing Optional Components T O Blade Overview i Adding or Removing Disk Drives Reconfigure the System Disk Drive s for RAID 1 System Overview System Models System Architecture System Features Modularity and Scalability Distributed Shared Memory DSM Distributed Shared I O ccNUMA Architecture Cache Coherency ee Non uniform Memory Access NUMA Reliability Availability and Serviceability RAS 47 47 47 48 49 50 50 51 51 252 54 54 56 56 57 58 59 60 6l 6l 67 68 70 72 72 72 73 3 73 74 74 vii Contents System Components Bay Unit Numbering Rack Numbering Optional System Components 4 System Control Levels of System Control System Controller Interaction L1 Controller L1 Controller Functions L1 Front Panel Display Ethernet Switch Console Hardware Requirements O
110. ore you receive your system Ensure that all site requirements are met before your system arrives If you have questions about the site requirements or you would like to order full size floor templates for your site contact a site planning representative by e mail site sgi com 007 4857 002 Installing a Rack Tools Required 007 4857 002 Table 1 1 lists the tools that you need to complete the installation Table 1 1 Installation Tools Tool Part Number Purpose 13 mm wrench 13 mm socket 3 8 in drive 19 mm socket 3 8 in drive Extension 6 in 3 8 in drive Ratchet reversible 3 8 in drive Level 9 in 7260744 7260726 9470618 7260655 7260755 9470556 Adjust the leveling pads Remove bracket bolts from tall rack shipping crate Remove bolts from short rack shipping crate Used with ratchet and sockets Used with extension and sockets Level the rack 1 System Installation Power Receptacle Verification Ensure that a qualified technician installs the correct power receptacles Your Altix 450 system uses one or two single phase power receptacles For North American sites the single phase receptacle is a 30 amp 200 to 240 volt receptacle that has two phase sockets and one ground socket For international sites the single phase receptacle is a 32 amp 200 volt receptacle that has one phase socket one neutral socket and one ground socket Note that the ground pin is slightly
111. ovides environmental control and monitoring initiates system resets stores identification and configuration information and provides console diagnostic and scan interface Each IRU enclosure has a system controller that can communicate with other IRU system controllers when they are NUMAlinked together under a single system image Each system controller constantly shares its information with all controllers in the system Note that optional mass storage enclosures do not have a system controller Figure 4 1 shows an example system control network using a separate remote workstation to monitor a single rack Altix 450 system 007 4857 002 79 4 System Control Remote workstation monitor Altix 450 gees ocal Area Network LAN www Cai 5 Ethernet Figure 4 1 SGI Altix 450 System Control Network Example Levels of System Control The system control network configuration of your server will depend on the size of the system and control options selected Typically an Ethernet connection to the system controller network is used This Ethernet connection is usually made from the IRU housing the system disk s 80 007 4857 002 System Controller Interaction The system controller is designed into all IRUs An Ethernet connection directly from the IRU to a local private or public Ethernet allows the system to be administered directly from a local or remote console Note that there is no interconnected system con
112. perating the L1 LI Mode Viewing System Configuration from an IRU s Perspective Command Targeting Console Mode from L1 L1 Console Selection Be bw ok che be Viewing Information Warnings and Error Messages L2 Operation Configuring an L2 s IP Address Viewing System Configuration Setting Command Targeting Viewing Information Warnings and Error Messages Powering On Powering Off and Resetting the System From the L2 Console Mode from the L2 Console Selection L1 Mode From L2 Upgrading L1 Firmware Upgrading L1 Firmware 5 Rack Information Overview viii 76 77 77 78 79 80 81 81 82 82 83 84 84 85 86 86 87 88 89 89 89 91 92 94 94 95 96 98 99 99 101 101 007 4857 002 Contents 007 4857 002 Altix 42U and 20U Rack Features Technical Specifications Maintenance and Upgrade Procedures Maintenance Precautions and Procedures Preparing the System for Maintenance or Upgrade Returning the System to Operation Installing or Removing Internal Parts Replacing IRU Components ASO ptt ES ek Remove and Replace the System Control Display Panel Removing or Adding IRU Power Supplies Removing and Replacing IRU Fans Blowers Removing or Replacing a System Blade From the IRU Removing and Replacing a Compute Memory Blade Memory DIMM Group Placement and Guidelines Remove and Replace the DVD Drive Overview
113. plicable power supply and then use the following set of steps to replace the system control panel 007 4857 002 Replacing IRU Components Figure 6 1 Removing the L1 Controller Panel Replace the system control panel using the following steps 1 Position the controller panel assembly slightly above the retention screw 2 Carefully slid it back into place see Figure 6 2 007 4857 002 111 6 Maintenance and Upgrade Procedures 3 Be careful not to snag the ribbon cable while replacing the panel 4 Use the T10 torx driver to tighten the retention screw holding the panel to the IRU 5 Restart the IRU or system as applicable Figure 6 2 Replacing the L1 Controller Panel 112 007 4857 002 Replacing IRU Components Removing or Adding IRU Power Supplies Under most circumstances a single power supply in an IRU can be replaced without shutting down the IRU or the complete system In the case of a fully configured loaded IRU this may not be possible In a fully loaded system power supply failure could result in shutdown of the IRU due to the power loss Note that when replacing either of the supplies it is necessary to remove the system controller panel see Remove and Replace the System Control Display Panel on page 110 before proceeding Use the following steps to remove and replace a power supply in the IRU 1 Disconnect the power cord
114. ration on page 89 for further details You can view an IRUs configuration information with the config command as in the following 001c01 L1 gt config 0 001c01 LOC 001c01 L1 gt This example is a system with one IRU The lt number gt that follows the colon 0 1 and 2 from top to bottom in this example refers to which local port the IRU is connected to or accessed through The local LOC IRU is the IRU that is processing the command On all IRUs 0 is the local IRU with other values referring to various ports The specific port description follows the IRU s rack type slot field i e LOC U F U G etc 021c01 L1 gt config 0 021c01 LOC 2 021c11 L1H 021c01 L1 gt All commands entered affect only the local IRU You can target a command to all IRUs including the local IRU by prefixing the command with an asterisk 1c01 L1 gt version 1c01 0 7 37 Image A Built 08 24 2006 14 59 42 2MB image Le 1 0 7 37 Image A Built 08 24 2006 14 59 42 2MB image Olc21 L1 0 7 37 Image A Built 08 24 2006 14 59 42 2MB image 01c01 L1 gt 0 0 One OO Or O E ie 007 4857 002 Operating the L1 Commands can be targeted to other L1s by preceding the command with a rack and slot 001c01 L1 gt 1 11 version The command above issues a version command to the IRU in rack 001 U position 11 Some commands can be targeted to a specific blade within an IRU Precede the comm
115. rd access door Using a Tx20 Torx driver remove the top PCI card access door screw Remove the PCI card access door Add replace remove a PCI card s following standard ESD rules Make sure any newly installed PCI card s are firmly seated in the slot and secure Figure 6 24 Removing the Card Access Panel From the IA Blade 136 007 4857 002 Adding or Replacing PCI PCI X Cards Figure 6 25 Replacing the IA Blade PCI Access Door 8 10 11 12 13 14 007 4857 002 Replace the PCI card access door by installing it into the slotted opening and sliding it forward as shown in Figure 6 25 Reinstall the top PCI card access door screw Reinstall the front PCI card access door screw Reinstall the IA blade in the IRU chassis Return the system to service Boot your operating system software See your software operation guide if you need instructions to boot your operating system Run the 1s pci PCI hardware inventory command to verify the installation This command lists PCI hardware that the operating system discovered during the boot operation 137 6 Maintenance and Upgrade Procedures Installing or Replacing a Disk Drive in the IA Blade 138 The IA IA2 blade base I O in an IRU has two system disk drive bays as shown in Figure 6 26 This section describes how to install or remove the drives q N 0 0 0 i i lt PCI card slot 0 C Ethernet port 0 Ethe
116. re system enter the following command L2 gt pwr u The default setting for the pwr u command is all racks and slots When the L2 prompt appears you will not see the output that is produced during the power on procedure unless you redirect the keyboard input from the L2 controller to the normal console by typing control d L2 gt ctrl d 007 4857 002 Operating Your System Powering On Manually or With L1 Console 007 4857 002 If your system does not have a a local system console you can manually power on and off the server by using the power button s To power on your system or an individual IRU manually follow these steps 1 Ifthe monitors and other peripherals are equipped with voltage select switches verify that they are set for the appropriate AC voltage and plug them in Note that they are normally plugged into power sources outside the rack system 2 Turn the circuit breaker switchs of the PDU to the on position see Figure 1 20 on page 39 3 Press the power buttons see Figure 1 21 on each of the IRUs that you want to power on Note that any optional storage modules should be powered up first L1 controller gt 9 lt Power button display Figure 1 21 Location of the Power Button To boot the system from the L1 interface you must have a console connected to the system via the console port Use the following commands depending on system type 1 For a system that contains one IRU enter
117. representative for the most current information on this topic In a DSM system each processor board contains memory that it shares with the other processors in the system Because the DSM system is modular it combines the advantages of low entry level cost with global scalability in processors memory and I O You can install and operate the Altix 450 series system in your lab or server room Each 20U SGI rack holds from one to four 5U high enclosures that support up to five compute memory and I O sub modules known as blades These blades are single printed circuit boards PCBs with ASICS processors memory components and I O chipsets mounted on a mechanical carrier The blades slide directly into and out of the Altix 450 IRU enclosures This chapter consists of the following sections e System Models on page 68 e System Architecture on page 70 e System Features on page 72 e System Components on page 76 Figure 3 1 shows the front view of a 20U rack used to house the Altix 450 system 67 3 System Overview System Models 68 Figure 3 1 SGI Altix 450 System Short Rack The basic enclosure within the Altix 450 system is the 5U high 8 68 inch or 22 cm individual rack unit IRU The IRU enclosure houses a maximum of four single wide blades and one double wide blade Each IRU comes with two built in high speed routers The routers connect to the installed blades via a backplane Each
118. rimary is PhysDisk 1 Secondary is PhysDisk 0 is active is Bus 0 Target 0 Type IM State optimal enabled Settings draws from Hot Spare Pools Size 139898 MB write caching disabled Stripe Size 0 KB Bus 0 Target 56 Bus 0 Target 1 2 physical disks are active Integrated Mirroring auto configure 0 2 Members For information on the two RAID 1 disks use the 1sscsi command as shown in the following example lsscsi PRP HO O oro disk SGI ST3146854SS x422 dev sg2 disk SGI ST3146854SS X422 dev sg3 disk LSILOGIC Logical Volume 3000 dev sdc dev sg4 This is an example of what you ll see on SLES10 or RHELS Linux operating systems Disk 1 0 0 0 and 1 0 1 0 are the individual drives in the RAID 1 but you cannot access them through the disk driver You access 1 1 0 0 which is dev sdc in this case Caution must be exercised when entering Linux SCSI generic sg commands via the sg devices 66 007 4857 002 Chapter 3 007 4857 002 System Overview This chapter provides an overview of the physical and architectural aspects of your SGI Altix 450 system The major components of the Altix 450 series systems are described and illustrated The Altix 450 series is a family of multiprocessor distributed shared memory DSM computer systems that initially scale from 2 to 76 Intel 64 bit processor cores as a cache coherent single system image SSI Contact your SGI sales or service
119. rnet port 1 lt Pwr fault ink LEDs lt USB port 2 2 lt USB port 1 2 lt USB port 2 1 lt USB port 1 1 H lt SAS LEDs lt SAS SATA II ports DVD drive Drive 1 Drive 0 Figure 6 26 Location of Disk Drive Bays on IA IA2 Blade 007 4857 002 Installing or Replacing a Disk Drive in the IA Blade Removing a System Disk Drive Before you remove a disk drive you must power off the IRU or system according to the instructions in the Powering the System On and Off on page 51 To remove a disk drive follow these steps 1 Open the front of the rack and access the two disk drive bays as shown in panel 1 of Figure 6 27 2 Remove the disk drive by depressing its handle lock with your thumb and pulling the handle away from the chassis until the handle disengages the disk drive connector from the blade s backplane connector 3 Carefully slide the disk drive out of the blade and place it on an ESD safe surface Do not use the handle to pull the disk drive out of the bay After you remove the disk drive go on to the next section Installing a System Disk Drive and install the replacement drive Power on the IRU according to the instructions in the Powering the System On and Off on page 51 Latches rrrri SPSSSSESSOSESEESOSS rz R ei R R R R R R R R R AAAI AAA IIA ES sescceccccesccees BE Cre Figur
120. roller should be powered on Note When powered on the L1 controller displays L1 running Ifit does not display L1 running check the connection between the base compute module and the power source In addition ensure that the circuit breaker of the power source is on Power on the system console Ensure that the system console has the following network settings e Baud 38400 e No parity e 8 data bits e 1 stop bit e Hardware flow control on RTS CTS Type cu 1 ttys0O S38400 fora Linux based console and then press Enter to display the L1 prompt If the console uses another operating system start a terminal emulation program as instructed in the console s user documentation 007 4857 002 Connecting the System Console Console port Reset button Ethernet port g i L1 controller panel Figure 1 19 Location of the L1 Console Port 007 4857 002 37 1 System Installation Operating Your System This section describes how to operate your Altix 450 system as follows Powering the System On and Off on page 38 L1 Front Panel Controls on page 44 Operating the L1 Controller on page 45 Powering the System On and Off Preparing to Power On 38 This section describes how to power on and power off your system or an individual IRU at a system console or manually as follows Preparing to Power On on page 38 Powering Off at the System Console on page 42 Power
121. router has two ports that are brought out to external NUMAIink 4 connectors The 20U or 42U rack for this server houses all IRU enclosures option modules and other components up to a 76 processor core configuration 38 processor sockets in a single rack The Altix 450 server system needs a minimum of one IA IA2 blade base I O The rack system requires a minimum of one 20U high rack with one single phase power distribution unit PDU installed in the rack Each single phase PDU has 5 outlets to support two IRUs Figure 3 2 shows an example configuration of a 42U Altix 450 system tall rack 007 4857 002 System Models 007 4857 002 Figure 3 2 SGI Altix 450 Server System Tall Rack You can also add additional PCI expansion blades or RAID and non RAID disk storage to your server system Figure 3 3 shows an individual blade IRU and Rack 69 3 System Overview Rack Contains 4 IRUs Individual Rack Unit IRU Contains 5 blades Blade slot 2 Blade slot 3 Blade slot 4 5 ww 2 2 o o 0 o ne ke amp amp a a Figure 3 3 Blade IRU and Rack Components System Architecture 70 The Altix 450 computer system is based on a distributed shared memory DSM architecture The system uses a global address space cache coherent multiprocessor that scales up to 76 Intel
122. rsonnel under the following conditions e When the power cord or plug is damaged or frayed e Tf liquid has been spilled into the product e Ifthe product has been exposed to rain or water e Ifthe product does not operate normally when the operating instructions are followed Adjust only those controls that are covered by the operating instructions since improper adjustment of other controls may result in damage and will often require extensive work by a qualified technician to restore the product to normal condition e Ifthe product has been dropped or the cabinet has been damaged e Ifthe product exhibits a distinct change in performance indicating a need for service If a lithium battery is a soldered part only qualified SGI service personnel should replace this lithium battery For other types replace it only with the same type or an equivalent type recommended by the battery manufacturer or the battery could explode Discard used batteries according to the manufacturer s instructions Use only the proper type of power supply cord set provided with the system for this unit Do not attempt to move the system alone Moving a rack requires at least two people Keep all system cables neatly organized in the cable management system Loose cables are a tripping hazard that cause injury or damage the system 007 4857 002 Regulatory Specifications Regulatory Specifications The following topics are covered in this section e
123. s Laser Compliance Statements 007 4857 002 The DVD drive in this computer is a Class 1 laser product The DVD drive s classification label is located on the drive Warning Avoid exposure to the invisible laser radiation beam when the device is open Warning Attention Radiation du faisceau laser invisible en cas d ouverture Evitter toute exposition aux rayons Warning Vorsicht Unsichtbare Laserstrahlung Wenn Abdeckung ge ffnet nicht dem Strahl aussetzen Warning Advertencia Radiaci n laser invisible al ser abierto Evite exponerse a los rayos Warning Advarsel Laserstraling vedabning se ikke ind i str len Warning Varo Lavattaessa Olet Alttina Lasersateilylle Warning Varning Laserstralning nar denna del ar ppnad Ala tuijota sateeseenstirra ej in i stralen Warning Varning Laserstralning nar denna del ar 6ppnadstirra ej in i str len Warning Advarsel Laserstraling nar deksel apnesstirr ikke inn i str len 171 B Safety Information and Regulatory Specifications Lithium Battery Statements 172 Warning Ifa lithium battery is a soldered part only qualified SGI service personnel should replace this lithium battery For other types replace the battery only with the same type or an equivalent type recommended by the battery manufacturer or the battery could explode Discard used batteries according to the manufacturer s instructio
124. s the short 20U rack does not The rear door of the tall 42U rack has a push button key lock to prevent unauthorized access to the system The tall rack s front door and rear door locks are keyed differently You cannot use the rear door key to secure the front door lock Cable entry exit area Cable access openings are located in the front floor and top of the rack Cables are only attached to the front of the IRUs therefore cable management occurs primarily in the front of the rack In the case of storage modules cables are attached to the front connectors on the IRU and routed to the rear of the storage enclosure I O and power cables normally pass through the bottom of the rack Rack structural features Each rack is mounted on four casters the two rear casters swivel There are four leveling pads available at the base of the tall rack The base of both racks have attachment points to support an optional ground strap and or seismic tie downs Power distribution units within the racks Up to four single phase power distribution units PDUs are needed in a tall rack depending on the number of IRUs and other optional modules installed The 42U rack systems are also available with one 18 socket three phase PDU Short racks use a maximum of two single phase PDUs Each IRU requires two outlets from the 5 outlet single phase PDUs 007 4857 002 Altix 42U and 20U Rack Features Figure 5 1 Tall 42U Altix Rack and Front Lock
125. sole known as console mode a control d is entered at either the L1 prompt or the L2 prompt To return to or escape back to the L1 or L2 mode a control t is entered This escape is only temporary and you will be returned to console mode once the Return Enter key is pressed To re engage L1 or L2 mode enter control t followed by either 11 or 12 depending on the original mode System control commands are always entered in lower case unless otherwise specified Powering the System On and Off This section explains how to power on and power off individual rack units or your entire Altix 450 system as follows e Preparing to Power On on page 52 e Powering On and Off at the Console on page 54 For servers with a system console you can power on and power off individual IRUs or the entire system at the system console You may also monitor and manage your server with tools such as VACM Console Manager and PCP For details see the documentation for the particular tool The Embedded Support Partner ESP program enables you and your SGI system support engineer SSE to monitor your server remotely and resolve issues before they become problems For details on this program see Using Embedded Support Partner ESP on page 56 007 4857 002 51 2 Operation Procedures Preparing to Power On To prepare to power on your system follow these steps 1 Check to ensure that the cabling between the
126. t Information 007 4857 002 Warning To avoid problems that could void your warranty your SGI or other approved system support engineer SSE should perform any replacement of parts or service of your SGI Altix 450 system not covered in the following list of items that you can perform yourself e Unpacking setting up and connecting the system rack s and individual rack units IRU s e Using your system console and your L1 controller to enter commands and perform system functions such as powering on and powering off as described in this guide e Adding and replacing system blades and DIMM memory as described in this guide e Adding and replacing PCI PCI X and PCI express cards as described in this guide e Replacing fan modules and power supplies in the individual rack unit IRU enclosures e Adding and replacing DVD or disk drives in the base IA IA2 I O blade e Using the On Off switch and other switches the reset and non maskable interrupt NMI switches on the system JA or IA2 blade base I O xvii About This Guide Using the ESI ops panel operating panel on optional mass storage bricks See the user s guide for your optional SGI mass storage enclosure Chapter Descriptions The following topics are covered in this guide xviii Chapter 1 System Installation guides an installer through the steps of unpacking and positioning the system rack s Rack mounting guidelines and basic cabling for the IR
127. t conditions that include the use of shielded cables between the system and its peripherals Your system and any peripherals you purchase from SGI have shielded cables Shielded cables reduce the possibility of interference with radio television and other devices If you use any cables that are not from SGI ensure that they are shielded Telephone cables do not need to be shielded Optional monitor cables supplied with your system use additional filtering molded into the cable jacket to reduce radio frequency interference Always use the cable supplied with your system If your monitor cable becomes damaged obtain a replacement cable from SGI Electrostatic Discharge 170 SGI designs and tests its products to be immune to the effects of electrostatic discharge ESD ESD is a source of electromagnetic interference and can cause problems ranging from data errors and lockups to permanent component damage It is important that you keep all the covers and doors including any plastics in place while you are operating the system The shielded cables that came with the unit and its peripherals should be installed correctly with all thumbscrews fastened securely An ESD wrist strap may be included with some products such as memory or PCI upgrades The wrist strap is used during the installation of these upgrades to prevent the flow of static electricity and it should protect your system from ESD damage 007 4857 002 Regulatory Specification
128. tacle for North American Sites 007 4857 002 9 1 System Installation 10 For international sites follow these steps to ensure that a single phase power receptacle is properly wired 1 Set the voltmeter to a high AC voltage range 2 Check the voltage between socket and socket 2 see Figure 1 2 The meter should read between 200 and 240 VAC 3 Check the voltage between socket 1 line and the ground socket The meter should read between 200 and 240 VAC Note The ground pin is slightly larger in diameter than the other two pins 4 Check the voltage between socket 2 neutral and the ground socket The meter should read approximately 0 VAC 5 Check the voltage between the ground socket and an earth ground location The meter should read 0 VAC 6 Change the voltmeter to a low resistance setting 7 Measure between the ground socket and an appropriate earth ground location and ensure that resistance is less than 1 ohm 8 Repeat steps 1 through 7 for any additional single phase power receptacles Caution If a voltage reading is incorrect or if the resistance measured in step 7 is more than 1 ohm contact a site approved electrician Do not proceed with the installation 007 4857 002 Installing a Rack 007 4857 002 Pin 2 neutral Power cord connector Pin 1 line gt me Ground socket e p K Socket 2 Ground li neutral 2 Receptacle 3 gt Socket 1 gt ___ J
129. talling the IRUs in a Rack A This section describes how to install the IRUs that compose an Altix 450 system in a rack This section also describes how to remove the IRUs from a rack You can use the instructions to rackmount one or more IRUs in a 20U short or 42U tall Altix rack Caution Always use at least two people when moving installing or removing a rackmounted Altix 450 IRU This rackmounting information is found in the following sections e Determining Space Requirements on page 24 e Determining Where to Attach Rails in a Rack on page 25 e Rackmounting IRUs with Shelf Rails on page 26 Determining Space Requirements 24 Table 1 2 specifies the space requirements when rackmounting an IRU in a 19 inch rack Table 1 2 IRU Space Requirements Height 5U 8 68 inches 22 cm Width 17 5 inches 44 45 cm Depth 32 5 inches 82 5 cm with blower attached Weight 130 Ib 59 kg max config Required front clearance for IRU Required rear clearance for IRU Required side clearance for IRU Required front clearance for rack Required rear clearance for rack 8 25 in 20 96 cm 10 in 25 40 cm No clearance requirement 48 in 122 cm 48 in 122 cm a Weight will vary slightly depending on the number of blades installed and the amount of DIMMs installed and on whether you have one or two disk drives in the IRU 007 4857 002 Installing the IRUs in a Rack Determining
130. tem components Altix 450 server 77 list of 76 system controller 79 system features 72 system overview 67 T tall rack features 102 tall rack system weight 18 technical specifications system level 153 technical support xxi to monitor IRUs 57 troubleshooting L1 controller error messages 145 problems and recommended actions 144 175 Index U unpack a tall rack 18 unpack and move the short rack 16 upgrading L1 firmware 99 USB type A connector pin assignments 163 Vv VGA video card 57 176 viewing system configuration L1 mode 86 voltage select switches for peripherals 39 WwW warning messages viewing in L2 mode 89 94 weight of the short rack 16 007 4857 002
131. th console communications e Subchannel OA specifies Blade 0 CPU A e Subchannel OC specifies Blade 0 CPU C e Subchannel 1A specifies Blade 1 CPU A e Subchannel 1C specifies Blade 1 CPU C e Subchannel 2A specifies Blade 2 CPU A e Subchannel 2C specifies Blade 2 CPU C e Subchannel 3A specifies Blade 3 CPU A e Subchannel 3C specifies Blade 3 CPU C e Subchannel consoleO Blade 0 console subchannel e Subchannel console Blade 1 console subchannel The output from the select command console input 001c01 console0d shows that the system controller will send input to IRU 001c01 blade 0 and the subchannel to be used is the console subchannel During the boot process there is a window of time in which all processors may be producing output This output can produce a somewhat jumbled output at the L1 However you can filter the console output so that the L1 shows output from only the processor chosen to receive console input You can turn filtering on and off with the select filter command If you attempt to communicate with a IRU that is not responding a time out condition results 007 4857 002 L2 Operation 001c01 L1 gt entering console mode 001c01 console lt CTRL T gt to escape to L1 no response from 001c01 junk bus console UART UART_TIMEOUT When this time out condition occurs either the IRU is hung or the subchannel is incorrect An IRU is identified by its rack type and slot
132. the block it is in a shared state a bit vector indicates which caches contain a copy When a processor modifies a block of data the processors that have the same block of data in their caches must be notified of the modification The Altix 450 server series use an invalidation method to maintain cache coherence The invalidation method purges all unmodified copies of the block of data and the processor that wants to modify the block receives exclusive ownership of the block 73 3 System Overview Non uniform Memory Access NUMA In DSM systems memory is physically located at various distances from the processors As a result memory access times latencies are different or non uniform For example it takes less time for a processor blade to reference its locally installed memory than to reference remote memory Reliability Availability and Serviceability RAS 74 The Altix 450 server series components have the following features to increase the reliability availability and serviceability RAS of the systems e Power and cooling IRU power supplies are redundant and can be hot swapped under most circumstances Note that this might not be possible in a fully loaded system If all the blade positions are filled be sure to consult with a service technician before removing a power supply while the system is running IRUs have overcurrent protection at the blade and power supply level Fans are re
133. the blade e One green LED shows power on complete status for the blade e One red LED shows power failure or bad voltage status within the blade e Two green NUMAIink indicators show NIO and NII connection status between the blade and the router board within the IRU Constant green is a good connection e Four amber heartbeat LEDs indicate compute activity the LEDs light up according to the number activity and type of processors installed in the blade If the IRU is fully powered on and booted and none of the amber LEDs are lit there is most likely a problem with the compute memory blade Try reseating the blade in the slot Confirm the two green NUMAlink status LEDs are on If there is no LED activity on the blade it must be replaced 148 007 4857 002 SGI Electronic Support SGI Electronic Support 007 4857 002 Power fail NIO link N NI1 link Power on mm Logical x oe processor A F Logical processor D Logical Logical processor B processor C Compute Blade Status LED Locations SGI Electronic Support provides system support and problem solving services that function automatically which helps resolve problems before they can affect system availability or develop into actual failures SGI Electronic Support integrates several services so they work together to monitor your system notify you if a problem exists and search for solutions to problems Figure 7 2 shows the sequence of events that occ
134. the following command at the L1 prompt to power on the server 001cO1 L1 gt power up 2 Fora system with multiple IRUs enter the following command at the L1 prompt 001c01 L1 gt power up 41 1 System Installation Powering Off at the System Console ZN 42 To power off your system using the system console follow these steps Caution If you power off the system before you halt the operating system you can lose data Shut down the operating system by entering the following command init 0 To access the L2 prompt direct the keyboard input to the L2 controller by entering Ctr1 T gt ctrl t From the L2 prompt L2 gt power off an individual IRU by entering the following command If you want to power off the entire system proceed to Step 4 L2 gt r lt rack gt s lt slot gt pwr d For example to power off the IRU in rack 1 slot 16 enter the following command L2 gt r 1 s 16 pwr d The slot number is the unit number of the IRU within the rack If you want to power off more than one IRU in the rack at the same time enter the rack number followed by the slot numbers of the IRUs that you want to power off For example to power off the IRUs in slots 16 and 21 enter the following command L2 gt r 1 s 16 21 pwr d If you want to power off all of the IRUs within the rack enter the following command L2 gt pwr d The default setting for the pwr d command is all racks and slots
135. the rack in which you will install the IRU See Determining Where to Attach Rails in a Rack on page 25 if you are uncertain 2 Inthe slot location from step 1 position the left shelf rail at a 90 degree angle to the front and rear rack rails see Figure 1 12 3 Insert the top front bracket tab into one of the narrow slots in the front rack rail 4 Insert the top rear bracket tab into the narrow slot in the rear rack rail Note Verify that the tabs are the same height to ensure that the shelf is level in the rack 5 Lower the bottom of the shelf rail until its two lower tabs engage with the wide slots in the front and rear rack rails The shelf rail is held in by gravity and the tabs 6 Repeat these steps for the right shelf rail Figure 1 12 Installing the Shelf Rails in the Rack 26 007 4857 002 Installing the IRUs in a Rack Note Even with the blades power supplies and fan module disconnected the IRU s base unit is heavy Always use two people when installing it into a rack 7 Working from the front of the rack set the rear edge of the IRU module on the shelf rails Then slide the module into the rack until the module ears are snug against the front rack rails See Figure 1 13 8 Fasten the module to the front rails of the rack with the screws provided See Figure 1 13 9 Working from the rear of the rack align the rear connectors of the fan module with the rear of the IRU b
136. tions Failure to do so can result in damage to the equipment 007 4857 002 Safety Precautions Safety Measures 007 4857 002 ZN O Observe the following safety measures when you install the system Use caution when you remove the system from the shipping crate Failure to handle the system carefully can result in personal injury or property damage Caution Ensure that the shipping crate is positioned close to its destination before you unpack the crate Warning Employ a minimum of two people to lift an IRU to move the IRU s from one location to another and to install the IRU s in a rack Otherwise someone could be seriously injured Note that IRUs should have blades and power supplies removed to make the unit easier to lift and position Do not move the system while it is connected to power Warning Keep fingers and conductive tools away from high voltage areas Failure to follow these precautions will result in serious injury or death The high voltage areas of the system are indicated with high voltage warning labels Ensure that a qualified electrician has properly installed the power receptacles Set all PDU circuit breakers to the OFF O position before you plug in the system power cord Warning Use the following guidelines to prevent the rack from toppling over Otherwise people could be seriously injured and or equipment could be damaged Follow these guidelines to prevent a r
137. to the equipment that could result from thermal shock and condensation 15 1 System Installation Removing a Short Rack from the Shipping Crate O A 16 Warning Be careful when you unpack and move the short rack system Use a pallet jack to move the rack to the unpacking site Ensure that the rack remains on a level surface and that the rack weight remains evenly distributed across the four casters If you must lift the casters over an obstacle such as a door threshold use proper lifting techniques and employ a minimum of two people Caution Do not subject the rack to any unnecessary shocks or vibration while you unpack and install the system See Figure 1 5 as you follow these steps the numbered illustrations correspond to the numbered steps 1 Ensure that the temperature of the rack is acclimated to the environment in which you are installing it and move the crate to the installation location 2 Remove all external banding and lift the crate s ramp away from the unit 3 Remove the cardboard carton top and packing insulation materials 4 Prepare to move the rack off the crate using the following steps a Remove the two bolts that hold the pallet s rear retention bracket in place b Remove the two bolts that hold the rear gate of the pallet in place c Remove the two bolts that hold the pallet s front retention bracket in place 5 Align the holes in the edge of the ramp with the holes at
138. tput not filtered During the boot process on a system with multiple CPUs there is a window of time in which the CPUs are all producing output This can result in a somewhat jumbled output from the L2 However you can filter console output so that the L2 will show output from only the IRU chosen to receive console input You can turn on filtering with the select filter on command and turn off filtering with the select filter off command If you attempt to communicate with an IRU chosen to receive console input but that is not responding a time out condition results L2 gt Ctr1 D entering console mode 001c01 CPU1A lt CTRL_T gt to escape to L2 no response from 001c01 Junk bus CPUI1A system not responding no response from 001c01 Junk bus CPUI1A system not responding When this time out condition occurs either the IRU is hung or the subchannel is not correct 97 4 System Control L1 Mode From L2 98 Generally the L2 will track the correct IRU rack slot blade and subchannel for system console input and no user setting should be required If the default value was manually overridden it can be reset by entering the following command 001c01 L1 gt select reset In L1 mode the prompt from a single L1 is visible and all input is directed to that L1 command processor To enter L1 mode enter the rack and a slot followed by 11 L2 gt r2s111 An alternate method is L2 gt 2 1 11 enterling L1 mode 001c01 lt CTR
139. troller function in the optional storage modules The L1 controllers within the system report and share status information via the NUMAIink cables thus maintaining controller configuration and topology information between all controllers in the system Note Mass storage option enclosures are not specifically monitored by the system controller network Most optional mass storage enclosures have their own internal microcontrollers for monitoring and controlling all elements of the disk array See the owner s guide for your mass storage option for more information on this topic System Controller Interaction L1 Controller 007 4857 002 In all Altix 450 servers all the system controllers communicate with each other in the following ways e All IRU enclosures within an Altix 450 system communicate with each other through their NUMAlink connections using low voltage differential signaling LVDS e When connecting to the L2 host Ethernet connection on the system control board of an IRU the system controller spawns an L2 application providing L2 functionality All Altix 450 IRUs have L1 controllers The following subsections describe the basic features of the L1 controllers e L1 Controller Functions on page 82 e L1 Front Panel Display on page 82 Note For additional information on L1 controller commands see the SGI L1 and L2 Controller Software User s Guide 007 3938 xxx 81 4 System Control
140. uct or components of this product on an unstable cart stand or table The product may fall causing serious damage to the product Slots and openings in the system are provided for ventilation To ensure reliable operation of the product and to protect it from overheating these openings must not be blocked or covered This product should never be placed near or over a radiator or heat register or in a built in installation unless proper ventilation is provided This product should be operated from the type of power indicated on the marking label If you are not sure of the type of power available consult your dealer or local power company Do not allow anything to rest on the power cord Do not locate this product where people will walk on the cord Never push objects of any kind into this product through cabinet slots as they may touch dangerous voltage points or short out parts that could result in a fire or electric shock Never spill liquid of any kind on the product 165 B Safety Information and Regulatory Specifications 10 11 12 13 14 166 Do not attempt to service this product yourself except as noted in this guide Opening or removing covers of blade and switch internal components may expose you to dangerous voltage points or other risks Refer all servicing of items not listed in this guide to qualified service personnel Unplug this product from the wall outlet and refer servicing to qualified service pe
141. urs if you use all of the SGI Electronic Support 149 7 Troubleshooting and Diagnostics a Customer s system Implement e mail solution Page or e mail Supportfolio amp lt a Online ale View the case solutions SGI customer and SGI global SGI support engineer customer support center SGI Knowledgebase Figure 7 2 Full Support Sequence 150 007 4857 002 SGI Electronic Support 007 4857 002 The sequence of events can be described as follows 1 2 6 Embedded Support Partner ESP monitors your system 24 hours a day When a specified system event is detected ESP notifies SGI via e mail plain text or encrypted Applications that are running at SGI analyze the information determine whether a support case should be opened and open a case if necessary You and SGI support engineers are contacted via pager or e mail with the case ID and problem description SGI Knowledgebase searches thousands of tested solutions for possible fixes to the problem Solutions that are located in SGI Knowledgebase are attached to the service case You and the SGI support engineers can view and manage the case by using Supportfolio Online as well as search for additional solutions or schedule maintenance Implement the solution Most of these actions occur automatically and you may receive solutions to problems before they affect system availability You also may be able to return your syste
142. which the receiver is connected Consult the dealer or an experienced radio TV technician for help Caution Changes or modifications to the equipment not expressly approved by the party responsible for compliance could void your authority to operate the equipment 007 4857 002 Regulatory Specifications Industry Canada Notice Canada Only This Class A digital apparatus meets all requirements of the Canadian Interference Causing Equipment Regulations Cet appareil num rique n met pas de perturbations radio lectriques d passant les normes applicables aux appareils num riques de Classe A pr scrites dans le R glement sur les interferences radio lectriques tabli par le Minist re des Communications du Canada VCCI Notice Japan Only TOREJ fe AGL eS aS A Eh ikea VCCI OSE HE IZ FEDS ae Alpi Ria CT TORE ERER CHAT SE RGSS SRCFCEMHOES COWS aA ws WREST SLIBRENSETEMHVET Figure B 1 VCCI Notice Japan Only Chinese Class A Regulatory Notice AERE SEEPANA EERE fea FARE APR ink THE ee Feil ee eR UHR Figure B 2 Chinese Class A Regulatory Notice Korean Class A Regulatory Notice Oo 77E AER OZ AJJAJ SF 7 7 0 9 Bry Ae a ALBA Als Z oja 7 D wu abo 2 yho t TAAS oye 7AL aAa y Figure B 3 Korean Class A Regulatory Notice 007 4857 002 169 B Safety Information and Regulatory Specifications Shielded Cables This SGI system is FCC compliant under tes
143. x 450 systems Use of other racks are at the discretion of the owner and no testing of usability serviceability or durability is implied Note While the Altix 450 can be installed in a non Altix 450 rack it is preferable to install it in an Altix 450 tall 42U or short 20U rack The mounting rails on the Altix racks are recessed to allow for flush mounting of the IRU in the rack The Altix rack also has provisions for PDUs to be mounted in the front Installing the Altix 450 in a third party rack will require that the unit protrude from the front of the rack It will not be possible to use a front door on the third party rack due to the protrusion of the unit Power cables for the IRU power supplies will need to be routed to the rear of the rack if the PDUs are located in the rear of the rack Airflow in the rack must be front to rear Other devices installed in a rack must have their airflow direction front to rear Cables may need to be routed from the rear of the devices to the front of the Altix 450 IRU Vertical space must be left U space for cable routing front to rear in the rack 101 5 Rack Information Altix 42U and 20U Rack Features 102 The Altix racks shown in Figure 5 1 and Figure 5 2 have the following features and components Front and rear door The front door is opened by grasping the wide end of the triangle shaped door piece and pulling outward Note that the tall rack uses a key lock for security purpose
144. ystem main memory DIMMs 4 per system image 228 per system image Number of base I O blades One per IRU 4 per system image a Single socket compute blades support two cores each Dual socket blades support two or four cores per blade b Two complete systems of four IRUs each can be installed in the same 42U tall rack 153 A Technical Specifications and Pinouts Physical Specifications Table A 2 shows the physical specifications of the Altix 450 system in a 42U rack Table A 2 Altix 450 Tall Rack System Physical Specifications Feature Specification Dimensions for a single tall 42U rack system including doors and side panels Shipping dimensions Weight of full rack maximum Shipping weight maximum Access requirements Front Rear Side Individual Rack Unit IRU enclosure specifications Height 79 5 in 201 9 cm Width 25 8 in 65 5 cm Depth 43 45 in 110 4 cm Height 81 5 in 207 cm Width 42 in 106 7 cm Depth 52 in 132 1 cm 1450 Ib 658 kg 1600 Ib 727 kg 48 in 121 9 cm 48 in 121 9 cm None Dimensions 6 68 in high x 17 5 in wide x 17 in deep 22 cm high x 44 45 cm wide x 43 2 cm deep Weight 130 Ibs 59 1 kg Table A 3 on page 155 lists the physical specifications of the Altix 450 system in a 20U rack 154 007 4857 002 Physical Specifications 007 4857 002 Table A 3 Altix 450 Short Rack System Physical Specifications Feature

Intel SGI Altix 450 User's Manual

Contents

Download Pdf Manuals

Related Search

Related Contents