Home

Cisco Systems 270 Switch User Manual

1. Fan Status LEDs Fabric Controller Status LEDs Fabric Controller a Master Status LEDs 19 Figure 5 1 Front Panel LEDs System Wide Status LED Locating the System Wide Status LED The system wide status LED is located on the front of the chassis and is visible whether the bezel cover is on or off This LED indicates the overall health of all modules in the system If you are facing the front of the chassis the system wide status LED is found on the top left corner of the switch In addition a mirror of the system wide LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 73 Interpreting the System Status LED Table 5 1 Interpreting the System Wide LED see Figure 5 1 Color State Indication off No system power or LED failure Green solid on System running with no detected errors Yellow solid on Failure within the system Operator attention required 69 Table 5 1 Interpreting the System Wide LED see Figure 5 1 Color State Indication Yellow blinking System identify This state is initiated by the user This LED can be initi
2. Fan ID 1 Fan ID 2 y 000000 00000 0000 ODODOD 2 2 000000 00000 0 eo 000000 oOoooog 000000 o00000 2 Slot 9 obooooo 6 o e 0000 obooooo DODOO Slot 10 oooo0o00 H 000000 oo0Do0Do 00000 Slot 11 ooooo00 o000D00 00000 nnmnnn Slot 12 S o 000000 000000 OODODO Slot 13 a na 5 oo0000 000000 Ea o Slot 14 o 000000 000000 000000 000000 000000 Figure 1 3 Chassis Slot Numbering in Front Non port Side 16 Slot 1 p Slot 2 Slot 3 Slot 4 Slot 5 Slot 6 Slot 7 Slot 8 Slot 15 Slot 16 Slot 17 og OOO A o00000 la googog DOOD Ope 288 E oO o lo MESES fa a pODoOO DODOO DO one ial eae T y ei ee Ae fafa esse OO0000000000000000000000 OO000000000000000000000000000 EE ESSE ES SHI o o OO000000O0000000ggggo0ggggogogogogoggoggogogogggggggoogogoogog000g00n0n00 ES EI SS EE ESE o BODODODODODO OOOO OOOO OOOO OOOO OOOO OOOO DOO ODO O0000o000d0o Y TO co 99 8 8 8 8 8 5 8 8 8 8 8 8 8 8 8 8 8 8 OCOOOOO GOOCOOCCCnn o OODOOOODODODODODODODOOO OOOO OODODODODODOODO 0000000000000 ESSE ESI iii ooooo0oooonooonoonocoococoocoococoocococoococoocooooooocononocooonooo00o00 AAA o BO00000000000000000000000000000
3. 3 i Jot Ale sie O HBooSeooHpoooes e SES SS SSS SES 4 H A H H 4 4 ONOO000O0OO00oOoOoOoOoOoOoOo0oOooognoooOoOogooOnoOoOoOoOoOoOoOoOnooOoOooooOooooonoooon HEI BeEceoess pS o BODODOOOOO OOOO DO OOOO OOOO OOOO OOOO OOOO ODO OOO DO ODODOOOODOo0O0oO SS oS SS OOOO 000000 00000000 OOO00O00000 00000000 DOOOOOOODODODOODO o o BDODOOOODO OOOO OOOOOOOO DO DO OOOO ECCUCI ECC iia ogoogo OODOO OOODO pBODODODOODOO DO OO DO DO OOOO ODO OOOO ODO DO OOOO OOOO 000000000000 olaaa o BOODODOOOOOODOCOOOODODOOOOOOOOOOOODOOOOOOODOOOODODOODODODODODO Figure 2 6 Serial and Ethernet Management Ports o o 0000 000000 000000 000000 000000 00000000 DODOODODO OOO DO DO OO OOO OOO OOO0O00 o00o0n000 DODODODOODODOOOOO OOO ODO 00000000 ODOODODOO ogoogo DOOODO Attach a Serial Console Cable to a PC or Terminal 1 Attach the RJ 45 console cables from the cable kit that is provided See Figure 2 7 a Connect the cables to the InfiniBand chassis serial console port on the Management Interface card If you have two Management Interface cards make sure that you connect to the left Management Interface module slot 15 which will be the primary Management card upon initial boot up The serial console port is labeled 10101 b Connect the other end of the serial cable to your terminal server or management workstation For detailed inform
4. Power on the Chassis Use only the power cable provided with your InfiniBand system 4 Remove the power cords from the shipping package One power cord UL rated 12 Amps 125 VAC or greater is provided by default If you have a high availability unit you will have a second power cord 5 Inspect the power cord and determine if it provides the proper plug and is appropriately certified for use with your electrical system Discard the cord if it is inappropriate for your national electrical system and obtain the proper cord as required by your national electrical codes or ordinances Grounding is supplied by the ground prong on the 3 prong power plug Do not attach a separate ground cable Do not use adapter plugs Do not remove the ground prong from the cable Ensure the ground connection on the power supply is correct and functioning before applying power to the chassis 6 Insert the power cords to the power jacks on the rear of the chassis If you only have a single power cord insert it into the left receptacle Refer toFigure 2 8 for power jack locations ADODODODODODO OOO ODOODO DODOODOODOOOOOODOODODODODOO O yooooo ea E BBB og T 20 sui Aso Asa o osooso ES o Ge oo0o00o00o00o0000o0oononooooooonooonoooooooocoocoocoooonoonoooon SSS REEESeseieie DODOODODOODO DO DO DO DO ODODO OOO OOOO OOOO OOOO OOOO DD 000000000 a DODOODODODO OOOO DO OO OOOO OOO OOOO OOOO OOO ODO 000000000 00000 SHS HE HEHE HEHEHA
5. within the Power Supply Table 5 3 Interpreting the Power Supply Identify LED Color Yellow State blinking Fan LEDs Location Indication Identify Assists in identifying a particular Field Replaceable Unit on the chassis see Figure 5 3 This state is initiated by the user This LED status is initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run identity test on fan 1 config mode gt power supply 1 gt test led gt gt start start flashing gt stop stop flashing 71 The Fan LEDs are located at the top of each Fan tray module Refer to Front Panel LEDs on page 68 In addition replicas of the Fan Tray LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 73 Interpreting the Fan LEDs Table 5 4 Interpreting the Fan LEDs see Figure 5 1 Color State Indication Green and off No power to the fan tray or LED failure Yellow Green solid on Fan tray running with no detected errors Green off No power to the fan tray LED failure or yellow LED is ON Yellow off Fan tray running with no errors detected Yellow solid on Operator intervention required Failure detected within the fan tray Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by
6. 5 to 90 RH at 65C for 24 hrs Non condensing Operating Humidity Gradient 10 maximum per 60 min Operating Shock 5G max 11ms half sine wave 10G max 5ms half sine wave Non Operating Shock 10G max 11ms half sine wave Operating Vibration Sinusoidal 0 25G max 3 200Hz 15min Non Operating Vibration Sinusoidal 0 50G max 3 200Hz 15min Operating Vibration random 1 02Grms 3 axis bottom top left right front back 106 Table 7 1 Chassis and Management General Specifications Operating Temperature Ranges 0 to 40C Non Operating Vibration random 2 09Grms 3 axis bottom top left right front back Max Operating Inclination 15 degrees Electrical Specifications Table 7 2 Electrical Specifications Category Specification AC Input Auto ranging 90 264VAC 47 63Hz Maximum Power lt 600 Watts consumption EMC Immunity Table 7 3 EMC Immunity Description FCC CFR 47 Part 15 Subpart B Class A ICES 003 Issue 2 EN 61000 3 2 Harmonics EN 61000 3 3 Flicker EN 55022 1998 EN 55024 1998 EN 61000 4 1 2 3 4 5 6 8 11 Japan VCCI V3 02 04 Safety Table 7 4 Safety Description UL60950 3rd ed CSA 22 2 No 60950 2000 IEC60950 EN60950 EN60825 1 and EN60825 2 Acoustics Table 7 5 Acoustics Country Description Deployment Sound Pressure 37dB at 25C ambient ISO 7779 and section 8 5 of ISO 3744 1994
7. 1 Launch Element Manager and navigate to your Topspin 270 Cisco SFS 7008 2 Verify that the images that run on the fabric controller modules do not match If they match you do not need to upgrade the new module a Right click the running fabric controller module and select Properties b Click the Inventory tab and note the image that runs on the card c Right click the new fabric controller module and select Properties d Click the Inventory tab and note the image that runs on the card to see if it matches the first card 3 Click the Maintenance menu and select File Management The File Management window opens Click the Import button The Import File window opens From the File Type pulldown menu select image Click the Remote FTP Server radio button Enter the DNS name or IP address of the FTP server that holds the file that you want to import in the Server Name or IP Address field Enter the user ID that logs you in to the FTP server in the User Name field Sh Se R 9 Enter the password that logs you in to the FTP server in the Password field 10 Enter the directory path and name of the file on the FTP server in the File Path and Name field 11 Enter the name that the file will take on your Server Switch in the File Name on System field 12 Click the Copy button 13 Click the line in the Current Files on System table that lists the file that you want to install then click the Install button A verification win
8. SES 2 7 Os 4 Copy the image to the switch if you do not have the correct img file copy ftp lt remote login gt lt remote passwd gt lt remote system gt lt filename gt img lt slot number of the controller gt image lt filename gt img Example SFS 270 copy ftp bob mypassword 10 0 0 5 SFS 270 TopspinOS 2 3 0 build531 img IMACS SINS Z I O Z o 1 6 Ojos EAS 7 sume LeEjOSsoOiMOS Z 1 0 000111 0497 ej 59 5 Activate the image After downloading the image file to the chassis controller it must be installed to become active The install command installs the specified image file into the system a Inthe privileged execute mode enter install image image file name img Example SES 2 70 ansia abr weasSrS 27 O TopsiolinOS 2 2 O0 lowiliclo3 i sine TTS TE LES OSLO COM le use SuceSsssicwulily 6 Specify the new boot image to be used a Find the name of your new system image by entering the dir image command Copy the name of the new image b Enter boot config primary image source then paste the copied image file name Example SFS 270 configure SFS 270 config boot config primary image source TopspinOS 2 2 0 build531 SFS 270 config exit 7 Reset the individual Controller card without rebooting the whole chassis Example SFS 270 config card 12 SFS 270 config card 12 action reset Upgrading Individual Node Cards Upgrading a Node Card with the CLI For the commands needed to upgr
9. e Note Fabric controllers in Node slots do not act in active standby configuration Node and Core Slots The fabric controller modules can occupy either a Node slot or a Core slot in the chassis The Node Slot There are four Node slots in the Topspin 270 Cisco SFS 7008 Fabric modules installed in Node slots provide fabric leaf switching as opposed to controller functionality The Core Slot There are two Core slots in the Topspin 270 Cisco SFS 7008 19 e At least one Fabric Controller module must be installed in the Core slot for the chassis to boot When a module is installed in a Core slot it acts as either the master or standby controller for the chassis with all modules in the system being slaves to the master controller The subnet manager runs on the fabric module installed in the Core slot that has arbitrated to be the master The Core slot module also acts as the switch fabric spine interconnecting the end nodes Each Core slot is connected to 1 of 2 ports on the Management Interface module The pairing of core slot to Management I O module is as follows e The Fabric Interface Module in the Core slot 11 is connected to the Management I O module in slot 15 e The Fabric Interface Module in the Core slot 12 is connected to the Management I O module in slot 16 Line Interface Modules LIM Refer to Installing a LIM Module on page 48 for installation instructions The Topspin 270 Cisco SFS 7008 h
10. f Click Ok 4 Install the new image 55 Note all cards in the chassis must have an oper status of up for the upgrade to be effective on all components a Select Maintenance gt File Management b Click the new img file and click the Install button c Click Yes when prompted d Repeat for all Server Switches you wish to upgrade File Management x You can import a file from an external FTP server export a file to an FTP server install an image in the system or delete a file from the system Current Files on Systern Slot ID File Name File Type Size Date Topspin270 TopspinOS 2 2 0 build531 img image 10657535 Thu Jan 27 01 07 38 11 TopspinOS 2 2 Ofuild530 i 6684672 Wed Jan 26 01 07 11 hwif_log log 34113 Thu Jan 27 01 04 4 Export 11 istartup config config 271 Wed Jan 26 01 04 4 11 tslog llog 239917 Thu Jan 27 01 08 2 Install X El 2 Figure 4 2 Watching the File Copy Progress e Click Ok when the Image Install Completed Successfully message appears Configure the image for the next boot a Select Maintenance gt Boot Config b Select the new image from the Image Source drop down menu and click the Apply button Boot Configuration Software Images Current Image Yersion TopspinOS 2 2 O build530 Image Source for Next Reboot MATAS a0 ec v Startup Configuration Overwrite startup configuration with Delete startup co
11. gt start start flashing gt stop stop flashing Management Interface Status LEDs Table 5 8 Interpreting the Management Interface Module Status LEDs see Figure 5 6 Green over Yellow off No module power or LED failure Green solid on The Management Interface module is running with no errors detected 75 Table 5 8 Interpreting the Management Interface Module Status LEDs see Figure 5 6 Fan Tray Module Status LEDs Color State Indication Green off No power to the Management Interface module LED failure or yellow LED is ON Yellow solid on Failure of the Management Interface module Operator attention required The label indicates a failure Yellow blinking Identify a Management Interface module Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example For example to run the identity test on Power Supply 1 config mode gt diag card 15 gt test led gt gt start start flashing gt stop stop flashing Table 5 9 Interpreting the Fan Tray Status LEDs see Figure 5 6 Power Supply Module Status LEDs Color State Indication Green over Yellow off No module power or LED failure Green solid on Fan tray is running with no errors detected
12. 1 phillips screw driver to tighten the captive screws E a Figure 3 8 Tightening Captive Screws 42 9 Check the LEDs on the fan module to verify the status Refer to Fan LEDs on page 71 Fabric Controllers About the Fabric Controllers Refer to Fabric Controller Modules on page 18 for more specific information about this module About a Failed Fabric Controller If you believe a fabric controller module has failed contact technical support Refer to Fabric Controller LEDs on page 72 and Monitor the Cards on page 81 for monitoring information Locating a Failed Fabric Controller The fabric controllers are located on the front of the chassis behind the bezel cover Refer to Figure 1 1 for a diagram of the chassis The slots are labelled on the chassis Considerations Before Removing a Fabric Controller Card Before removing a fabric controller card from either a node or a core slot of the Topspin 270 Cisco SFS 7008 keep the following in mind 1 The fabric controllers cards in the core slots are paired to the Management Interface modules on the other side of the chassis e Fabric controller card in slot 11 is paired to the Management Interface module in slot 15 e Fabric controller card in slot 12 is paired to Management Interface module in slot 16 Therefore before removing a fabric controller from a core slot make sure that the second pairing of Fabric Controller Management Interface module
13. El a Management Ports O ib4xFX down SAs Maintenance O 52 52 ib4xFX up down 0 a mi On 5 3 5 3 ib4xFX up down 0 O 5 4 5 4 ib4xFX up down 0 OTOS 5 5 ib4xFX up down 0 On 0 5 5 6 ib4xFX up down 0 e 57 5 17 ib4xFX up down 0 On pis 5 8 ib4xFX up down 0 SF 5 9 ib4xFX up down 0 On SAD AO ib4xFX up down 0 CN Sahl pst ib4xFX up down 0 Jae AZ ib4xFX up down 0 Data Refreshed At Friday April 23 2004 1 03 31 PM Figure 5 13 Chassis Manager Ports Window 83 7 View the Admin Status and the Oper Status of the ports The Admin Status refers to the state that the admin has chosen the port to be The Oper Status refers to the actual state of the port For example an admin may have selected a state of Up for a port but the Oper Status would display down if a cable had not been connected to the port 8 Click the radio button next to a port whose details you want to view 9 Click the Properties button The Port Properties dialog box appears Port Properties Port 5 3 Name 5 3 Type ib4xFX Admin Status up O down Oper Status down Auto Negotiation Supported true Auto Negotiation Enable Set Port Speed 2500M 10G 30G Current Speed unknown MTU 0 Last Changed On none Figure 5 14 Chassis Manager Port Properties Dialog Box Monitor the Power Supplies 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Log into the chassis 3 The Device Re
14. e Ifyou have determined that the switch will be justified toward the front of the rack then use a 2 phillips screwdriver to unscrew the rack ears from the back of the switch 24 e If you have determined that the switch will be justified toward the back of the rack then use a 2 phillips screwdriver to unscrew the rack ears from the front of the switch Figure 2 2 Unscrew Extra Rack Ears from the Chassis 6 Separate the rack ears from the chassis when all screws have been loosened 7 Use 2 3 people to lift the switch into the rack Figure 2 3 Chassis Set onto Shelf 8 Push the chassis into the rack until the rack ears are flush with rails Figure 2 4 Rack Ears Flush with Rails 9 Use screws that fit your rack to attach both rack ears to the rack rails Figure 2 5 Use Appropriate Screws to Attach Rack Ears to Rails 23 26 Install Optional HA Components If you have purchased a High Availability HA package install the following components e Redundant Power Supply Refer to Power Supply Modules on page 35 Redundant Fan Tray Refer to Fan Tray Modules on page 39 Redundant Management Interface Module Refer to Management Interface Modules on page 49 Connect Management Devices To connect the management ports use either a serial cable and or an Ethernet cable Ethernet Serial Ethernet Serial Port Port Port Port to a Ee Ages DDD B o oo ae oooogg Y MO o o Milo 32
15. 29 Configure Basic Connectivity 1 Configure an IP address for the Ethernet Management port It is necessary to configure connectivity to the out of box Ethernet management port in order to use Telnet the Web GUI or the Java GUI The system is not pre configured when it ships from the factory IP addresses can be assigned statically which is the default or dynamically assigned using DHCP e Assigning an Address Statically on page 29 e Assigning an Address Dynamically on page 30 Assigning an Address Statically Refer to the static example below a When the system has completed booting press lt Enter gt several times to display the CLI prompt LCi La 2 b Enter the user name and password The default user name is super and the default password is super Login super Password super SES 2 70 gt c Enter the enable command SFS 270 gt enable SFS 270 d Enter the configure command SFS 270 configure SFS 270 config e Enter the interface mgmt ethernet command SFS 270 config interface mgmt ethernet f Enter the IP address of the management port followed by the netmask SFS 270 config if mgmt ethernet ip address 10 10 0 22 255 255 255 0 g Set the default gateway for the management port For example SFS 270 config if mgmt ethernet gateway 10 10 0 1 h Enable the management port by entering the no shutdown command SFS 270 config if mgmt ethernet no shutdown i Save the
16. Controller Module on page 63 Cards in a Chassis These procedures assumes you are upgrading all fabric controllers and node cards in the chassis Upgrading a Topspin 270 Cisco SFS 7008 with the CLI For the commands needed to upgrade a Topspin 270 Cisco SFS 7008 chassis using the CLI refer to the CLI Reference Guide 54 Upgrading a Topspin 270 Cisco SFS 7008 with Element Manager The following procedure describes the upgrade procedure using the Element Manager GUI This procedure also assumes you are upgrading all fabric controllers in the chassis 1 Launch Element Manager for the server switch 2 Delete any old img files to make room for the new image a Select Maintenance gt File Management b Click on a img file if one exists and click the Delete button c Repeat for all Server Switches you wish to upgrade 3 Copy the new image to the Server Switch a Select Maintenance gt File Management b Click the Import button e File Type Image Select Remote FTP Server or Local File to navigate to the file Import File Copy a file to the system a File Type image v Copy From Remote FTP Server Local File alfile Topspin270 TopspinOS 2 2 0 buildS31 img Copy To File Name on System Topspin270 TopspinOS 2 2 D build531 img Figure 4 1 Copy the img File to Server Switch e Click the Copy button and wait for the message that indicates the copy was successful
17. E Sound Power 50dB at 25C ambient ISO 7779 section 8 6 of ISO 3744 1994 E Product Markings UL FCC Statement VCCI CE ICES 107 108 Index A active controller card 11 admin status 83 C card operational state 81 reset 82 card reset 82 chassis ID about 19 LEDs 76 chassis manager launching 33 LEDs 80 controller card identify active 11 upgrade 59 controllerFabric12x 81 core slot about 18 D diag card 92 diag chassis test standard 95 diag power supply 99 F fabric controller LEDs general 72 master status 72 module status 72 fabric controller modules about 18 fabric12x 81 fail over fabric controller 42 management interface module 18 management setup 27 fan LEDs 71 H hot standby 11 I ib12port4xTX 81 idModule 81 inactive controller card 11 InfiniBand port LEDs 77 L launching Chassis Manager 33 LEDs 73 76 chassis ID 76 chassis manager 80 ethernet management port 76 fabric controller 72 fans 71 InfiniBand ports 77 LIM 77 line interface module 77 management interface module 73 master management connection 76 power supply 69 75 system status 68 LIM about 19 LEDs 77 line interface modules about 19 link 78 M management interface module LEDs fans 75 general 73 master connection 76 mgmt interface 74 power supply LEDs 75 system status 74 mgmt O 81 more command 94 MTU 83 N node slot about 18 O oper status 83 operational state 81 P port admin status 83 oper status 83 po
18. Example i SFS 270 show card 25 So 4 Do Card Information Ge es admin oper admin oper oper os slot type type status status code Q SRS SSS SS SSS SSS SS SS SS SS 22222 222225 SS SSS SS SSS SS SSS SSS SS SSS SSS 110 o 11 controllerFabricl2x controllerFabricl2x up up normal ae 12 controllerFabricl2x controllerFabricl2x up up standby The oper code field of the command output identifies the controller card as the active card normal or the hot standby card standby By default when you power on a Topspin 270 Cisco SFS 7008 the card in slot 11 becomes active and the card in slot 12 becomes the standby The master and standby controllers automatically synchronize state and configuration information InfiniBand Connectivity The Topspin 270 Cisco SFS 7008 is an enterprise class managed 960Gb S 6U InfiniBand IB Switch that is capable of supporting up to 96 4X ports In addition the Topspin 270 Cisco SFS 7008 is designed to support the 12x InfiniBand ports when they become available Flexibility The architecture of the Topspin 270 Cisco SFS 7008 offers flexibility in that the switch fabric modules are generic and will assume the role of fabric controller or fabric node depending on the slot into which they are installed This increases the ease of replacing components because only one type of fabric card has to be stocked and will become a controller or node depending on what is needed Hi Availability Hardware The To
19. State Indication Green off Management module is not connected to the active master controller or is connected to the system s standby master controller Green solid on Management module is connected to the system s active master controller Ethernet Management Port RJ 45 connector LEDs Refer to Figure 5 5 to view the location of this LED Table 5 12 Interpreting the Ethernet Management Port LEDs see Figure 5 6 Location State Indication Left port LED off Port is not connected or logical link is down Left port LED on Port is connected and logical link is up Right port LED off No traffic is moving on the Ethernet Management port link Logical link may or may not be Up Right port LED on Traffic is moving on the Ethernet Management port link Chassis ID LEDs Figure 5 7 Chassis ID LEDs The Chassis ID Status LED indicates the status of this module Table 5 13 Interpreting the Chassis ID Status LEDs see Figure 5 7 77 LIM Status LEDs Location Color State Indication Green Yellow off No module power or LED failure Green solid on Chassis ID module is running with no errors detected Green off No power to the Chassis ID module LED failure or yellow LED is On Yellow solid on Failure of the Chassis ID module Operator attention required The label indicates a failure Yellow blinking Identify a Chassis ID module Assists in identifying a particul
20. TOPSPIN or 650 316 3300 Fax 650 316 3269 Telephone Technical Support 800 499 1473 Email Technical Support support topspin com Web site Support http support topspin com 11 Features Overview The Topspin 270 Cisco SFS 7008 Server Switch provides data center managers with a high performance low latency interconnect e System Architecture on page 11 e About the Topspin 270 Cisco SFS 7008 Chassis on page 13 e Administrative Features on page 20 System Architecture Hot Standby On the Topspin 270 Cisco SFS 7008 the inactive controller card remains on hot standby so that it can assume control in the event that the active controller card fails over The hot standby feature lets the Server Switch fail over without rebooting or resetting any cards in the chassis You can access the hot standby controller card with a connection to the serial console port A CLI session to the standby controller provides read only access to the controller You can view but not configure the card Because the hot standby features keeps both controller cards up the show card command identifies the card on which your CLI session runs In the command output an asterisk identifies the controller card to which you are connected In the example that follows the CLI session applies to the controller card in slot 11 the active card as identified by the asterisk next to the slot number line 10 12
21. Topspin 270 Cisco SFS 7008 has port to port latency of less than 600ns Non Volatile Memory The memory supports up to e three stored system images not including recovery image e one week of log files at normal verbosity and one day of log files at maximum verbosity Vital Product Data Storage Vital Product Data VPD is stored in non volatile memory in the Chassis ID module refer to Chassis ID Module on page 19 and the Fabric Controller that is installed in the Core slot refer to Fabric Controller Modules on page 18 and is available electronically Some VPD info can be accessed via CLI the Java GUI or the Web GUI The following Vital Product Data can be recovered by returning a Field Replaceable Unit to the original manufacturer Note This requires the appropriate support contract Power on hours Manufacturing part number Serial number e Final test date Card ID e Failure code e Failure date e Operation status e Failure log OEM part number Diagnostics Refer to About Diagnostic Tests on page 91 for more detailed information The following tests are run to determine operational status Power On Self Test POST is performed on all system components during power on to determine operational readiness e Redundant components operational status is ensured periodically during normal operation including the logic required to perform the transition from faulted primary to redundant co
22. chassis 4 Remove the bezel cover from the front of the switch Locate the power supply that you want to remove Refer to Power Supply LEDs on page 69 If you have two power supply modules installed you can remove one of them without removing power from the chassis However before removing a power supply make sure that the redundant power supply has the corresponding power cord installed Each power supply is supported by a the power cord that is directly behind it on the chassis 6 Determine the power cord that is linked to the power supply module you want to remove The power cord and its corresponding power supply are situated in tandem on the same half of the chassis CAUTION Make sure you do not remove the power cord of the functioning power supply 37 7 Pull the power cord from the chassis Power pai i Power Supply front oooooo nunnan DODODO OOOoOo00 a DODODO DODODO 90 900 ooooo0 DODODO OoOoooo oooooo oooog0o E EEE EEE oooooo nanana oooooo mannna nannan DODODO numana DODODO nanana DODODO A DODODO nnmnnn DODODO nnmnnn DODODO on ooooog ooooog BEEBE DODODO DODODO DODODO DODODO o 0 o 0 H o o Figure 3 1 Diagram of Power Supply to the Associated Power Cord 8 Use a l phillips screwdriver to unscrew the fasteners that hold the power supply module in place Fi
23. chip speed fabric12x Management Interface modules mgmt O e Chassis ID modules idModule Note the Current Status column which should read up The current status refers to the Up or Down state of the card Note the Operational State column The possible states are as follows unknown 82 normal bootFailed tooHot booting checkingBootlmage wrongBootImage rebooting standby recoverylmage 9 Click the radio button next to the card that you want to monitor 10 Click the Properties button to view details about the card or to reset the card Reset a card to return it to the default state The cards cannot be shut down because they are architected so that they are always in service once they are inserted into the chassis Monitor the InfiniBand Ports 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Log into the chassis The Device Rear View and the Device Front View appear 4 In the left navigation tree select the Chassis gt Cards The list of cards appear 5 Click the radio button next to the LIM card whose ports you want to view 6 Inthe drop down menu above the card list select Show Ports The list of IB ports for the selected LIM card will be displayed The Ports view appears X 10 3 108 4 Ports y Chassis 10 3 108 4 gt Chassis gt Ports Y Cards Ports Filter Card 5 a Sensors 3 Backplane a a OTE
24. engages fully Wiggle your connector back and forth to be sure that both sides of the connector have locked firmly into place Connecting to a Router or Switch To connect directly to an Ethernet switch router or a Fibre Channel switch you must add a Topspin device with the optional Ethernet or Fibre Channel gateway such as the Topspin 90 or Topspin 360 to your fabric Connecting Multiple IB Switches Two InfiniBand systems such as a Topspin 120 and a Topspin 270 Cisco SFS 7008 can be connected by an InfiniBand cable inter switch link ISL System You can manage the InfiniBand server switch system using the following methods 32 SFS 270 gt enable SFS 270 configure e Command Line Interface CLI a text based interface accessible through a direct serial connection Telnet over IP or SSH over IP e Chassis Manager GUI A web based graphic user interface Element Manager GUI An optionally available graphic interface installed on a workstation and accessible over IP Refer to the InfiniBand User Guide the CLI Reference Guide or the Chassis Manager User Guide for more information about managing the InfiniBand systems Optional Launch the Chassis Manager Chassis Manager is a web based GUI that can be used to manage a single InfiniBand chassis Prepare Your Device To launch Chassis Manager on your Topspin 270 Cisco SFS 7008 you must e Configure an IP address on the Ethernet management port if you h
25. is intact 2 Removing the fabric controller in the core slot that is currently acting as master will cause a fail over to the standby pair of fabric controller Management Interface modules It takes up to 3 seconds for the standby fabric controller to become a new master a Ifyou are removing a fabric controller card from a node slot the card will be located in slots 9 10 13 and 14 marked on the chassis b Ifyou are removing a fabric controller card from a core slot the card will be located in slots 11 or 12 Removing a Fabric Controller 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 3 Remove the bezel cover from the front of the switch 4 Determine whether the Fabric Controller you want to remove is installed in a Core slot or a Node slot Refer to Figure 1 1 for a diagram of the chassis If you are removing a Fabric Controller from one of the core slots make sure that the redundant core Fabric Controller is functional Removal of the Fabric Controller currently acting as master will cause a fail over It takes up to 3 seconds for the standby fabric controller to become a new master 43 5 Usea l phillips screwdriver to unscrew the captive screws from both ends of the Fabric Controller card Figure 3 9 Loosen Captive Screws Before R
26. off solid on Physical link OK No logical link established solid on solid on Physical link OK Logical link established blinking solid on Physical and logical link OK Traffic active Monitoring the System with Chassis Manager For information regarding launching the Chassis Manager GUI refer to the Chassis Manager User Guide Monitor the Device 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser 2 Press the Enter key 79 3 The Device Rear View and the Device Front View appear SS TOPSPIN IP Address tem Name X 10 3 108 4 Device View 4 Chassis 10 3 108 4 gt Device View A Cards 9 Ports Power Supplies Fans Device Rear View Sensors ee we eee eee eee eee eee eee eee eee eae 3 Backplane ae SEE R 3 Management Ports Enet sia A Maintenance InfiniBand 0 Help Bl Figure 5 10 Chassis Manager Topspin 270 Cisco SFS 7008 Device Rear View SS TOPSPIN stem Name x 10 3 108 4 nunnan 34 Chassis old S a mii sh Ors pee 3 Power Supplies A Fans 5 Sensors Device Front View A Backplane Management Ports 4 Maintenance 2 InfiniBand 0 Help Figure 5 11 Chassis Manager Topspin 270 Cisco SFS 7008 Device Front View 4 View the information at the top of the window to quickly check the sta
27. percentage completed 100 result string Rack Locator Identify Final report PASSED Displaying Hardware Errors To display POST results non fatal errors that can are recovered informational logging such as firmware updates and hardware errors SES 2 70 gt SES 2 705 Thu Thu Thu TENU Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu Thu lt ou 1 Use the more syslog hwif_log command to locate the topspin log hwif_log Example of output without errors enable more syslog hwif_log 4 10 57 49 2004 POST SEEBROM PASSED 4 10 57 49 2004 POST FPGA PASSED A OSA OO ADO SES UVM ED AS SBD A 10S3713390 2004 BOSTS ERTES PRAS SED AO OMA 200 POST FAN PASSED 4 10 57 54 200 Carcel Steuecwyo gt x 3 Care 1S Sitawciiag UN 4 10 57 54 2004 Anafa2Init a2update set to IGNORE 4 10 58 16 2004 Anafa2 POST firmware check PASSED 419201255 2004 POST SEEPROM PASSED 4 iSsOless 2004s POSS PEGAs PASSIaD 4 AUGsOilsgss ZOO4s POSWs SUMMING LEAS SED 4 iSs iles6 20043 POSTS RUCS PASSED 4 19 02 00 2004 POST FAN PASSED A 19502300 2004 Caro startup x 3 Caro 1S Sirawciling UN 4 19 02 00 2004 Anafa2Init a2update set to IGNORE truncated gt 102 You can also view POST error codes with the show diagnostic post command This command displays the results of each POS
28. 000000000000000000000000 EIS EES Ei o 1001 0 0 Dl a c t o i i i E o o e e e le o ol 0000000000000000 ooo ooo ooo ooo or DUDO DO OOO DODODODO 000000 000000 DODOODODO OOO0ODO0O0DODODOODODOOOODODOOOOOODOO ODOODODOODOO OODOODO OOODODODOODODOO DDODOOOODODOOODOOODODOOODODDOD DODODODOOD ODO DOODOODOO DOODOODODODODODO Figure 1 4 Chassis Slot Numbering in Back Service Side Dimensions Height 6U unit Width 17 04 Depth 8 23 7 Weight maximum 106 pounds full chassis configuration with 4x copper Line Interface Modules Connections e 96 ports of 10 Gbps 4X Copper InfiniBand e Two 10 100 Ethernet RJ 45 Management Ethernet ports for out of band management Two RJ 45 Console Port used to configure and monitor the Topspin 270 Cisco SFS 7008 e Rack Alarm All cabling is on the rear of the chassis LEDs LEDs are located on both the front and back of the chassis Refer to Interpreting the Front Panel LEDs on page 67 and Interpreting the Back Panel LEDs on page 73 for more information Power Supplies Refer to Installing a New Power Supply Module on page 38 for installation instructions The Topspin 270 Cisco SFS 7008 has two AC DC bulk power supply modules e Only one power supply in either of the two slots is required to power the system However a blanking panel is required for any unused bay 17 The power supplies are 1U 48V 1200
29. 004 POST FAN PASSED Tai Mei 4 108578354 20048 Carci Staiccwjo Carce 1S Sesame us Thu Mar 4 10 57 54 2004 Anafa2Init a2update set to IGNORE Thu Mar 4 10 58 16 2004 Anafa2 POST firmware check PASSED Mo Meese 4 iIOeOilsss 2004s BOSS Sigg PROM IVS SAID Moi Mes 4 L9s Oilesas 2OO4s ROSE WEA S PASSED Wa Mew A ilOegOiless 2004s WOSWs SUMINIWS BRASO ED Hag Mene 4 19 01 56 20048 POSTS IRWCS ASS Wee Weve A l9gO2Z300 2ZOO4s BOS WANS PASSED Tau Mei 4 192902800 20048 Carci suicidas Carco 1S sirsucicima us Thu Mar 4 19 02 00 2004 Anafa2Init a2update set to IGNORE Thu Mar 4 19 03 56 2004 POST SEEPROM PASSED Wee SOSA OA BOS EE GA IPAS SE Thu Mar 4 19 0S337 20043 POSTS SUMMONS LYAS SIRI mar Maese 4 1920323938 200483 ROSWs IRNOS PRASSiaD Thu Mar 4 19 04 02 2004 POST FAN PASSED Tow Mare 4 19204202 2004 Care suscita Card 1S suecia Us Thu Mar 4 19 04 02 2004 Anafa2Init a2update set to IGNORE Thu Mar 4 19 04 23 2004 Anafa2 POST firmware check PASSED Thu Mar AS Oks AO LOSA SEE PROME ERAS S ED Thu Mar 419207707 2004 POST FPGA PASSED Thu Mar A 193078207 20048 POSWs SUMMOMPE YAS SIRID Wag Mew 4 1892072083 20048 ROSWs IRWCS PASSiaID Thu Mar AES AON see OA ae PO SEAN PALS SED Thu Mar 19 0751272 2004 Caro srenteg x Card 15 Sicaicciing va Thu Mar 4 19 07 12 2004 Anafa2Init a2update set to IGNORE Thu Mar 4 19 07 34 2004 Anafa2 POST firmware check PASSED mest Wieue Sy IOsilGeais ZOO4s ROSS SEE ROME SS Ein mica Wi
30. 3 21 Remove the Chassis ID Module 5 Do not allow the bay to remain empty as overheating may occur The Topspin 270 Cisco SFS 7008 should never be run without the replacement module so you should immediately install the new Chassis ID module Refer to Installing a Replacement Chassis ID Module on page 52 52 Installing a Replacement Chassis ID Module 1 Remove the new Chassis ID module from the package 2 Ground yourself appropriately 3 Remove the old Chassis ID module if you have not already done so Refer to Removing a Chassis ID Module on page 50 if a Chassis ID module is still in place 4 Align the chassis ID module with the slot and push the module completely into the bay jpg E te 5 Use the LEDs to verify the status of the module Refer to Chassis ID LEDs on page 76 The new Chassis ID has been previously serialized with the information that is unique to your chassis 6 Reboot the chassis by using the GUI or by using the reload command Example SFS 270 reload System configuration has been modified Save ves ceraulic inoe cirg yes Proceed with reload confirm SFS 270 Cone pho hm Om LOS O sien 53 Administrative Tasks Upgrade all The following sections appear in this chapter Upgrade all Cards in a Chassis on page 53 Upgrading an Individual Controller Card on page 56 Upgrading Individual Node Cards on page 59 Upgrading a New Fabric
31. 5 Removing a Power Supply Module ninia 36 Installing a New Power Supply Module a id 38 By Ura A a os sda e ve ees ec as states Wo aa n n Slane ase 39 Aboot the Fan Tray Mods sd 39 Removmea Fan Md reacia 39 Installing a Parr Mod ler cic A Ave sues AA A Sada Ara aes ea a Ave 41 Fabric Controller AS AA N ona a ted ea 42 About the Fabric Controllers dacs cred ait ohisec at sauecay vhedauncent areca pioneateav shen mae nesiea aes 42 Removing a Fabric Controller iss casstehsiosncs october sn dd 42 Installing a Fabric Control telnet acon tacts ws e Atte 44 Upgrading the Controller Cal as 44 Line Interface Management Modules LIM as da E 45 About LIM Cars A a a ad tt 45 Removing a LIM Cal dis 45 stas LN EM a a a a Udala 48 Management Interface Mo a e e e e a icy 49 About Management Interface Modules ses ibi 49 Removing a Management Interface Modul8 ooooncccinccnocccicocononconncconocnnnconncnonocnnn cono nooo noc nocnnncnnncns 49 Installing a Management Interface Module il dd sd Nata 50 Chassis BED IN US 6 ie cae alas olaaa vs Bod gee va Os OUR DOE 50 Abo t Chassis ID Modules ri oe ag ies O 50 Removing a Chassis ID Module dl obli 50 Installing a Replacement Chassis ID Module cecceeceecceeseceseceseceeeeeeseecnaeceseeeeeeenseecseeeneenseees 52 4 Administrative TasSkS cccccecceceececeees 53 Upgradeall Cards in a E ad o a o E 53 Upgrading a Topspin 270 Cisco SFS 7008 with the CLI ec eeeeccecsecsteceeeceeeecsseeeteeseeeens
32. C Rev 4 129 pvr 4012 0481 ae 0004031fe8 i Yet 58 2 Show the standby Controller card Example SFS 270 show card inventory Card Resource Inventory Information slor rel iL ik used memory 18092 kbytes free memory 109676 kbytes used disk space 30891 kbytes free disk space 71488 kbytes IASE LMACa SOuIceS 8 WojosjoimOS 2 2 0 lowricls3i pramana magen S Cur CeO Pn OSa A 200 05 LS 37 image TopspinOS 2 2 0 build531 cpu descr PPC 440GP Rev C Rev 4 129 pvr 4012 0481 fpga firmware rev ae ib firmware rev 0004031fe8 slow tel LZ used memory 16017 kbytes free memory 107432 kbytes used disk space 30891 kbytes free disk space 71488 kbytes lasi mesea sobiscs 3 Tossoliaos 2 2 0 1 Lo487 primary image source TopspinOS 2 2 0 bui1d487 image TopspinOS 2 2 0 build487 cpu descr PPC 440GP Rev C Rev 4 129 pvr 4012 0481 fpga firmware rev ae ib firmware rev 0004031fe8 3 Check to see if you already have a img file that can be installed Delete the img file if it not the one you need to install Use the delete s ot number limage file name command Example SFS 270 dir image Existing Boot Images on System slot date created size file name 113 Taw Jam 27 01 07 36 2005 10657935 OS SENO AMOS OSO O ASS ii mau Jam 27 OlgZIsO2 2003 CGEA MepsoimMOS 2 52 0 ow les 3
33. E DODODO rt ODO DoOO Doo ooDoDooooooo EIEEE EEEE occoooocooonoococcoconcooononoroooonoooooooonoonoooonoooon0n0o0n 2 EEE TS EE Sd BODOODODODODO OOO OO OOOO OOOO OOO OOO OOOO DO DO OD ODO DODOOo0oo0o Ae SS SS SS aa BODODOODODODO OO DO OOO OOOO OO 000 0000000000000 000000000000 ide Ss eS Ss pDODOD oo0000 D i be Sonunu aM OOOOOO0O00O0O0 000000 00000 DODODODOODOODOODODODODOO DOOOODOCDOODOOOODOODODODODO DOODODODOODODODODODODOO DODODODODODOODODODDODO ODODOODOODODODOODODODOO o o o o Figure 2 8 Power Jack Locations 7 Plug the other end of each AC power cable into a 90 264 VAC power outlet operating at 47 63Hz Attach the chassis only to approved power sources as indicated by the electrical ratings label If you are unsure of the correct power source to use contact your support personnel or your local power company The chassis automatically starts and boots You can watch the running status via the serial console Use the correct external power source 8 Check the LEDs on the front of the Topspin 270 Cisco SFS 7008 system When the system first powers up it performs a power on self test POST Refer to System Wide Status LED on page 68 You will know that the system has completed the boot process when you see a login prompt
34. ED Please reboot syst em SFS 270 100 5 Use the show diag fru error command to display any errors that appear in the result string Example SFS 270 show diag fru error ie Ul ETEO tios Lote SIRO seuceal 9 none cara 10 none excel iL i none censal 112 none carre LS none card 14 none Oangel La none csucal Ly none fan 1 none fan 2 none item 3 none fan 4 none power supply 1 none power supply 2 none SFS 270 6 Reboot the chassis when the self test is complete Example SFS 270 reload System configuration is modified Save yes default no filename y Proceed with reload yes default no y Run a Rack Locator Test The rack locator test causes the LEDs on the tested chassis to blink for the duration of the test To run a rack locator test perform the following steps 1 Start the test The LEDs of the Server Switch begin to blink Example TS270 1 gt enable TS270 1 configure terminal TS270 1 config diagnostic rack locator TS270 1 config diag rack locator 1 start 2 Stop the test The LEDs return to normal functions Example TS270 1 config diag rack locator 1 stop 3 View 101 the results of the test Example SFS 270 gt show diag rack locator Diagnostic Tests For Rack Locator module type rack locator module number 1 test led iterations 1 AGIELOM 8 STOD result success
35. Green off No power to the fan tray LED failure or yellow LED is On Yellow solid on Failure of the fan tray Operator attention required The label indicates a failure Yellow blinking Identify a fan tray module Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example For example to run the identity test on Power Supply 1 config mode gt diag fan 1 gt test led gt gt start start flashing gt stop stop flashing 76 Table 5 10 Interpreting the Power Supply Status LEDs see Figure 5 6 Color State Indication Green over Yellow loff No fan module installed or LED failure Green solid on Power supply is running with no errors detected Green off No power to the Power supply LED failure or yellow LED is On Yellow solid on Failure of the Power Supply Operator attention required The label indicates a failure Master Management Connection Status LED The Master Management Connection Status LED is a single LED that indicates whether that particular Management Interface module is connected to the system s active master controller Refer to Figure 5 5 to view the location of this LED Table 5 11 Interpreting the Master Management Connection LEDs see Figure 5 6 Color
36. Prepare the Physical Environment for the Topspin 270 Cisco SES 7008 System ccee 21 Mount the Topspin 270 Cisco SFS 7008 Chassis in a Rack ceececcceceseceseceeeeeeseecseeeeeeeseecsuecneeseeeensees 22 REQUIEM IA A E A ddan alae ise hed eos 22 CAC Volt Ta AAN a A e RAR E ag A EEA 22 Install Optional HA Components ca e sestinetsvca ote ha a 26 iii Connect Management IICA A tied Aue bcos A eo eas ana Aas cis 26 Attach a Serial Console Cable to a PC or Terminal ooooonnccinocinococonoconnnonnnonnncconocono conc cconocnnncnnnnnos 26 Connect an Ethernet Cable to the Ethernet Management Port ooooccnnccciocanoccnoccconccconoconnconnnconccnno 27 Recommended Management Cono ura A co Seuss ait as alec esetpiadave nai 27 Power on the Chassis rosera ae e acs ety a a ev age wis esos a a AE EAEE 28 Configure Basic COmMent A a 29 Assigning an Address ii AAA A AAA ENE 29 Assigning an Address Dynamic iii ii a dd A it 30 Connect Network DEVICES a a gst 30 About COME E O e h e e a EE neseneceet eas 30 Connecting InfiniBand Devices ties dd 00 31 Manage the Systemi it das 32 Optional Launch the Chassis Manager ccscccsccessecesseeeseceeceseecsaeceseeeeeeeeseecsaeceeeeeeeenseeeaeens 32 Optional Install the Element Manager CUL ai iia 34 3 Managing Individual Components 35 About the Field Replaceable A EE Os 35 Power Supply MOUSE suet vane Hatin acta a i a E RA RR a 35 About the Powersupply Modules lt a 3
37. SI TOPSPIN Topspin 270 Cisco SFS 7008 Hardware Guide Release 2 3 0 Part Number 10 00044 04 A0 Copyright 2004 2005 Topspin Communications Inc All rights reserved The Topspin Switched Computing System Topspin Host Channel Adapter Topspin Element Manager and collateral software programs and documentation are subject to and made available only pursuant to the license agreement signed by you and Topspin Communications Inc or if no signed license agreement exists the license agreement included with your original media and may only be used in accordance with the terms of that agreement Making copies modifications or compilation works of the software except as specifically allowed in that agreement is prohibited by law and constitutes a punishable violation of the law This documentation may not be copied modified or reduced to any electronic or machine readable form without Topspin Communication Inc s prior written consent As defined in FAR section 2 101 DFAR section 252 227 7014 a 1 and DFAR section 252 227 7014 a 5 or otherwise the collateral software programs provided to you are commercial items commercial computer software and or commercial computer software documentation Consistent with DFAR section 227 7202 FAR section 12 212 and other sections any use modification reproduction release performance display disclosure or distribution thereof by or for the U S Government shall be governed sol
38. T that executed at power on Example SFS 270 show diagnostic post Post Status n SL Oe post status error codes card 1 failed Ll 203030 fan 1 passed 003050 fan 2 passed 003070 power supply 1 passed OROROORO To view FRU errors on your Topspin 270 Cisco SFS 7008 enter the show diagnostic fru error command Example SFS 270 show diagnostic fru error PIG ESE OT Licu S Lot LiPU Sieie ie sauce Ly FRU POWER CONTROLLER CONFIG ERR fan 1 none fan 2 none power supply 1 none The following types of hardware errors are logged for the Topspin 270 Cisco SFS 7008 power supply Table 6 1 Topspin 270 Cisco SFS 7008 Power Supply Errors Error Number Description 1 Power Supply DC Fail Error 2 Power Supply Fan Fail Error 3 Power Supply I2C Communication Error Specifications and Compliance Certifications This chapter details the Topspin 270 Cisco SFS 7008 specifications and compliance certifications Chassis and Management Interface Table 7 1 Chassis and Management General Specifications 105 Operating Temperature Ranges 0 to 40C Operating Temperature 0 to 42C Non Operating Temperature 40 to 70C Operating Temperature Gradient 20C max per 60 minutes Operating Altitude 0 to 10 000 ft Non Operating Altitude 0 to 40 000 ft Operating Humidity 8 to 80 non condensing Non Operating Humidity
39. W The supplies provide regulated 48V DC to all other modules in the system Each power supply has self contained fans for cooling Fan Trays Refer to Installing a Fan Module on page 41 for installation instructions The fan trays are redundant hot swappable cooling units Only one fan tray in either of the two slots is required to cool the system However a blanking panel is required for any unused bay e The replacement of any one fan tray does not disrupt the operation of the device in any way and can be successfully completed without having to remove the device from a rack or disconnect any cables The fan modules use DC Voltage regulators to control fan speed and contain a hardware fan control circuit which controls and monitors the fan speed The fan module will default to full speed after a power on or reset condition e If one the fans fails the redundant fan will go to full speed if one is available A CAUTION The fan modules do not have guards Do not remove a fan module completely from the bay until the blades have stopped moving approximately 15 seconds Management Interface Modules See Figure 1 2 on page 14 for a diagram of the chassis Refer to Installing a Management Interface Module on page 50 for installation instructions The Topspin 270 Cisco SFS 7008 supports redundant hot swappable Management Interface modules each of which is paired to one of the Fabric Controller Core m
40. able protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the instruction manual may cause harmful interference to radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case the user will be required to correct the interference at his own expense Safety Information e Electrical Cautions on page vii e General Cautions on page ix Electrical Cautions A CAUTION Use only the power cable provided with your Topspin system Inspect the power cord and determine if it provides the proper plug and is appropriately certified for use with your electrical system Discard the cord if it is inappropriate for your country s electrical system and obtain the proper cord as required by your national electrical codes or ordinances Vill A CAUTION Grounding is supplied by the ground prong on the 3 prong power cable Do not attach a separate ground cable Do not use adapter plugs Do not remove the ground prong from the cable Ensure the ground connection on the power supply is correct and functioning before applying power to the Topspin system CAUTION Always ground yourself before touching any internal Topspin system component to avoid damage from electrostatic discharge ESD A CAUTION The Topspi
41. ade a node card using the CLI refer to the CLI Reference Guide 60 Upgrading a Node Card with Element Manager The following procedure describes the upgrade procedure for individual node cards using the Element Manager GUI This procedure assumes you are upgrading only one or more node cards and not the Controller cards Ls 2 4 Launch Element Manager for the server switch Delete any old img files to make room for the new image a Select Maintenance gt File Management b Click on a img file if one exists and click the Delete button c Repeat for all Server Switches you wish to upgrade Copy the new image to the Server Switch a Select Maintenance gt File Management b Click the Import button c File Type mage a Select Remote FTP Server or Local File to navigate to the file Import File Copy a file to the system File Type image v Copy From Remote FTP Server Local File salfile Topspin270 TopspinOS 2 2 0 build531 img Copy To File Name on System Topspin2 70 TopspinOS 2 2 0 build531 img Figure 4 5 Copy the img File to Server Switch e Click the Copy button and wait for the message that indicates the copy was successful f Click Ok Install the new image Note all node cards in the must have an oper status of up for the upgrade to be effective a Select Maintenance gt File Management b Click the new img file and click the Install b
42. ar Field Replaceable Unit on the chassis This state is initiated by the user Refer to the Command Line Reference Guide for more information The LIM Status LED is located at the far left of each LIM card if you are facing the back of the chassis Refer to Figure 5 5 for a complete diagram of the back panel of the chassis LIM Status Indicator Figure 5 8 LIM Status Indicator LEDs Interpreting the LIM Status LEDs Table 5 14 Interpreting the LIM Status LED Color State Indication Green and off No module power or LED failure Yellow Green solid on Line Interface module is running with no errors detected Green off No power to the Line Interface module LED failure or yellow LED is On Yellow solid on Failure of the Line Interface module Operator attention required The label indicates a failure InfiniBand Port LEDs The InfiniBand LED represents the logical link and the logical link activity This LED can be initiated manually by using the diag LED test 78 Location The InfiniBand port LEDs are located to the right of each InfiniBand port if you are facing the LIM card in the back of the chassis SS So oo Fo Fo oe IB Port LEDs Figure 5 9 InfiniBand Port LEDs Interpreting the InfiniBand Port LEDs Table 5 15 Interpreting the InfiniBand Port Status LED Top tx rx LED Bottom link LED Indication off off No link signal detected
43. ar View and the Device Front View appear 4 View the Power Supply status at the top of the window to quickly check the status color and the number of modules installed If only one number appears then only one power supply is installed The actual integer refers to the slot in which the power supply module is installed A field that read Power Supplies 1 2 would indicate that a power supply was installed in both possible slots The color of the integer indicates the status of the power supply A green number indicates that the power supply is Up A red number indicates that the power supply has failed 5 Click on Power Supplies at the top of the window to view details 84 The Power Supplies view appears 5 TOPSPIN X 10 3 108 4 Power Supplies a Chassis 10 3 108 4 gt Health gt Power Supplies amp Cards 3 Ports S Power Supplies 9 Fans Ps iD Type Current Status JUtiization Pepa 2 ac up 0 48 y Backplane a Management Ports e Maintenance InfiniBand O Help Data Refreshed At Friday April 23 2004 1 22 11 PM Figure 5 15 Chassis Manager Power Supplies Window Monitor the Fans Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser Log into the chassis The Device Rear View and the Device Front View appear View the Fan status at the top of the window to quickly
44. as hot swappable Line Interface Modules LIMs which are installed in the rear of the chassis this is opposite of the fabric modules see Figure 1 2 on page 14 The LIMs provide the physical InfiniBand port interfaces and are 4x over copper The number of LIMs in your Topspin 270 Cisco SFS 7008 chassis depends on your particular configuration Chassis ID Module The chassis ID module is hot swappable and is located on the rear of the chassis see Figure 1 2 on page 14 Figure 1 6 Chassis ID Module How the Chassis ID is Used Upon initial boot up of the Topspin 270 Cisco SFS 7008 the Chassis ID is read by the system and the unique information is populated to the rest of the chassis The Chassis ID module contains model and serial number information and logs certain error information Refer to Vital Product Data Storage on page 20 About Replacing a Chassis ID Module In the event that a Chassis ID module must be replaced the replacement module will arrive with your previous serial number and GUID When a Chassis ID modules is installed the new Chassis ID is detected This allows your chassis to maintain the same identification and GUID Rack Alarm The Rack Alarm also referred to as the Rack Locator feature is supported with release 2 3 0 and later 20 Administrative Features Real Time Clock The Topspin 270 Cisco SFS 7008 maintains correct time regardless of power conditions or connectivity Latency The
45. ated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run identity test on Power Supply 1 config mode gt diag power supply 1 gt test led gt gt start start flashing gt stop stop flashing Power Supply LEDs Locating the Power Supply LEDs The Power Supply LEDs are located in the outside of each Power Supply module if you are facing the front of the chassis with the bezel cover removed Figure 5 2 Power Supply LEDs 70 In addition a Power Supply Identify LED is located on the outside of the chassis next to the corresponding power supply This LED allows the user to initiate a command that causes the Identify LED to blink which assists in locating a particular Field Replaceable Unit Figure 5 3 Power Supply Identify LED Replicas of the Power Supply LEDs are positioned on the back of the chassis Refer to Management Interface Module LEDs on page 73 Interpreting the Power Supply LEDs Table 5 2 Interpreting the Power Supply LEDs see Figure 5 2 Color State Indication Green and off No power to the power supply or LED failure Yellow Green solid on Power Supply running with no detected errors Green off No power to the power supply LED failure or DC output failure Yellow off Power Supply running with no detected errors Yellow solid on Operator intervention required Failure detected
46. ation on how to connect the serial console cable please see the documentation included with the serial cable kit 2 Open a terminal emulation window using a program such as HyperTerminal for Windows Set your terminal parameters to the following 21 Baud 9600 bps Data Bits 8 e Parity None e Stop Bits 1 e Flow control None Connect an Ethernet Cable to the Ethernet Management Port 3 Connect an Ethernet cable to the Ethernet port of the left Management Interface module slot number 15 See Figure 2 7 a Ifyou have two Management Interface modules you may also want to connect an Ethernet cable to the standby module slot 16 to maintain Ethernet connectivity in the event of a fail over Recommended Management Configuration The diagram below shows the recommended management port connections To have constant access to the chassis remotely redundantly connect the serial ports on two Management Interface cards to a terminal server l Remote Connetion Terminal Server 0000000000000 oo000g0gn0D00000000000b00000 2382499005 Aa Ave a a FTODDOODODODODO DO OOOO ODOODOOOODOODODO A Ethernet Switch f EEEE Connect left module first oooooooooooooooooooooooooooooooooooo00000O0 o CS EES ES i OoO000D0000000000000000000000000000000000000 o Bee 000000000000000000000000000000000000000000 O t he sh RII SI SI I Sk Figure 2 7 Recommended Management Configuration
47. ave not already done so e Configure an IP gateway on the Ethernet management port if you have not already done so e Optional Enable Telnet services should be enabled by default e Optional Enable HTTP or HTTPs services HTTP should be enabled by default If you followed the previous steps you should already have these items configured If your device meets these requirements proceed to Launch Chassis Manager on page 32 Otherwise to prepare your device perform the following steps NOTE Consult your network administrator for an IP address subnet mask and gateway address before you begin this process 1 Ifyou have not yet configured an IP address for the Management port refer to Configure Basic Connectivity on page 29 2 Enter the telnet enable command in global config mode to start telnet services on your device You may need to use the exit command to return to global config mode Example SFS 270 config telnet enable 3 Enter the ip http server command to enable HTTP services on your device HTTPs is also supported Example SFS 270 config ip http server Launch Chassis Manager To launch Chassis Manager perform the following steps 1 Launch your Web browser Chassis Manager supports the following browsers e Microsoft Internet Explorer version 6 e Netscape Navigator version 6 33 e Mozilla version 1 4 2 Type the Management IP address of your Server Switch in the address field of yo
48. blink the LED and identify a specific power supply Example SFS 270 gt enable SFS 270 config SFS 270 config diag power supply 1 SFS 270 config diag power supply 1 test led SFS 270 config diag power supply 1 start SFS 270 config diag power supply 1 stop SFS 270 config diag power supply 1 99 Run a Self Test on a Power Supply 1 Show the ID of the power supply that you want to diagnose if you do not already know the number a Enter the show power supply command Example SFS 270 show power supply Power supply Information ps type admin status oper status WE sLIL aL Src LOA voltage 1 AC up up n a 48 2 AC up up tS 47 2 Enter the diag power supply lt id gt command and specify the type of test self test is chosen by default Example SFS 270 gt enable SFS 270 config SFS 270 config diag power supply 2 SFS 270 config diag power supply 2 test self test 3 Start the test Example SFS 270 config diag power supply 2 start 4 Use the show diag fan command to show the progress of the test Example SFS 270 config diag power supply 2 exit SFS 270 config exit SFS 270 show diag power supply Diagnostic Tests For Power Supply module type power supply module number 2 test self test iterations 1 action 3 Start result success percentage completed 100 result string Self Test Final report PASS
49. check the status color and the number of modules installed Ifonly one number appears then only one fan is installed The actual integer refers to the slot in which the fan module is installed A field that read Fans 1 2 would indicate that a fan trays are installed in both slots e The color of the integer indicates the status of the fan A green number indicates that the fan is Up A red number indicates that a fan has failed Click on the word Fans at the top of the window to view details The Fans view appears IP Address SS TOPSPIN System Name X 10 3 108 4 Fans a4 Chassis 10 3 108 4 gt Health gt Fans amp Cards 9 Ports y Power Supplies D Bs Current Status Speed E ensors 1 up 74 y Backplane 14 Management Ports 2 BB 70 a Maintenance a4 InfiniBand Data Refreshed At Friday April 23 2004 1 18 08 PM 4 0 Help Figure 5 16 Chassis Manager Fans Window x 10 3 108 4 a Chassis amp Cards Ports Power Supplies B Fans a Sensors y Backplane Management Ports HS Maintenance InfiniBand Help 85 Monitor the Temperature 1 Launch the Chassis Manager by entering the IP address of the InfiniBand switch into your web browser Press the Enter key The Device Rear View and the Device Front View appear Click open the Chassis file in the left navigation tree The individual components appear beneath the Chassis men
50. ck the InfiniBand port LEDs to determine the status of the port Refer to InfiniBand Port LEDs on page 77 Management Interface Modules About Management Interface Modules Refer to Management Interface Modules on page 17 for additional information About a Failed Management Interface Module If you believe a Management Interface module has failed contact technical support Management Interface modules are hot swappable Refer to Management Interface Module LEDs on page 73 and Monitor the Cards on page 81 for monitoring information Determining Which Management Interface Module has Failed The Management Interface modules are located on the back of the chassis Refer to Figure 1 2 fora diagram of the chassis The slots are labelled on the chassis Removing a Management Interface Module 1 Verify that you have a method of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 2 Determine if the Management Interface module that you are removing is currently connected to the master fabric controller Note The removal of the Master Management Interface module will trigger a fail over of the Fabric Controller module Switch traffic will be disrupted If the system is left without a Management Interface card the switch will become inoperative If the Master LED is lit then the module is connected
51. configuration to preserve it between reboots SFS 270 config if mgmt ethernet exit SFS 270 config exit SFS 270 copy running config startup config 30 Assigning an Address Dynamically a When the system has completed booting press lt Enter gt several times to display the CLI prompt Logia e b Enter the user name and password The default user name is super and the default password is super Login super Password super SRESER 10 gt c Enter the enable command SFS 270 gt enable SFS 270 d Enter the configure command SFS 270 configure SFS 270 config e Enter the interface mgmt ethernet command SFS 270 config interface mgmt ethernet f Enter the option dhep command The chosen method will be persistent across module reboots SFS 270 config if mgmt ethernet option dhcp g Enable the management port by entering the no shutdown command SFS 270 config if mgmt ethernet no shutdown h Save the configuration to preserve it between reboots SFS 270 config if mgmt ethernet exit SFS 270 config exit SFS 270 copy running config startup config If you have two Fabric Controllers installed in Core slots you will receive a message asking if you want to save the configuration to the Standby Controller i Answer Yes to save the configuration across both controllers Connect Network Devices This section describes how to connect the InfiniBand system to other network devices InfiniBa
52. d switch into your web browser Press the Enter key The Device Rear View and the Device Front View appear Click open the Chassis file in the left navigation tree The individual components appear beneath the Chassis menu Click Cards in the left navigation tree The Cards view appears XxX 10 3 108 4 E Chassis lt g DDBDO a Cards TH Ports Power Supplies Fans Sensors y Backplane Management Ports Maintenance InfiniBand O Help Cards 10 3 108 4 gt Chassis gt Cards MECO Current Status Operational State Boot Stage Boot Status ib12port4xTX up normal z 2 ib12port4xTX up normal ejs ib12port4xTX up normal O 10 fabric12 up normal 11 controllerFabric12x up normal O 13 fabric12x up normal O 15 mgmtlo up normal O 16 mgmtlo up normal O 17 idModule up normal Data Refreshed At Wednesday April 21 2004 12 30 59 PM Figure 5 12 Chassis Manager Cards Window You can monitor the following types of cards none none none done done done none none none none none none success success success none none none InfiniBand 12 port Line Interface modules with 4x transmission capacity ib12port4xTX e Fabric controller modules installed in a Core slot The 12x refers to the internal chip speed controllerFabric12x e Fabric controller modules installed in a Node slot The 12x refers to the internal
53. d this shelf unit separately refer to the documentation that is provided in the shelf kit package Requirements In addition to the accessories provided with the server switch you should have 2 phillips screwdriver e Rack with 6Us of available space e Shelf installed in the rack not provided The shelf must support the weight of the chassis which can be up to 106 pounds with all possible components installed e Screws appropriate to your rack which are used to secure the rack handles to the rack posts e Three people are recommended to lift the switch into the rack however the procedure can be performed by two people yo NOTE The Topspin 270 Cisco SFS 7008 chassis weighs up to 106 pounds fully loaded Rack Mount Installation To mount the Topspin 270 Cisco SFS 7008 chassis in a rack 1 Remove the CD ROM parts bag and documentation from the box 2 Remove the packaging from around the chassis 23 3 Install a shelf for the chassis if you have not already done so Verify that the shelf can support the weight of the chassis which can be up to 106 pounds with all possible components installed T Figure 2 1 Install Appropriate Rack in Shelf 4 Determine the direction that the switch will be installed and justified in the rack The direction that the switch will be justified will determine which set of rack ears that need to be removed 5 Use a 2 phillips screwdriver to remove one set of rack ears
54. dow opens NOTE Before you install an image verify that you have brought up all of the cards on the chassis that you want to run the new image Cards that run a different image from the chassis cannot pass traffic NOTE Alert other users that you plan to install a new image to your Server Switch 14 Click the Yes button to install the image 15 Close the File Management window 16 Click the Maintenance menu and select Reboot 17 Click Yes at the prompt that appears 18 Click the do not failover to standby controller checkbox in the prompt that appears then click OK 66 67 5 Monitoring the Topspin 270 Cisco SFS 7008 System This chapter describes how to install and manage the Topspin 270 Cisco SFS 7008 system hardware Interpreting the Front Panel LEDs on page 67 Interpreting the Back Panel LEDs on page 73 Monitoring the System with Chassis Manager on page 78 Monitoring the System with Element Manager on page 86 Monitor the System With the CLT on page 88 Interpreting the Front Panel LEDs The front of the Topspin 270 Cisco SFS 7008 chassis has the following types of LED indicators System Wide Status LED on page 68 Power Supply LEDs on page 69 Fan LEDs on page 71 Fabric Controller LEDs on page 72 68 System Power Power Supply Status LED Supply Status LEDs Identify LED
55. e ambient temperature of the unit and will increase as the temperature rises Example SFS 270 gt show fan Fan Information fan oper status speed 1 up 76 2 up Y 3 up 74 4 up V7 Fan Seeprom product pca pca fru fan serial number serial number number number Monitor the Sensors e show sensor Example SFS 270 show sensor Sensor Information sensor oper status oper code temperature c alarm temp c shutdown temp c 10 1 up normal 35 713 85 Lif up normal Salt 15 85 12 1 up normal 20 75 85 LS 1 up normal Cull 75 35 15 1 up normal 38 70 80 16 1 up normal 37 70 80 Note the Temperature field The temperature is displayed in Celsius The system s maximum external ambient temperature is 30 degrees C 0 10 000 ft Acceptable internal temperature ranges are 30 degrees C above external ambient plus 1 degree for every 1 000 ft above sea level The acceptable temperatures for the Topspin 270 Cisco SFS 7008 varies depending on the card that is being monitored 90 91 Hardware Diagnostic Tests This chapter describes how to run diagnostic tests on the Topspin 270 Cisco SFS 7008 system hardware e About Diagnostic Tests on page 91 e Displaying Hardware Errors on page 101 About Diagnostic Tests Hardware diagnostic tests can be performed through the CLI or the Element Manager GUI Refer to the Element Manager User Guide for instruct
56. ees 53 Upgrading a Topspin 270 Cisco SFS 7008 with Element Manager oocccnccninciconoconcnonncconocnnncnnnos 54 Upgrading an Individual Controller Card iia 56 Upgrading a Controller Card with the Cll ag 57 Upgrading Individual Node Cards A a e e aa 59 Upgrading a Node Card with Element Managet cccccccecssesssecsseceseceeeeeeseecaeceeeeseecsaeeneenaees 60 Upgrading a New Management I O Card AAA 63 Upgrade a New Management I O Card with the CLI 0 cece eccececeeeceeneecneeceeeeeeeeeseeenaeeneensees 63 Upgrade a New Management I O Card with Element Manager c ccccecceceseceseeeeeeeetseeneeeeees 64 5 Monitoring the Topspin 270 Cisco SFS 7008 SY SIC Mii 67 Interpretinsthe Front Panel LEDS A is 67 System Wide tas LED odia 68 Power Supply ED e O E E 69 PAM E mA D E A E E E E A dm nbarell evened 71 Fabric Controller LEDS aid 72 interpreting the Back Panel LEDS noirin irad Nao 73 Management Interface Module LEDS ii 5 c ccccucsssscievacdusseeaccossctentesstesestessteanssthateedsscavnecusgetuctesetenns 13 Chassis ID LEDS ee a e ce Lal 76 EA DA ESAE ka S LEDS doit 77 InfiniBand Port EEDS 0 A sued E A a Meas 71 Monitoring the System with Chassis Manage ccccccsccesscseceesceceseceseeeeccesaecesecseeeeeseecsaeceeeseeeenseeesaeee 78 Monitor th Device o aia 78 Monitor the Cards a a hls aa Pad tis ati cae ela ale a teamed 81 Monitor the InfiniBand Ports yc ad A A e va asus a aac 82 Monitor the Power Supplies tt 83 Mont
57. ely by the terms of the license agreement and shall be prohibited except to the extent expressly permitted by the terms of that agreement Any technical data provided that is not covered by the above provisions shall be deemed technical data commercial items pursuant to DFAR section 227 7015 a or equivalent Any use modification reproduction release performance display or disclosure of such technical data shall be governed by the terms of DFAR section 227 7015 b or equivalent This documentation may include technical inaccuracies or typographical errors and is subject to correction and other revision without notice TOPSPIN PROVIDES THIS DOCUMENTATION AS IS AND WITHOUT ANY WARRANTIES OF ANY KIND EITHER EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE Some states or jurisdictions do not allow disclaimer of express or implied warranties in certain transactions therefore this statement may not apply to you Copyright 2004 2005 Topspin Communications Inc All rights reserved Topspin is a registered trademark and the Topspin logo TopspinOS and Topspin Switched Computing System are trademarks of Topspin Communications Inc Other company product or service names are the property of their respective owners Document Version 2 3 0 Part Number 10 00044 04 A0 June 2005 Printed in the United States of America Regulatory Notices ooc
58. emoving 6 Pull the levers on both sides of the card toward you The card will be released Figure 3 10 Opening Fabric Card Levers 7 Use the levers to pull the card toward you and remove it from the bay 8 Place a blanking panel over the open bay No bay should ever be run without a blanking panel or a module installed as overheating may occur or Refer to Installing a Fabric Controller on page 44 to install a new fabric controller 44 Installing a Fabric Controller To install a Fabric Controller perform the following steps NOTE If the card that you want to install features Anafa 2 chips verify that your Server Switch runs TopspinOS v2 3 0 or later Remove the new Fabric Controller module from the packaging Ground yourself appropriately Remove the bezel cover if you have not already done so Refer to Removing a Fabric Controller on page 42 if a Fabric Controller module is still installed Remove the blanking panel if no fabric controller was previously installed Align the new Fabric Controller module with the bay into which you want to install it Gently push the Fabric Controller module into the bay as far as you can AN Me we pa When the module is nearly completely installed press the levers in toward the Fabric Controller until they are flat against the module 9 Check the LEDs on the Fabric Controller module and the chassis to verify the status Refer to Fabric Controller LEDs o
59. ese 5 IlOgilQsds 2ZOO4s POSS HREGAg AS SED mies Wie IS ASA O OIE OS ISO PASSED ies Mai ES SAO OE O OE ASS ED EN ROSA O OLE O SEAN BS SED Fil Mee 5 1619233 2004 carcl Seaicuyo s 8 excel 1S Sieelicciine un Fri Mar 5 16 19 53 2004 Anafa2Init a2update set to IGNORE Fri Mar 5 16 20 15 2004 Anafa2 POST firmware check PASSED ENS OA O OLA OS SEE O MEE ASS Ein EAN IS SOI OA O SEE SA PASSED lt output truncated gt 4 Enter reload to reboot the switch after the test is complete If you do not reboot the switch after running the self test the switch will remain in an unknown state Run Chassis Tests The following tests are available to locate and diagnose the chassis LED 95 Standard page 95 Extended Extended Memory Run a Chassis Standard Test The chassis standard diag test runs a diagnostic test on all cards in the chassis 1 Enter the diag chassis test standard and start commands to initiate the test Example SFS 270 gt enable SFS 270 config SFS 270 config diag chassis test standard SFS 270 config diag chassis start SFS 270 config diag chassis 2 Enter the show diag chassis command to show the status of the diag test Example SFS 270 config diag chassis exit SFS 270 config exit SFS 270 show diag chassis Diaqgaostlie Tests Foz Chassis module type chassis module number test standard iterations option none acu lo start result success perce
60. ests For Fan module type fan module number ik test self test iterations 3 iL action start result in progress percentage completed 70 result string Self Test Current report SFS 270 Lid SIPFOCIPSS Sic 5 o 3 Use the show diag fan command to show the progress of the test Example SFS 270 config diag fan 1 exit SFS 270 config exit SFS 270 show diag fan Diagnostic Tests For Fan module type fan module number i test self test icerarioWas 3 L action start result in progress percentage completed 70 resil e s eae 8 Salir Wes Curz ae Peor SES 2 1 Os L a DOG SSS o 0 98 4 Use the show diag fru error command to display any errors that appear in the result string Example SFS 270 show diag fru error Bie O4 LieU S LOE ECU SIDO caraidh none card 2 none cardi none card 8 none card 9 none card 10 none euecl 111 none senal 12 none caraio none card 14 none ssusel LS none aeuecl 117 none fan 1 none fan 2 none 5 Reboot the chassis when the self test is complete Example SFS 270 reload System configuration is modified Save yes default no filename y Proceed with reload yes default no y Run Power Supply Tests The following tests are available to locate and diagnose fan modules LED page 98 e Self Test page 99 Run a LED Test on the Power Supply Use the LED test to
61. front of the chassis e System wide status Table 5 7 Management Module status Table 5 8 e Power Supply 1 and 2 status Table 5 10 e Fan Tray 1 and 2 status Table 5 9 In addition the Ethernet Management port RJ 45 connector on the Management Interface module has built in status LEDs for link and traffic status 74 Location The Management Interface modules are located at the top of back the chassis Refer to Figure 5 5 to view the chassis diagram IN U EEE ESE amp 3 53 05 Oo Figure 5 6 Management Interface Module LEDs Interpreting the Management Interface LEDs System Status LEDs Table 5 7 Interpreting the Management Interface Module System Status LEDs see Figure 5 6 Color State Indication Green and Yellow off No module power or LED failure Green solid on System running with no errors detected Green off No power to the Management Interface module LED failure or yellow LED is ON Yellow solid on At least one module failure in the system Operator attention required The label indicates a failure Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example Run the identity test on Power Supply 1 config mode gt diag card 15 gt test led gt
62. gure 3 2 Loosening Captive Screws 38 9 Use the handle to pull the power supply module from the bay Figure 3 3 Pulling the Power Supply Module from the Bay 10 Install the blanking panel in place of the power supply module The device should never be run without either a blanking panel or module in any bay as overheating may occur or Install a new Power Supply module Refer to Installing a New Power Supply Module on page 38 if you are installing a new Power Supply module Installing a New Power Supply Module Caution Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin 270 Cisco SFS 7008 chassis To insert a power supply unit 1 2 3 Remove the new power supply from the package Ground yourself appropriately Insert the provided power cord into the receptacle if one is not already connected and plug it into a power source The power cord receptacles are located on the back of the chassis Each power cord receptacle controls the power supply that is closest to it on the same end of the chassis Remove the bezel cover from the front of the switch if you have not already done so Refer to Removing a Power Supply Module on page 36 if a power supply is still in the bay Remove the blanking panel from the power supply bay with a 1 phillips screwdriver if it is still in place 39 7 Insert the new power supply uni
63. he Element Manager or Chassis Manager to upgrade downgrade one Controller card to match the image of another However the GUIs do not support the ability to reboot the standby Controller Therefore you must physically remove and reinsert the standby Controller card to activate the image 57 Upgrading a Controller Card with the CLI For the commands needed to upgrade a single Controller card using the CLI refer to the CLI Reference Guide 1 Check the version of your Controller card with the show card inventory command In the example below only the Controller card in slot 12 needs to be upgraded Example SFS 270 show card inventory Card Resource Inventory Information alorre used memory free memory used disk space free disk space last image source primary image source image cpu descr fpga firmware rev ib firmware rev alors iol used memory free memory used disk space free disk space last image source primary image source image cpu descr fpga firmware rev ib firmware rev bal 18092 kbytes 109676 kbytes 30891 kbytes 71488 kbytes MEOSOiMOS 2 2 0 90 els 31 MOMS Os Os ay Ol ASUS Sl MEOSOLiAOS 2 2 O owin els 31 PPC 440GP Rev C Rev 4 129 pvr 4012 0481 ae 0004031fe8 a E year 12 18092 kbytes 109676 kbytes 30891 kbytes 71488 kbytes Topspin0s 2 2 0 bui1d487 Topspin0s 2 2 0 bu11d487 TopspinOS 2 2 0 build487 PPC 440GP Rev
64. ides on your active fabric controller module skip this step and proceed to step 5 Example SFS 270 copy ftp bob mypassword 10 0 0 5 SFS 270 TopspinOS 2 3 0 build210 img image SFS 270 TopspinOS 2 3 0 build210 img 5 Enter the install command with the image that you want to install NOTE Before you install an image verify that you have brought up all of the cards on the chassis that you want to run the new image Cards that run a different image from the chassis cannot pass traffic NOTE Alert other users that you plan to install a new image to your Server Switch Example SFS 270 install image OS image 230 img Proceed with install yes default no 6 Enter the configure terminal command to enter Global Configuration mode 7 Enter the boot config command with e the primary boot image keyword e the directory of the installed image that you installed in step 5 to assign the image that you want both fabric controller modules to boot Example SFS 270 config boot config primary image source SFS 270 Topspin0S 2 3 0 build210 8 Enter the exit command to return to Privileged Execute mode SFS 270 config exit 9 Enter the reload command with the no failover keyword to reboot your Topspin 270 Cisco SFS 7008 and upgrade your new fabric controller module Example SFS 270 reload no failover 65 Upgrade a New Fabric Controller Module with Element Manager To upgrade the card perform the following steps
65. iew the progress of the test The test takes approximately 5 8 minutes Example of test in progress SFS 270 show diagnostic card 11 Diagnostic Tests For Cards test self test slloie ilel s 11 icsraclioms 3 1 AGIELOM 3 Siente result In progress percentage completed O result string Example of completed test SFS 270 config diag card 11 exit SFS 270 config exit SFS 270 show diagnostic card 11 Diagnostic Tests For Cards test 3loOw 0l iterations action result percentage completed result string system self test Lil start success 100 Cardi TSSE Final report PASS ED Please reboot 94 3 Use the more file system file name command to display the log file and view the detailed results of the diag test Example of log results SFS 270 more syslog hwif log Thu Mar 4 10 57 49 2004 POST SEEPROM PASSED Thu Mar 4 10 57 49 2004 POST FPGA PASSED Thu Mar A Os S7S4Zo ZOO4s IWOSWs SUIMIMALIES ieYAS Sap Maw Mes 4 1025 8390 20048 MOSS IRWCS IPAS SED Thu Mar le MOE Story Bes 2
66. ine Reference Guide for more information Table 5 6 Interpreting the Fabric Controller s Master Status LED see Figure 5 4 Color State Indication Active Master Status Indicator Green on This module is currently the system master Note that only fabric modules in one of the two core slots can become master Green off This module is not the active system master Note that only fabric modules in one of the two core slots can become master 173 Interpreting the Back Panel LEDs The back of the Topspin 270 Cisco SFS 7008 chassis has the following types of LED indicators e Management Interface Module LEDs on page 73 e Chassis ID LEDs on page 76 e LIM Status LEDs on page 77 e InfiniBand Port LEDs on page 77 Ethernet i Management Master Mirror of LEDs an oe py bi Chassis ID Status gt Connection LED chassis 7 LIM Card Status LEDs IB Port Status LEDs BODODODODO Oo000o0g0 EEEE OO0000000 OO0000000000000000000000 Figure 5 5 Back Panel LEDs Management Interface Module LEDs The Management Interface modules have six LED yellow over green indicator pairs which show critical system module statuses The six pairs of LED indicators are a mirror of the LEDs that appear on the
67. ion on running tests in with the Java GUI A Power On Self Test POST is run automatically on various components at power on The following examples refer to running tests and displaying test with the CLI For complete diagnostic command information refer to the Command Line Reference Guide LED Tests LED tests allow you to blink a particular LED to more easily identify a specific component in a rack Self Tests The Topspin 270 Cisco SFS 7008 provides the following diagnostic tests e Card Test page 92 e Chassis Test page 94 e Fan Test page 96 Interface Test 92 e Power Supply page 98 e Rack Locator Run Card Tests References to Cards refers to fabric controllers management I O modules node cards and the chassis ID module Fans and power supplies are not included in the Card tests they can be tested through the standard chassis test page 95 or as individual components The following tests are available to locate and diagnose one or more cards in the chassis LED e Selftest page 92 Run a Card Self Test To perform a diagnostic self test on a card 1 Enter the diag card command You cannot stop the card test once it has begun Example SFS 270 gt enable SFS 270 config srs 270 config diag card 11 SFS 270 config diag card 11 test self test SES 70 Gomiricg Clilag Gaccl 11 3 seeks SFS 270 config diag card 11 exit SFS 270 config exit 2 Exit the test to v
68. mponent Detection of abnormal status is reported 21 Installing the Topspin 270 Cisco SFS 7008 This chapter describes how to install and manage the Topspin 270 Cisco SFS 7008 system hardware Prepare the Site on page 21 Configure Basic Connectivity on page 29 Mount the Topspin 270 Cisco SFS 7008 Chassis in a Rack on page 22 Connect Network Devices on page 30 Manage the System on page 31 Prepare the Site This section provides information that you need to safely and successfully prepare your environment for your Topspin 270 Cisco SFS 7008 Read this section carefully before you install your device Read the Cautionary Statements Refer to the FCC Statement on page vii and the Safety Information on page vii Prepare the Physical Environment for the Topspin 270 Cisco SFS 7008 System Make sure you have the right cables and sufficient ventilation Unpack the Topspin 270 Cisco SFS 7008 package 22 e Prepare a management workstation such as a PC running a terminal program and a straight through M F DB 9 serial cable included Mount the Topspin 270 Cisco SFS 7008 Chassis ina Rack This section describes how to install the Topspin 270 Cisco SFS 7008 chassis in an equipment rack The following procedure is the standard installation using your own rack shelf Topspin also has an optional shelf that is appropriate for pre racking and shipping a unit in the rack If you have purchase
69. n page 72 10 Tighten the captive screws on both ends of the Fabric Controller module Figure 3 11 Tight the Captive Screws 11 Refer to Upgrading the Controller Card on page 44 Upgrading the Controller Card It is likely that the image version on the new Fabric Controller card will not match the firmware on the rest of the chassis After the new Fabric Controller module has been installed upgrade the firmware on the chassis and the new Fabric Controller will be upgraded automatically 45 The version of the firmware is verified during the upgrade process If a module does not need to be upgraded the module will be skipped and move on to the next module The image can be upgraded with the following methods e Upgrade the Image with the CLI on page 45 e Upgrade the Image with the Chassis Manager on page 45 e Upgrade the Image with the Element Manager on page 45 Upgrade the Image with the CLI To upgrade the image of the Controller card with the Command Line Interface refer to Upgrading a Controller Card with the CLI on page 57 and the following document e CLI Reference Guide Upgrade the Image with the Chassis Manager To upgrade the system image with the Chassis Manager web GUI refer to the following document e Chassis Manager User Guide Upgrade the Image with the Element Manager To upgrade the system image with the optionally licensed Element Manager GUI refer to the following document
70. n system has two power cables The operator must disconnect all power cables before attempting to remove the system from the rack A CAUTION Remove power cables by grasping the cable connector and pulling straight out If the chassis comes equipped with a power plug retainer move the bail wire used to retain the plug to the side before attempting to remove the power plug Replace power supply cables be grasping the cable connector align the connector with the power jack and insert the connector straight into the jack If the chassis comes equipped with a power plug retainer move the bail wire back into place to keep the power plug firmly in place CAUTION The Topspin system contains lithium batteries Do not attempt to replace or discard these batteries The batteries may only be serviced by Topspin service personnel CAUTION Observe and follow service markings Do not service the Topspin system except as explained in the Topspin documentation supplied with your system Opening the chassis or removing the enclosure cover exposes you to electrical shock and may damage system components If one or more of the following situations occurs remove power to the Topspin system chassis and contact your Topspin support representative The power cable or power plug is damaged e The system has been exposed to water e The system has been dropped or damaged in any way The system does not operate correctly after following the installation inst
71. n the Captive Screws 5 Release the LIM module by pulling both levers toward you then toward the outside of the chassis Il Il YE DUO ii d Figure 3 15 Releasing the LIM Module Levers 6 Use the levers to pull the module from the bay Figure 3 16 Pulling a LIM Module from the Slot 48 7 Install a blanking panel in place of the LIM module or refer to Installing a LIM Module on page 48 for instructions on replacing the LIM module Installing a LIM Module ee Ne Remove the new LIM module from the package Refer to Removing a LIM Card on page 45 if a LIM module is still in place Ground yourself appropriately Remove the blanking panel if a LIM module was not currently in place Align the LIM module and slide it into the bay as far as possible The module will not go into the bay completely without closing the levers When the LIM module is nearly installed in the bay press the levers in toward each other Figure 3 17 Closing the LIM Module Levers Check the LIM status LED to determine that the module is functioning properly Refer to LIM Status LEDs on page 77 Tighten the captive screws on both ends of the LIM card NI A n Aoi ve oe e SM 021 REV Bo Figure 3 18 Tighten the Captive Screws 49 9 Connect the InfiniBand cables to the ports a To plug in an InfiniBand cable push the connector into the interface until you hear feel a click Che
72. nd devices can be connected to InfiniBand enabled servers About Connectors The Topspin 270 Cisco SFS 7008 device supports the following types of connectors e Management ports e Serial console port male DB 9 connector Ethernet Management port RJ 45 jack for unshielded twisted pair connections e Switch card 4X 10 Gbps InfiniBand connectors Manage the 31 Connecting InfiniBand Devices Attaching the IB Cable to the IB Port To connect using InfiniBand to other workstations or switches standard 4X InfiniBand cables are required InfiniBand cables can be used to connect any two InfiniBand devices whether switch or host 1 Plug InfiniBand cables into the InfiniBand switch ports To plug in an InfiniBand cable push the connector into the interface until you hear feel a click Figure 2 9 Removing a Pinch Connector a Ifthe IB cable is already connected to an IB host the physical LED light will turn green on the switch IB port when connected properly b To remove a cable with a pinch connector pinch both sides of the back of the connector and pull the connector away from the port Connecting to Ethernet or Fibre Channel Devices Connecting to the Host The opposite end of the InfiniBand cable will most likely be connected to an InfiniBand enable host refer to the HCA Installation Guide If your host does not provide an ample amount of free space around a given IB port double check that your IB cable connector
73. nfiguration use factory defaults Figure 4 3 Set the Boot Config c Repeat for all Server Switches you wish to upgrade 56 6 Select Maintenance gt Reboot to reboot the controller cards and activate the image Click Yes at the prompt 7 View the new image on the cards a Select View gt Device Front View b Right click one of the cards and select Properties c Click the Inventory tab and view the version name in the Current Image Source field IB Fabric Core Card 11 10 3 108 6 TT Card Inventory Used Memory 17672 KB Free Memory 110088 KB Used Disk Space 30879 KB Free Disk Space 71500 KB Current Image Source TopspinOS 2 2 O build531 Image Source for Next Reboot TopspinOS 2 2 Ofouild531 Image One TopspinOS 2 2 O ouild531 Image Two CPU Description PPC 440GP Rev C Rev 4 129 pvr 4012 0481 IB Firmware Revision 0004031 fes Figure 4 4 Upgraded Current Image 8 Repeat for additional switches in the fabric Upgrading a Topspin 270 Cisco SFS 7008 with the Chassis Manager For the procedure needed to upgrade a Topspin 270 Cisco SFS 7008 chassis using the Chassis Manager GUI refer to the Chassis Manager User Guide Upgrading an Individual Controller Card When upgrading a single Controller card for example when replacing an old fabric Controller card with a new one it is recommended that you use the CLI if you need to change the image on the new card You can use t
74. ntage completed 100 result string Standard System Test Completed Final report Passed 1 Failed 0 Total 1 SESE AOH 96 3 Enter the show diag post command to show any errors on individual components Example TS270 1 show diag post Post Status cru s Lore post status post error card 1 failed _FRU_LIM CARDTYPE ERR card 3 failed _FRU_SEEPROM_ PROGRAM ERR card 5 passed none card 6 passed none card 7 passed none card 8 passed none card 9 passed none Gard i failed _FRU_POWER_CONTROLLER_CONFIG ERR card 12 passed none Samara passed none sauce 15 passed none card 16 passed none euecl 17 passed none fan 1 passed none Run Fan Tests The following tests are available to locate and diagnose fan modules LED e Self Test page 96 Run a Self Test on a Fan 1 Identify the number of the fan you want to diagnose if you do not already know the number a Enter the show fan command Example SFS 270 gt enable SFS 270 show fan Fan Information fan oper status speed 1 up 74 2 up 70 3 up 89 4 up 87 2 Enter the diag fan lt id gt command Example SFS 270 config SFS 270 config diag fan 1 SFS 270 config diag fan 1 test self test SFS 270 config diag fan 1 start SFS 270 config diag fan 1 exit SFS 270 config exit SFS 270 show diag fan 97 Diagnostic T
75. o Management Interface Module LEDs on page 73 7 Connect the Ethernet or serial cables to the ports 8 Tighten the outside screws with a 1 phillips screwdriver Chassis ID Modules About Chassis ID Modules Refer to Chassis ID Module on page 19 for additional information Failed Chassis ID Module If you believe a chassis ID module has failed contact technical support Refer to Chassis ID LEDs on page 76 and Monitor the Cards on page 81 for monitoring information e A failure of the Chassis ID module will cause the yellow error LEDs to be lit on the Chassis ID module The System Status LED will turn yellow as well A CAUTION Never remove a Chassis ID unless Technical Support has specifically instructed you to do so Locating the Chassis ID Module The Chassis ID module is located on the back of the chassis Refer to Figure 1 2 for a diagram of the chassis Removing a Chassis ID Module 1 Ifyou must remove a Chassis ID module verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 2 Ground yourself appropriately 51 3 Use a l phillips screwdriver to loosen and remove the screws from the left and right side of the Chassis ID module tees Figure 3 20 Loosen the Captive Screws 4 Pull the Chassis ID module from the bay jg Figure
76. odules The Management Interface modules provide the following functions RS 232 serial console port e 10 100 Ethernet management port e System real time clock RTC Intra system Ethernet switch which provides a system configuration management data path to all fabric modules e Provide mirrors of the LED indicators from the front side of the chassis for the power supply fans and over all system status refer to Management Interface Module LEDs on page 73 Rack Alarm Connection refer to Rack Alarm on page 19 Redundant Ethernet Serial Rack Alarm LEDs Port Port Figure 1 5 Management Interface Module 18 Understanding the Core Slot to Mgmt Interface Module Connection Each Core slot in the chassis refer to The Core Slot on page 18 is connected to 1 port on the Management Interface module The controller fabric interface module inserted in the core slot selects the Management Interface module to which it will connect The pairing of Management Interface module to core slot is as follows The Management Interface module in slot 15 is connected to the Fabric Controller Module in the Core slot 11 The Management Interface module in slot 16 is connected to the Fabric Controller Module in the Core slot 12 For example removing the Fabric Controller from the Core slot 11 would cause both the Fabric Controller card and its paired Management Interface module in slot 15 to fail over The ne
77. ooccooccconicononono vil FCO St tement A SAS anes vil A eit since us seedicie Bat oa dans E a a ae eats vii Electrical Cautions ds vii General Cautions enean e aa a it aa aai a a araa ix C ntact Infotm t Ms ics A iaa ix 1 Features Overvi W cccccccececcececeeeecees 11 System Architectures iresi a E R A E E A E cet E E ER RS 11 Hot Standby 235 os orien ccb dan RR A 11 InfiniBand Connectivity AAA A AT EA Dae at Aas es Ala ats E Aes atk 12 PIDA A ld 12 A doa a N T E E E eae mse E 12 INon Blocking Architectures cenieni sta dure A 13 About the Topspin 270 Cisco SFS 7008 Chassis scssccssccsssssssseseccsssscssccesctesaccssscessecesseesaceenacersetuaces 13 Diagram of the COI nirin A A RE setae 13 Chassis Slot NU dt ias 15 A fis vas phen n iA eA a asa AEA dae ee ibe aaa edna etal nal das Raga 16 AEN HO NS i ae ea ee NA a a EAE A A ete att 16 Mia ths E cat ict ccna a le 16 Power SU A UA 16 IPAM TAY Sia A E E A EE EA 17 Management Interface Mod ia 17 Fabric Controller Mad Ges Az Ea 18 A E EA EET 18 Line Interface Modules LIM sis ar cigs Seis A 19 AE ct scot a vec abn y aa ae a a est a E a cen pcan Taa 19 Rack Alarni A aa 19 Administrative Esas A 20 Real Time Clos 20 o A RN 20 Non Volatile MO Uds 20 Vital Product Data MIA od 20 Di O AS A E Poneman nea aces ie 20 2 Installing the Topspin 270 Cisco SFS 7008 21 IAR OINEN A A ate S nda cna cers ant te ea eal A 21 Read the Cautionary aleman ana haem a mane 21
78. os Pa AAA ESE ad 84 Motor the TEMPE A a a ia a dia anak 85 Monitoring the System with Element Manager ao bd 86 Interpreting the SUMMA Ud ios 86 Interpreting the Power Supplies Tab i 87 Interpreting the Fans AD A e E a aci n 87 Interpreting the Sensors Taba cias 87 Monitor tie System With the Cl ears n iea i e A ad 88 6 Hardware Diagnostic Tests 91 About DIAGNOSIS o a TRR 91 EED Testset anna a ra os el e bs o eee 91 ea l D KE EEEE dusa usawadaseea ones Gy ensnseas E E E E EE 91 R n Card A a A A a A 92 Run a Cata Soles ti 92 A O O natant as os Wotan Ala ena 94 Run a Chassis Standard Testi aii 95 vi R n Pati TOS ia aei iiaia ede AG alee ene A ae edie Oe a eae ce aaah 96 R a Belle lesiona Fil ida 96 Run Power SUP Tests id acaso 98 Run a LED Test on the Power Supply a eae wee ohio mae 98 Run a Self Test on a Power Supply cr Aa 99 Displaying Hardware Errors AN 101 7 Specifications and Compliance Certifications 105 Chassis and Management Interface ni da 105 Electrical Specifications os A T A E Pin tea Untied onan 106 EAC Tannin ity fos tetas aa ane A A AA Ei 106 E etn agl Cun E al nas 106 PRC OUISTICS as ce tate alado 106 Product Maras ee a Sate a os rl e alae 107 OL INGO A 109 vil Regulatory Notices FCC Statement This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reason
79. power from the chassis 36 About a Failed Power Supply If you believe a power supply module has failed contact technical support Do not remove a failed power supply unless you have a new power supply to install immediately or a blanking panel to place over the open bay as overheating may occur In most cases vital information can be recovered from the failed module If you have the appropriate support agreement you may want to return the unit for diagnosis Refer to Vital Product Data Storage on page 20 Locating the Power Supply Module Both power supply units are located on the front of the chassis When facing the front of the switch with the bezel cover removed the power modules are located in the top center bays of the Topspin 270 Cisco SFS 7008 chassis Refer to Figure 1 1 for a diagram of the chassis Removing a Power Supply Module Caution Do not remove a Power Supply module without first loosening the screws with a phillips screw driver 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove a failed power supply unless you have a replacement or a blanking panel to install in its place as overheating may occur 3 Make sure you have a 1 phillips screwdriver to disengage the fasteners Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin 270 Cisco SFS 7008
80. pspin 270 Cisco SFS 7008 features hot swappable and redundant e Power Supply Modules e Fan Trays Line Interface Modules redundant connection from HCA e Fabric Controller Modules in redundant Core or Node slots Management Interface module The Topspin 270 Cisco SFS 7008 features hot swappable e Chassis ID Ports No failure on any single InfiniBand port will result in interruptions in service for any of the other ports 13 Fabric For redundancy InfiniBand Host Channel Adapters HCAs can be dual connected to a redundant pair of Topspin 270 Cisco SFS 7008s In an InfiniBand fabric that includes more than one Topspin 270 Cisco SFS 7008 if the subnet manager on the Topspin 270 Cisco SFS 7008 that is acting as the master fails another subnet manager will take over within seconds All necessary state information is kept in sync Non Blocking Architecture The Topspin 270 Cisco SFS 7008 provides non blocking switch element architecture with full bi directional bandwidth for the switch chassis About the Topspin 270 Cisco SFS 7008 Chassis An overview of the Topspin 270 Cisco SFS 7008 hardware features are described in this section Diagram of the Chassis POWER SUPPLIES FAN FAN L O oo y ODODOD ODODOD DODOODO o ODODOD 00000 e
81. ready done so 40 4 Usea l phillips screw driver to unscrew the captive screws Figure 3 5 Loosening Captive Screws 5 Carefully begin to pull the fan tray 1 out of the bay Do not pull the fan tray completely from the bay Nor The fan tray does not have guards over the blades You must wait until the fan blades have stopped moving before completely removing the fan tray from the bay or risk injury 6 Wait 15 seconds to allow the fan blades to stop moving e Figure 3 6 Partially Removing Fan Tray 7 Pull the fan tray completely out of the bay 8 Install the blanking panel in place of the fan module Only one fan module is required to run the switch however the device should never be run without either a blanking panel or module in any bay as overheating may occur 41 or Install a new fan module Refer to Installing a Fan Module on page 41 if you are installing a new fan module Installing a Fan Module Remove the new fan module from the package Ground yourself appropriately Remove the bezel cover from the front of the switch if you have not already done so Refer to Removing a Fan Module on page 39 if a module is still in place Remove the blanking panel if no fan module was in place Oy Ou eee A Align the fan module with the bay and slide the module into the bay Figure 3 7 Pushing Fan Module into Slot 7 Push the module until it is firmly in place 8 Use the
82. rt details 83 port LEDs 77 port properties 83 port RJ 45 LEDs 76 power off card 82 power supplies about 16 power supply LEDs 69 prepare your switch 32 prerequisites 32 R rack locator 92 reload 94 reset card 82 reset controller card 59 S safety information vii setup 32 show card 11 show diag chassis 95 show diag fru error 98 100 show diag post 96 show power supply 99 slot numbers 15 speed port 83 starting Chassis Manager 33 switch configuration 32 switch fabric end nodes about the fabric controller module 18 switch fabric spine about the fabric controller module 18 system controller about the fabric controller module 18 system wide status LED 68 T temperature monitor with chassis manager 85 tx rx 78 v view port properties 83
83. ructions A CAUTION You must ensure the operating environment has adequate air circulation Keep the Topspin system in a cool well ventilated room Do not block cooling vents Install filler panels into all unused slots ix A CAUTION Never place your hand inside an empty card or module bay You should never have cause to place a hand anywhere inside the Topspin chassis Unused card and module bays should always have a Topspin cover over the bay to ensure proper safety ventilation and cooling General Cautions CAUTION No user is authorized to remove the Topspin system enclosure cover The internal chassis contains no user serviceable components Removing the enclosure cover voids your warranty See the warranty card for further details regarding the servicing of your chassis In general Do not spill food or liquids on your system components e Protect your Topspin system from sudden power surges and interruptions by using a surge suppressor line conditioner or uninterruptable power supply UPS e Place cables appropriately so that they do not obstruct any egress within the data center and they do not block any ventilation inlets or outlets e Rack mount the Topspin chassis according to the rack manufacturer recommendations Contact Information Table lIl 1 Customer Contact Information Address Topspin Communications Inc 515 Ellis St Mountain View CA 94043 Telephone Corporate Headquarters 866
84. ry Card Type Enable Disable Card Current Card Status Operational State normal Card Boot Stage done Card Boot Status success Serial Number 3 PCA Serial Number PY 0340 000003 PCA Assembly Number 95 00036 01 FRU Number O Figure 4 8 Disable and Enable the Node Card s c Click the up radio button and select the Apply button View the new image on the node card s when the card comes back online a Select View gt Device Front View b Right click one of the cards and select Properties c Click the Inventory tab and view the version name in the Current Image Source field IB Fabric Core Card 11 10 3 108 6 Inventory Used Memory 17672 KB Free Memory 110088 KB Used Disk Space 30879 KB Free Disk Space 71500 KB Current Image Source TopspinOS 2 2 Ofuild531 Image Source for Next Reboot TopspinOS 2 2 O ouild531 Image One TopspinOS 2 2 0 build531 Image Two CPU Description PPC 440GP Rev C Rev 4 129 pvr 4012 0481 IB Firmware Revision 0004031 fed Figure 4 9 Upgraded Current Image Card Card 63 Upgrading Node Card s with the Chassis Manager For the procedure needed to upgrade node cards using the Chassis Manager GUI refer to the Chassis Manager User Guide Upgrading a New Fabric Controller Module Login super Password XXXXX SFS 270 gt enable SFS 270 When you add a new fabric controller module to a production Topspin 270 Cisco SFS 7008 you mu
85. s Element Manager User Guide Line Interface Management Modules LIM About LIM Cards Refer to Line Interface Modules LIM on page 19 for specific information about the LIM modules About a Failed LIM Module If you believe a line interface module has failed contact technical support Refer to LIM Status LEDs on page 77 and Monitor the Cards on page 81 for monitoring information Locating a Failed LIM Module The Line Interface Management modules are located on the back of the chassis Refer to Figure 1 2 for a diagram of the chassis Removing a LIM Card 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 46 3 Remove the InfiniBand cables from the ports a To remove a cable with a pinch connector pinch both sides of the back of the connector and pull the connector away from the port Figure 3 12 Removing a Pinch Type Connector b To remove a cable with a pull connector grasp the connector with one hand and push it toward the port then pull the latch away from the port with your other hand and gently wiggle the connector away from the port Figure 3 13 Removing a Pull Type Connector 47 4 Use a l1 phillips screwdriver to unscrew the captive screws on both sides of the LIM card Figure 3 14 Loose
86. s eo ODODOD ODODOD ODODOD CE E E E EC nAad000 Fabric Controller Module o Node 20000 O Node siot 1 ee O Slots y m 5 Ooooo Fabric Controller Module o ODODOO o Node Slot 2 ool DOODOOD fa DOODOD 279000 Fabric Controller Module s Core 0000 Core Slot 1 O Slots DODO 0000 Fabric Controller Module o ODODOD oo aaa O Core Slot 2 OQ ODODOD OOOOOO Fabric Controller Module o oonoo000 Node Slot 3 se N oooo ode oooo z Slots zee Fabric Controller Module o BEER EG Node Slot 4 one O ooooo0o0 I oo0000 oODoODOD ooooo0o0 Figure 1 1 Front Chassis View without Bezel 14 POWER PLUG 2 MANAGEMENT I O CHASSIS ID MAIN AIR EXHAUST K MODULES o g m o e DO DODODOODOODOOODOOO OOOODOODOOoO DOPODOOODOOOODOODOO DO 4x COPPER LINE INTERFACE MODULE DODODO mannna DODODO DODODO DODODO DODODO DODODO DODODO DODODO DODODO DODODO DODODO DODODO DODODO OO0000 DODODO DODODO DODODO DODODO DODODO DODODO DODODO Figure 1 2 Back Chassis View Chassis Slot Numbering Slot numbers are printed on the chassis for easy identification Power Supply ID 1 Power Supply ID 2 15
87. ss to a Topspin management port The GUI can be found in the following location on the CD ROM e Linux install_linux_ia64 bin e Linux install_linux_x86 bin e Windows install exe e Solaris install_solaris_sparc bin Run the installation wizard a On Windows double click install exe b On Solaris and Linux systems enter sh install_yourOS bin c Follow the on screen instructions d Enter location and preference information as requested by the installation program 6 Start the Element Manager 35 Managing Individual Components This chapter describes how to install the Field Replaceable Units FRUs in the Topspin 270 Cisco SFS 7008 system About the Field Replaceable Units The following Field Replaceable Units FRUs are a part of the Topspin 270 Cisco SFS 7008 chassis e Power Supply Modules on page 35 e Fan Tray Modules on page 39 e Fabric Controllers on page 42 e Line Interface Management Modules LIM on page 45 e Management Interface Modules on page 49 e Chassis ID Modules on page 50 Power Supply Modules About the Power Supply Modules The Topspin 270 Cisco SFS 7008 power supply units are hot swappable Only one power module is necessary to power the unit However two power supplies are recommended for redundancy You can add a second module without powering off the chassis If you have two power supply units installed you can remove one of them without removing
88. st upgrade the new module so that it runs the same firmware as the existing module If you neglect to upgrade the second fabric controller module your Topspin 270 Cisco SFS 7008 will experience problems in the event of a failover because the failover will boot a different OS version The critical difference between this upgrade process and the standard upgrade process occurs when you force the Topspin 270 Cisco SFS 7008 to skip the standard controller module failover when you install and reboot The instructions that follow assume that e You added the new fabric controller module to the Topspin 270 Cisco SFS 7008 as it ran e You have not yet rebooted the chassis Upgrade a New Fabric Controller Module with the CLI To upgrade the module perform the following steps 1 Launch a CLI session to your Topspin 270 Cisco SFS 7008 and log in as a user with administrative privileges Example 2 Enter the enable command to enter Privileged Execute mode Example 3 Enter the show card inventory command to view the images that run on each fabric controller module the devices in slots 11 and 12 If the images match you do not need to upgrade the new management I O module 4 Enter the copy command with e the location and file name of the image file that you want to copy e the destination and file name for the file on your chassis to copy an image file to your Topspin 270 Cisco SFS 7008 64 NOTE Ifthe appropriate image file already res
89. t into the open slot until it is fully seated You may need to push the unit with your thumbs to get it completely into the bay Figure 3 4 Inserting Power Supply Module 8 Secure the captive screws with a 1 phillips screwdriver 9 Check the LEDs on the power supply to verify the status of the module Refer to Power Supply LEDs on page 69 Fan Tray Modules About the Fan Tray Modules Refer to Fan Trays on page 17 for more information about the fan tray modules About a Failed Fan Tray If you believe a fan module has failed contact technical support Refer to Fan LEDs on page 71 and Interpreting the Fans Tab on page 87 for monitoring information Locating the Fan Tray Both fan modules are located on the front of the chassis When facing the front of the switch with the bezel cover removed the fan modules are located in the left and right receptacles of the Topspin 270 Cisco SES 7008 chassis The slots are labelled on the chassis Refer to Figure 1 1 for a diagram of the chassis Removing a Fan Module Note Do not remove Fan module without first loosening the screws with a phillips screwdriver 1 Ground yourself appropriately 2 Verify that you have a means of covering the slot before removing the module Do not remove the module unless you have a replacement or a blanking panel for the slot as overheating may occur 3 Remove the bezel cover from the front of the switch if you have not al
90. the Speed field The integer in this field represents a percentage The percentage changes based on the ambient temperature of the unit and will increase as the temperature rises Interpreting the Sensors Tab Use the Health gt Status gt Sensors tab to view the operating status of the temperature sensor of the system 88 S Health Status normal normal normal sor DE O eee Figure 5 21 Element Manager Sensors Tab 6 row s 6 Note the OperStatus field A status of up indicates that the sensor is functioning properly 7 Note the Temperature field Each slot has an individual temperature sensor The Alarm and Shutdown temperatures vary by slot type Monitor the System With the CLI For a complete list of CLI commands refer to the CLI Reference Guide Use the following CLI commands to monitor the Power Supplies the fans and the sensors Monitor the Power Supplies e show power supply Example SFS 270 gt show power supply Power supply Information ps type oper status WELL gatiloem voltage 11 AC up n a 48 2 AC down n a 48 Power supply Seeprom produ pca pca EU ps serial number serial number number number al 2 Monitor the Fans e show fan An oper status of up means that the blower is operating correctly within the fan unit 89 Note the Speed field The integer in this field represents a percentage The percentage changes based on th
91. the status of the Power Fans and Temperature Sensors at once Interpreting the Power Field A green check next to the Power summary indicates that at least one power source is connected and functioning properly e A red check mark indicates that power supply AC is disconnected Interpreting the Fans field e A green check next to the Fans summary indicates at least one fan is present and functioning properly A red check indicates a fan failure Interpreting the Sensors field e A green check next to the Sensors summary indicates that the system temperature is at an acceptable level A red check for Sensor is a warning of high temperature 87 Interpreting the Power Supplies Tab Use the Health gt Status gt Power Supplies tab to view the operating status of the power supplies Health Status Summary Fans Sensors Pote Type OperStatus Utilization Voltage ProductSerialNum PcaSerialNum PcaAssemblyNum FruNum lac up n a 112 Infa infa Infa Infa Retesh Ma E E ciose Het Figure 5 19 Element Manager Power Supplies Tab Data cached at 10 43 13 Interpreting the Fans Tab Use the Health gt Status gt Fans tab to view the operating status of the fans Health Status Retesh Ma E E ciose et Figure 5 20 Element Manager Fans Tab Data cached at 10 43 16 4 Note the OperStatus field A status of up means that the fan is operating properly 5 Note
92. the user This LED can be initiated manually by using the diag LED test Refer to the Command Line Reference Guide for more information Example To run the identity test on fan 1 config mode gt diag fan 1 gt test led gt gt start start flashing gt stop stop flashing 72 Fabric Controller LEDs Each Fabric Module has two sets of LED indicators The two LEDs green or yellow are the module status indicators Table 5 5 The single LED green only indicates whether of not the module is the active system master Table 5 6 Location Refer to Figure 5 1 to view the front panel of the chassis Module Active System Maste Status gt lt Figure 5 4 Locate Fabric Controller LEDs Interpreting the Fabric Controller LEDs Table 5 5 Interpreting the Fabric Controller s Module Status LEDs see Figure 5 1 and Figure 5 4 Color State Indication Green over off No system power or LED failure Yellow Green solid on System running with no errors detected Green off No power to the fabric controller LED failure or yellow LED is On Yellow off No failure on the fabric controller Yellow solid on Module error detected Operator intervention required Yellow blinking Identify Assists in identifying a particular Field Replaceable Unit on the chassis This state is initiated by the user This LED can be initiated manually by using the diag LED test Refer to the Command L
93. to the master fabric controller module and a fail over will occur when you remove the management interface module The system will fail to the redundant management interface module and the redundant fabric controller module 3 Ground yourself appropriately 4 Unplug any Ethernet management or serial cables that are connected to the ports of the Management Interface module 5 Use a l phillips screwdriver to unscrew the Management Interface module 6 Use the handle to pull the module from the bay 0 4 1 4 4040401404013 a EL EE E e Figure 3 19 Example of the Management Interface Module 50 7 Ifthe Master Management Interface module was removed the standby module becomes active The Master led becomes green 8 Install a blanking panel in the empty bay or refer to Installing a Management Interface Module on page 50 to replace the Management Interface module Installing a Management Interface Module 1 Remove the new Management Interface module from the package 2 Ground yourself appropriately 3 Refer to Removing a Management Interface Module on page 49 if a Management Interface module is still in place 4 Remove the blanking panel if a Management Interface module was not previously in place Align the Management Interface module and push the module completely into the bay 6 Use the LEDs to verify the status of the management interface module and the corresponding core card Refer t
94. tus color and the number of modules installed The numbers associated with the Card field indicate the slot numbers where modules are installed The numbers associated with the Power Supplies and Fans fields indicate the ID for the module If only one number appears next to any of the modules then only one item is installed The actual integer refers to the slot in which the power supply module is installed A field that read Power Supplies 1 2 would indicate that a power supply was installed in both possible slots The color of the integer indicates the status of the power supply A green number indicates that the power supply is Up A red number indicates that the power supply has failed 5 Use the visual representation of the LEDs to monitor the status of the cards ports power supplies and fans e A green LED indicates that a hardware component is installed in the slot and that the component is functioning without errors A white LED indicates that is no component is installed in the slot A grey LED indicates one of the following states e A component is installed but is unmanaged or unreachable 81 e In the case of the fabric controller cards the card that is currently on standby not the master will be grey The master fabric controller will be green A red LED indicates that an error has been detected Monitor the Cards 1 Launch the Chassis Manager by entering the IP address of the InfiniBan
95. u Click Sensors in the left navigation tree The Sensors window appears Sensors 10 3 108 4 gt Health gt Sensors Refresh Slot ID Sensor ID Current Status Oper Code Current Temp c Alarm Temp c Shutdown Temp c 10 1 up 58 91 98 11 12 13 15 16 normal 1 up normal 56 91 98 1 up normal 57 91 98 1 up normal 59 91 98 1 up normal 34 65 75 1 up normal 34 65 iS Data Refreshed At Friday April 23 2004 1 05 28 PM Figure 5 17 Chassis Manager Sensors Window 2 6 Verify that the sensor is Up by checking the Current Status column Compare the Current Temp c column to the Alarm Temp and Shutdown Temp columns The Current Temp should be lower than the Alarm Temp The Shutdown Temp indicates the degree in celsius at sea level at which the switch will reboot Each slot is monitored with individual sensors The Alarm and Shutdown temperatures vary depending on the slot 86 Monitoring the System with Element Manager For information regarding installing the optionally licensed Element Manager GUL refer to the InfiniBand User Guide 1 Launch the Element Manager 2 Select Health gt Status The Health Status window opens Health Status Power Supplies Fans Sensors Up Time 2h 30rm 28s 10 01 2003 10 37 39 Power Fans Ya Sensors Yi Refresh Close Help Figure 5 18 Element Manager Summary Tab Interpreting the Summary Tab 3 Use the Summary tab to view
96. ur browser and press Enter You configured the IP address in Step f File Edit View Favorites Tools Help ay QO dla JO Search she Favorites Queda ZQ A HUSB Address 10 2 182 44 g g v Eso Links gt Search for J LO web search 19 9 14 E Popups blocked z POS BSAA Figure 2 10 Web Browser Window A log in window opens Figure 2 11 displays the log in window Please login Figure 2 11 Chassis Manager Log In Window 3 Enter your Server Switch user name and password in the log in window and click the OK button Chassis Manager loads in your browser window SS TopsPIN Xx 10 3 108 4 Device View ag Chassis 10 3 108 4 gt Device View Y Cards amp Ports Refresh amp Power Supplies B Fans Device Rear View amp Sensors FERRE NANA E Backplane ain JEG 3 Management Ports Mk Maintenance InfiniBand O Help 1 12 333 1 2 3 4 5 6 7 8 0 sl el la la ea a a a la le le le Figure 2 12 Chassis Manager Opening 34 Optional Install the Element Manager GUI For more information regarding the Element Manager tasks refer to the InfiniBand User Guide 1 Check that you have sufficient system resources You will need e 32 MB free RAM e 50 MB disk space 50MB additional temporary space during installation 300 Mhz processor e 800x600 screen resolution 16bit color depth Insert the Topspin CD ROM into the CD ROM drive of a server that has network acce
97. utton c Click Yes when prompted File Management E You can import a file from an external FTP server export a file to an FTP server install an image in the systern or delete a file from the system Current Files on System Slot ID File Name File Type Size Date Topspin270 TopspinOS 2 2 0 build531 img image 10657535 Thu Jan 27 01 07 34 11 TopspinOS 2 2 O build530 i 6684672 Wed Jan 26 01 07 1 11 hwif_log log 34113 Thu Jan 27 01 04 47 config Wed Jan 26 01 04 14 Figure 4 6 Watching the File Copy Progress d Click Ok when the Image Install Completed Successfully message appears Configure the image for the next boot a Select Maintenance gt Boot Config 61 b Select the new image from the Image Source drop down menu and click the Apply button Boot Configuration pSoftware Images Current Image Yersion TopspinOS 2 2 O build530 Image Source for Next Reboot MASA an ey i Apply h Startup Configuration Overwrite startup configuration with O Delete startup configuration use factory defaults Figure 4 7 Set the Boot Config 62 6 he Activate the image on the node card s by enabling and disabling each node card Do not reboot the chassis as this will not result in the image being applied to the node card s a Right click a node card and select Properties b Click the down radio button and select the Apply button Card Invento
98. w primary Controller would be slot 12 and the primary Management Interface would be slot 16 Fabric Controller Modules See Figure 1 2 on page 14 for a diagram of the chassis Refer to Installing a Fabric Controller on page 44 for Fabric Controller module installation instructions The Topspin 270 Cisco SFS 7008 contains six fabric control modules The fabric controller modules perform the following roles within the system e System Controller e Switch fabric spine e Switch fabric end nodes e Stores redundant Vital Product Data VPD from the Chassis ID module This allows the chassis to continue functioning and the VPD information to be recovered even if a Chassis ID module fails Each Fabric Controller module runs the management software and contains non volatile storage The behavior and responsibility of each fabric controller module is determined by the type of slot into which it is inserted Each fabric controller module can be installed in either a Node slot or a Core slot refer to Node and Core Slots on page 18 Master Slave and Standby Roles When the CPU of a fabric module detects that is in a Core slot it will arbitrate for system mastership and run system wide control functions in addition to its local management functions e One fabric controller in the Core slot acts as the Master e One fabric controller in the Core slot acts as the Standby to the Master Fabric controllers in Node slots act as slaves

Cisco Systems 270 Switch User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents