Home

Mellanox MLNX-OS® User Manual for IBM 90Y3474

1. 38 4 24 Downgrading MLNX OS Software 39 4 2 5 Upgrading System Firmware 42 4 3 File Management uns eoo wget D e De ee C e te ae et 43 4 3 1 Saving a Configuration 43 4 3 2 Loading a Configuration 43 4 3 3 Restoring Factory Default Configuration on a Switch System Single Management Module 44 4 4 Managing Configuration 44 4 4 1 BIN Configuration 44 4 4 2 Text Configuration 44 4 5 EOCENE ein ere vebergeenbexWesqsh emi pu as om alee 45 4 515 IMOHHOL evans teat Elli ess Edu rl pi En dA Ss ke uL IE 45 4 5 2 Remote Logging is orce isere sie cee ee ag HURRY 46 4 6 Event Notifications s suse E LENS A ESCAS E RE A gus 46 4 6 1 Supported Events Maece que eq geret 46 4 6 2 SNMP Trap 48 4 6 3 Terminal Notifications 2 48 4 6 4 Email 48 4270 iBiagnostes z weit athe ES 49 4 7 1 Retrieving Return Codes when
2. ee PIPER HEAR 26 3 22 a ree let so dei te det 27 32 3 Security eX ak ER ades 28 32 4 Ports Men og s s URDU uURXerUREREEROXEREWeg gg 28 3 25 tatus 29 3 26 IB SM M gimt Ete de PINE TR TORTE p din ie 29 32 7 Kabrie Inspector eu oL RUE en peta eee 30 3 2 8 ETEEM git Its 31 Chapter 4 System 32 4 1 Management 1 32 4 1 1 Configuring Management Interfaces with Static IP Addresses 32 4 1 2 Configuring IPv6 Address the Management 32 41 3 Dynamic Host Configuration Protocol 32 4 T4 Default Gateway ossis op vuoi RARE SES 33 4 1 5 In Band 33 4 2 Software 34 4 2 1 Upgrading MLNX OS Software 34 Mellanox Technologies 3 Mellanox Technologies Confidential 1 6 9 4 2 2 Upgrading MLNX OS Software 35 4 2 3 Deleting Unused
3. 1 6 9 Step 6 Optional Set the sampling rate of the mechanism Run Step 7 Optional Set the maximum size of the data path sample Run Step 8 Optional Set the frequency in which counters are polled Run Step 9 Optional Set the maximum size of the datagrams sent to the central collector Run Step 10 Enable the sFlow agent on the desired interfaces Run 5 11 6 Verifying sFlow To verify the attributes of the sFlow agent Mellanox Technologies 82 Mellanox Technologies Confidential
4. Submenu Title Description Modules Displays a graphic illustration of the system modules By moving the mouse over the ports in the front view a pop up caption 1s displayed to indicate the sta tus of the port The port state active down is differentiated by a color scheme green for active gray black for down By moving the mouse over the rear view a pop up caption is displayed to indicate the leaf part information Mellanox Technologies Confidential 1 6 9 3 2 3 3 2 4 Mellanox Technologies 28 Table 12 System Submenus Submenu Title Description Inventory Displays a table with the following information about the system modules mod ule name type serial number ordering part number and Asic firmware version Power Management Displays a table with the following information about the system power supplies power supply name power voltage level current consumption and status A total power summary table is also displayed providing the power used the power capacity and the power available MLNX OS Upgrade Displays the installed MLNX OS images and the active partition to upload a new image and to install a new image Reboot Used to reboot the system Make sure that you save your configuration prior to clicking reboot Security Menu The Security menu makes available the following sub menus listed in order of appearance from to
5. MIB Standard Comments INET ADDRESS MIB RFC 4001 SNMPV2 CONF SNMPV2 TC RFC 2579 SNMPV2 TM RFC 3417 SNMP USM AES MIB RFC 3826 IANA LANGUAGE MIB 2591 IANA RTPROTO MIB RFC 2932 IANAifType MIB IANA ADDRESS FAMILY NUMBERS MIB Table 23 Standard MIBs Structure Management Interface and General SNMP MIB Standard Comments SNMPv2 MIB RFC 3418 SNMP FRAMEWORK MIB RFC 2571 SNMP VIEW BASED SM MIB RFC 3414 SNMP VIEW BASED ACM MIB RFC 3415 SNMP MPD MIB RFC 2572 IP MIB RFC 4293 Management interface TCP MIB RFC 4022 Management interface UDP MIB RFC 4113 Management interface IP FORWARD MIB RFC 4292 Management interface HOST RESOURCES MIB RFC 2790 Management interface HOST RESOURCES TYPE Table 24 Standard MIBs Chassis and Switch MIB Standard Comments RFC1213 MIB RFC 1213 Mellanox Technologies 54 Mellanox Technologies Confidential 1 6 9 Table 24 Standard MIBs Chassis and Switch MIB Standard Comments IF MIB RFC 2863 ifXTable only supported ENTITY MIB RFC 4133 ENTITY SENSOR MIB RFC 3433 Fan and temperature sensors ENTITY STATE MIB RFC 4268 Fan and temperature states Bridge MIB RFC 4188 doti dTpFdbGroup and dotldStaticGroup are not supported in this MIB it is supported as a part of Q Bridge MIB This MIB is not relevant to InfiniBand Q Bridge MIB RFC 4363 The following SNMP gro
6. 1 6 9 Table 19 Supported Event Notifications MIB Mapping Event Name Event Description MIB OID Comments interface down An interface s link state has changed to DOWN RFC1213 linkdown SNMPv1 Supported for Ethernet InfiniBand and management interfaces for 1U and blade systems interface up An interface s link state has changed to UP 213 linkup SNMPv1 Supported for Ethernet InfiniBand and management interfaces for 1U and blade systems internal bus error liveness failure Internal bus error A process in the system is detected as hung Mellanox EFM MIB internalBusError Not implemented low power Low power supply Mellanox EFM MIB lowPower low power recover Low power supply recover Mellanox EFM MIB too high lowPowerRecover new_root Local bridge became a Bridge MIB Supported for root bridge newRoot Ethernet paging high Paging activity has risen N A Not supported power redundancy mismatch process crash Power redundancy mis match A process in the system has crashed Mellanox EFM MIB powerRedundancyMis match Mellanox EFM MIB procCrash Supported for SX65XX only systems process exit A process in the system unexpectedly exited Mellanox EFM MIB procUnexpectedExit snmp authtrap An SNMPv3 request has failed authentication Not implemented topology change Topolo
7. Standalone Wi Virtual IP Active node Chassis master E3 Subnet Manager is not running T E 2 87 setup System Security Ports 3 Status EH B sm Q ETH System Serial Number Interfaces HA Routing Installed Licenses DNS License Hostname Key LK2d4m MU ume E 7 1 Hosts Feature EFM SX ARP Valid yes Neighbors Max num ufm ports supported 200 Virtual Switch Mgmt Active yes web EN Add New License s Email Alerts XML gateway Please enter one or more licenses each on a separate line Logs Configurations Date and Time NTP Licensing 2009 2012 Mellanox Technologies Inc Step 4 Save the configuration to complete the license installation If you do not save the installation session you will lose the installed licenses at the next system boot 2 4 3 Retrieving a Lost License Key In case of a lost MLNX OS license key contact your authorized Mellanox reseller and provide the switch s chassis serial number gt To obtain the switch s chassis serial number Step 1 Login to the switch Step 2 Retrieve the switch s chassis serial number using the command show inventory switch config show inventory Module Type Part number Serial Number CHASSIS 5 1035 5 6036 1121 02692 MGMT 5 1035 MSX6036F 1BFR MT1121X02692 FAN 5 0
8. 1 6 9 The default behavior in congestion situations 1s to drop any excessive frames that may clog the system ETS PFC and FC configurations do not apply to the destination port aa 5 10 1 5 Truncation When enabled the system can truncate the mirrored packets into smaller 64 byte packets default which is enough to capture the packets L2 and L3 headers 5 10 2 Configuring Mirroring Sessions Figure 12 presents two network scenarios with direct and remote connectivity to the analyzer equipment Direct connectivity is when the analyzer is connected to the analyzer port of the switch In this case there is no need for adding an L2 header to the mirrored traffic Remote con nectivity is when the analyzer is indirectly connected to the analyzer port of the switch In this situation adding an L2 header may be necessary depending on the network s setup Figure 12 Mirroring Session Analyzer 12 Switch Eth 1 m Eth 1 1 Eth 1 3 Eth 1 1 Eth 1 3 Host Host Host Host Switch Switch Direct local connectivity Remote connectivity gt To configure a mirroring session Step 1 Create a session Run switch config monitor session 1 This command enters a monitor session configuration mode Upon first implementa tion the command also creates the session Step 2 Add source interface s Run Switch config monitor session 1 add source interface ethernet 1 1 directio
9. FAN MSX60 FF MT1121X02722 PS1 5 0 PS N A N A CPU CPU 5 000103 1120 01027 switch config Mellanox Technologies 18 Mellanox Technologies Confidential 1 6 9 Step3 your Mellanox reseller the following information to obtain the license key The chassis serial number The type of license you need to retrieve Refer to MLNX OS Licenses on page 15 Step 4 you receive the license key you can install the license as described in the sections above Mellanox Technologies 19 J Mellanox Technologies Confidential 1 6 9 3 User Interfaces 3 1 Command Line Interface CLI MLNX OS is equipped with an industry standard CLI The CLI is accessed through SSH or Telnet sessions or directly via the console port on the front panel 1f it exists Refer to the MLNX OS Command Reference Guide for complete set of commands syntax and examples 3 1 1 Modes The CLI can be in one of following modes and each mode makes available a certain group or level of commands for execution The different CLI configuration modes are Table 8 CLI Modes and Config Context Mode Context Description Standard When the CLI is launched it begins in Standard mode This is the most restrictive mode and only has commands to query a restricted set of state information Users cannot take any actions that directly affect the system nor can they change any configu ration En
10. Step 1 Enter to Config mode Run switch switch gt enable switch configure terminal Step 2 Set your mailhub to the IP address to be your mail client s server for example Microsoft Out look exchange server switch config email mailhub IP address Step 3 Add your email address for notifications Run switch config email notify recipient email address Step 4 Configure the system to send notifications for a specific event Run Switch config email notify event event name Mellanox Technologies 48 J Mellanox Technologies Confidential 1 6 9 Step 5 Show the list of events for which an email is sent Run switch config show email events Failure events for which emails will be sent process crash process in the system has crashed unexpected shutdown Unexpected system shutdown Informational events for which emails will be sent asic chip down ASIC Chip Down cpu util high CPU utilization has risen too high cpu util ok CPU utilization has fallen back to normal levels disk io high Disk I O per second has risen too high disk io ok Disk 0 per second has fallen back to acceptable levels disk space low Filesystem free space has fallen too low switch config Step 6 Have the system send you a test email Run switch email send test The last command should generate the following email From Admin User mailto do not reply switch Se
11. 5 4 MAC Address Table eere ver DEPO 64 5 4 1 Configuring Unicast Static MAC 44 64 5 5 Spanning Tree re 64 9 91 Port Prionty zandGOst AR poke ESTA DN ESSE DES 65 52592 Port Types gura CD Uer eve vele ea e ea ERE etre 65 Mellanox Technologies 4 Mellanox Technologies Confidential 1 6 9 69 22 iei eec do 65 5 5 4 boop Guard S e eae Gu eee EE VR 66 5 5 5 Root reie utie REPERI UR Ra e 66 5 6 IGMP Snooping i cnni RR NERS de ee AEN s 66 5 6 1 Configuring IGMP Snooping 67 5 6 2 Defining a Multicast Router Port on a 67 5 7 Link Layer Discovery Protocol 68 5 7 1 Configuring LLDP iis scire dock eee eRe ee RES 68 5 8 Quality of Service 0 5 69 5 8 1 Priority Flow Control and Link Level Flow 69 5 8 2 Enhanced Transmission Selection 5 71 5 9 Access Control Last Tee Se esas EY bue eg 73 5 9 1 Configuring Access Control 1486 73 5 92 AGtlOnS cb vis tet ee teen cites es ere MD UU Eu nique
12. i osos Un ace Mis p Sie s e ph e 10 ET MENX OS EFeatures WOO IGNES RISE ES 10 Chapter2 Getting 12 2 1 Configuring the Switch for the First Time 12 2 2 Starting the Command Line 12 2 3 Starting the Web 13 ZA IACenses oon ede eode oe dde ida aed Maar t uf 15 2 4 1 Installing MLNX OSQ License 1 15 2 4 2 Installing MLNX OS License 16 2 4 3 Retrieving a Lost License 18 Chapter 3 User 20 3 1 Command Line Interface 20 3 ET GEDMOGeS EAE ea 20 3 1 2 Syntax Conventions 21 331 3 Getting Help 25 sat eh C Hb 21 3 1 4 Prompt and Response 22 3 L5 Using the no Form o dai RE e Rr RR EA 23 3c1 6 Parameter Keys 3 vs COSI Sx de 24 3 2 Web Interfaces ic voe X EVE 25 3 2 T Setup rhei
13. in these cases above follow the steps below Step 1 Run the command switch config no boot next fallback reboot enable Step 2 Set the boot partition Run switch config image boot next Step 3 Save the configuration Run switch config configuration write Mellanox Technologies 41 Mellanox Technologies Confidential 1 6 9 Step 4 Reload the system Run switch config reload 4 2 5 Upgrading System Firmware Each MLNX OS software package version has a default switch firmware version When you update the MLNX OS software to a new version an automatic firmware update process will be attempted by MLNX OS This process is described below 4 2 5 1 After Updating MLNX OS Software Upon rebooting your switch system after updating the MLNX OS software MLNX OS software will first compare its default firmware version with the currently programmed firmware versions on all the switch modules leafs and spines on director class switches or simply the switch card on edge switch systems If one or more of the switch modules is programmed with a firmware version other than the default version then MLNX OS will automatically attempt to burn the default firmware version instead If a firmware update takes place then the login process will be delayed for a few min utes aa To verify that the firmware update was successful login to MLNX OS and run the command show asic version
14. switch config vlan 10 switch config vlan 10 Step 4 Change back to config mode Run switch config vlan 10 exit switch config Step 5 Enter the interface context Run switch standalone master config interface ethernet 1 35 switch standalone master config interface ethernet 1 35 Step 6 From within the interface context configure the interface mode to Trunk Run switch standalone master config interface ethernet 1 35 switchport mode trunk switch standalone master config interface ethernet 1 35 5 3 4 Configuring Hybrid Mode VLAN Membership gt To configure Hybrid mode VLAN membership Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create a VLAN Run switch config vlan 10 switch config vlan 10 Mellanox Technologies 63 Mellanox Technologies Confidential 1 6 9 Step 4 Change back to config mode Run switch config vlan 10 exit switch config Step 5 Enter the interface context Run Switch config interface ethernet 1 35 Switch config interface ethernet 1 35 Step 6 From within the interface context configure the interface mode to Hybrid Run switch config interface ethernet 1 35 Switchport mode hybrid switch config interface ethernet 1 35 Step 7 From within the interface context configure the allowed VLAN membership Run Switch config interface ethe
15. Class 4 switch config Traffic class priorities are 0 37 where 0 is the lowest and 3 is the highest The sum of all traffic class bandwidth value percentage should be 100 otherwise the command will fail aa Mellanox Technologies 72 J Mellanox Technologies Confidential 1 6 9 Step 5 Run the command show ets to verify the configuration Switch config show dcb ets ETS enabled 1 Bandwidth Number of Traffic Class 4 switch config 5 9 Access Control List An Access Control List ACL is a list of permissions attached to an object to filter or match switches packets When the pattern is matched at the hardware lookup engine a specified action e g permit deny is applied The rule fields represent flow characteristics such as source and destination addresses protocol and VLAN ID ACL support currently allows actions of permit or deny rules and supports only ingress direc tion ACL search pattern can be taken from either L2 or L3 fields e g L2 L3 source and destina tion addresses protocol VLAN ID and priority or TCP port 5 9 1 Configuring Access Control List Access Control List ACL is configured by the user and is applied to a port once the ACL search engine matches search criteria with a received packet gt To configure ACL Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create MAC IPv
16. IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellanox TECHNOLOGIES Mellanox Technologies Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2013 Mellanox Technologies All Rights Reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale MLNX OS PhyX SwitchX UFM Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB ExtendX FabricIT Mellanox Open Ethernet Mellanox Virtual Modular Switch MetroX MetroDX ScalableHPC Unbreakable Link are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners 2 Mellanox Technologies Document Number 3956IBM Mellanox Technologies Confidential 1 6 9 Table of Contents Document R vision History About this Manual nei e RE VR RE Se ew e e ara eee mw 7 Intended Audience eo cp et eL beca es 7 Related Documentation ges uos er pentes or bes us 7 Glossary eee xc Rex we QUIDNE E OSEE PIE EN E E ERE 8 Chapter 1 Introduction
17. Log out of the CLI help View description of the interactive help system no Negate or clear certain configuration options Mellanox Technologies 21 Mellanox Technologies Confidential 1 6 9 show Display system configuration or statistics slogin Log into another system securely using ssh switch Configure switch on system telnet Log into another system using telnet terminal Set terminal parameters traceroute Trace the route packets take to a destination switch 11a596 standalone master gt If you type a legal string and then press 2 without a space character before it then you will either get a description of the command that you have typed so far or the possible command parameter completions If you press after a space character and lt gt is shown this means that what you have entered so far is a complete command and that you may press Enter carriage return to execute it Try the following to get started show show c show clock show clock show interfaces from enable mode You can also enter help to view a description of the interactive help system Note also that the CLI supports command and or parameter tab completions and their shortened forms For example you can enter instead of the enable command or cli cl instead of cli clear history In case of ambiguity more than one completion option is available that is then you can hit doubl
18. Run switch gt enable switch configure terminal Step 3 Enable IGMP snooping globally Run switch config ip igmp snooping switch config Step 4 Create a VLAN Run switch config vlan 200 switch config vlan 200 Step 5 Change back to config mode Run switch config vlan 200 exit switch config Step 6 Change the interface switchport mode of the port the interface is member of VLAN 1 by default Run switch config interface ethernet 1 36 switch config interface ethernet 1 36 switchport mode hybrid Step 7 Attach the VLAN to the port s interface Run switch config interface ethernet 1 36 switchport mode hybrid allowed vlan 200 switch config interface ethernet 1 36 Step 8 Change to config mode again Run switch config interface ethernet 1 36 exit switch config Step 9 Define the MRouter port on the VLAN Run switch config vlan 200 switch config vlan 200 ip igmp mrouter interface ethernet 1 36 switch config vlan 200 5 7 Link Layer Discovery Protocol LLDP The Link Layer Discovery Protocol LLDP is a vendor neutral Link Layer protocol in the Inter net Protocol Suite used by network devices for advertising their identity capabilities and neigh bors on a IEEE 802 LAN The protocol is formally defined in IEEE 802 1AB 5 7 1 Configuring LLDP To configure the LLDP on the switch Step 1 Log in a
19. Standalone Virtual IP Active node setup f lone E Subnet Manager is not running Hifi ETH jAi it S Lion 112721 Mgmt Save Ports Information Phy Profile Protocol Type Port channels Port Info Port number 1 Mac address 00 02 c9 72 0d 2d Port type ETH MTU 1522 bytes Port description Flow control receive off send off Admin state Disabled Actual speed 1 Gbps Operational state Down Switchport mode access Port Counters Clear Port 1 Counters RX frames 0 TX frames 0 RX unicast frames 0 TX unicast frames 0 3 2 1 Setup Menu The Setup menu makes available the following submenus listed in order of appearance from top to bottom Table 11 Setup Submenus Submenu Title Description Interfaces Used to obtain the status of configure or disable interfaces to the InfiniBand fabric Thus you can set or clear the IP address and netmask of an interface enable DHCP to dynamically assign the IP address and netmask and set inter face attributes such as MTU speed duplex etc HA Not functional Mellanox Technologies 26 Mellanox Technologies Confidential 1 6 9 3 2 2 Mellanox Technologies 27 Table 11 Setup Submenus Submenu Title Description Routing Used to set remove or display the default gateway and the static and dynamic routes Hostname Used to set or modify the
20. ar aM 74 5 10 Port Mirroring sd ack a teet ava telo e e eee datei ede nee de Mura 74 5 10 1 Mirroring Sessions 5 10 2 Configuring Mirroring 5 0 8 78 5 10 3 Verifying Mirroring 79 2 bl SSEIOQW tate cet o a ER cdd at 80 5 EE T Flow Sarnples 2 LIS RS MEC RAN qe A e 80 SA12 Statistical Samples debe ue e eR E S be 81 5 11 3 sFlow 81 5 11 4 Sampled Interfaces i i ossis RR AERA E 8l 5 11 53Configuring SEloWw ecg eke ix RASEN ENS deed age 81 5 11 6 Verifying SFIOW 22 22 22 Ree ce me ER 82 Mellanox Technologies 5 Mellanox Technologies Confidential 1 6 9 Document Revision History Table 1 Document Revision History Ethernet Document Revision Date Changes Rev 1 6 9 September Updated Section 4 6 Event Notifications on page 46 2013 Updated Section 4 4 2 Text Configuration Files on page 44 Added Section 4 8 mDNS on page 50 Rev 1 6 8 August 2013 Added Section 4 4 2 Text Configuration Files on page 44 Updated Section 2 4 Licenses on page 15 Rev 1 6 7 June 2013 Updated Section 6 2 3 Configuring OSPF on page 101 U
21. based configuration file Switch config configuration text file my filename apply When applying text based configuration file the configuration is appended to switch s existing configuration Reboot is not required 4 5 Logging 4 5 4 Monitor To print logging events to the terminal Set the modules or events you wish to print to the terminal For example run Switch config logging monitor events notice switch config logging monitor sx sdk warning These commands print system events in severity notice and sx sdk module notifications in severity warning to the screen For example in case of interface down event the following gets printed to the screen switch config Wed Jul 10 11 30 42 2013 Interface IB1 17 changed state to DOWN Wed Jul 10 11 30 43 2013 Interface IB1 18 changed state to DOWN switch config To see a list of the events refer to Table 19 Supported Event Notifications and MIB Mapping on page 46 Mellanox Technologies 45 J Mellanox Technologies Confidential 1 6 9 4 5 2 Remote Logging To configure remote syslog to send syslog messages to a remote syslog server Step 1 Enter Config mode Run switch switch gt enable switch configure terminal Step 2 Set remote syslog server Run switch config logging lt IP address gt Step 3 Set the minimum severity of the log level to info Run switch config logging lt IP address gt trap
22. be converged by the RSTP learning forwarding mechanism It converges to forwarding quickly It is recommended to configure the port type for all ports connected to hosts as edge ports normal is assumed to be connected to a switch thus it tries to be converged by the RSTP learning forwarding However if it does not receive any BPDUS it is operation ally moved to be edge e network is assumed to be connected to a switch If it does not receive any BPDUs it 1s moved to discarding state Each of these configuration options is mutually exclusive Port type is configured using the command spanning tree port type It may be applied globally on the switch Config level which configures all switch interfaces Another option 15 to configure ports individually by entering the interface s configuration mode Global configuration switch config spanning tree port type edge normal network default Interface configuration switch config interface etherent lt inf gt spanning tree port type edge normal net work 5 5 3 BPDU Filter Using BPDU filter prevents the CPU from sending receiving BPDUs on specific ports Mellanox Technologies 65 Mellanox Technologies Confidential 1 6 9 BPDU filtering is configured per interface When configured the port does not send any BPDUs and drops all BPDUS that it receives To configure BPDU filter use the following command switch config interface
23. can be run in any mode This command lists all of the switch modules along with their firmware versions Make sure that all the firmware versions are the same and match the default firmware version If the firmware update failed for one or more modules then the following warning is displayed Warning Some of the subsystems are not updated with default FW lt ver gt If you detect a firmware version mismatch for one or more modules of the switch sys tem please contact your assigned Mellanox Technologies field application engineer 4 2 5 2 Importing Firmware and Changing the Default Firmware To perform an automatic firmware update by MLNX OS for a different switch firmware version without changing the MLNX OS version import the firmware package as described below MLNX OS sets it as the new default firmware and performs the firmware update automatically as described in the previous subsections Mellanox Technologies 42 Mellanox Technologies Confidential 1 6 9 Default Firmware Change Standalone Systems Step 1 Import the firmware image tgz file Run switch config image fetch Switch config image default chip fw fw SX rel 9 1 2090 tgz Default Firmware 9 1 2090 updated Please save configuration and reboot for new FW to take effect switch config Step2 Save the configuration Run switch config configuration write switch config Step 3 Reboot the system to enable auto update
24. display and manage SL to VL mappings 3 2 7 Fabric Inspector The Fabric Inspctr menu is not applicable when the switch profile is not InfiniBand Fabric Inspctr menu requires a license LIC fabric inspector The Fabric Inspetr menu makes available the following sub menus listed in order of appear ance from top to bottom Table 17 Fabric Inspctr Submenus Submenu Title Description Summary Displays a fabric status summary including the time of last fabric update what systems are in the fabric what InfiniBand devices are identified etc IB Systems Displays information about all identified InfiniBand systems in the fabric adapt ers switches etc Mellanox Technologies 30 J Mellanox Technologies Confidential 1 6 9 3 2 8 Mellanox Technologies 31 Table 17 Fabric Inspctr Submenus Submenu Title Description IB Nodes Displays information about InfiniBand nodes in the fabric It is possible to filter display by the type of InfiniBand node HCA adapter switch etc IB Ports Displays all active InfiniBand ports in the fabric It is possible to filter display by the type of InfiniBand port HCA port switch port switch management port etc by the port rate speed or width by the Subnet Manager status on the node by node traffic etc Connections Displays all active connections in the fabric It is possible to filter display b
25. exchange Lag Aggregation Control Protocol LACP packets in order to align the functionality between both endpoints of the LAG To equally send traffic on all LAG links the switch uses a hash function which can use a set of attributes as key to the hash function As many as 16 physical ports can be aggregated on a single port channel 5 2 1 Configuring Static Link Aggregation Group LAG To configure a static LAG Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create a port channel entity Run switch config interface port channel 1 switch config interface port channel 1 Step 4 Change back to config mode switch config interface port channel 1 exit switch config Step 5 Enable LACP in the switch Run switch config lacp switch config Step 6 Add a physical port to the port channel Run switch config interface ethernet 1 4 channel group 1 mode on switch config interface ethernet 1 4 Mellanox Technologies 60 Mellanox Technologies Confidential 1 6 9 If the physical port is operationally up this port will be an active member of the aggre gation Consequently it will be able to convey traffic 5 2 2 Configuring Link Aggregation Control Protocol To configure LACP Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Ste
26. hostname Used to set or delete static hosts Note Changing hostname stamps a new HTTPS certificate DNS Used to set remove modify or display static and dynamic name servers Login Messages Used to edit the login messages Message of the Day MOTD Remote Login message and Local Login message ARP Used to add static and dynamic ARP entries and to clear the dynamic ARP cache IPSec Used to configure IPSec feature Neighbors Used to display IPv6 neighbor discovery protocol Virtual Switch Mgmt Used to set the system profile Web Used to configure Web user interface and proxy settings SNMP Used to configure SNMP attributes SNMP admin user and trap sinks Email Alerts Used to define the destination of email alerts and the recipients to be notified XML gateway Provides an XML request response protocol to get and set hardware manage ment information Logs Used to set up system log files remote log sinks and log formats Configurations Used to manage activate save and import MLNX OS SwitchX configuration files and to execute CLI commands Date and Time Used to set the date time and time zone of the switch system NTP Used to set NTP Network Time Protocol and NTP servers Licensing Used to manage MLNX OS licenses System Menu The System menu makes available the following sub menus listed in order of appearance from top to bottom Table 12 System Submenus
27. separate vir tual links on the physical link and allows any of these links to be paused and restarted independently enabling the network to create a no drop class of service for an individual virtual link PFC has 8 possible priorities 3 bits in VLAN header Each priority can be mapped to one of 4 possible queues in the ingress The PFC software offers the following features Provides per priority enabling or disabling of flow control Mellanox Technologies 69 J Mellanox Technologies Confidential 1 6 9 Transmits PFC PAUSE frames when the receive threshold for a particular traffic class is reached Provides the management capability for an administrator to configure the flow control properties on each port of the switch Keeps flow control disabled for all priorities on all ports by default Allows an administrator to enable or disable flow control per port and per priority level Supports flow control only on physical ports not on logical interfaces such as tunnels or interfaces defined by sharing a physical port in multiple virtual switch contexts Uses the configured threshold values to set up the queue buffer spaces accordingly in the datapath e Provides hardware abstraction layer callouts for the following Enabling or disabling of flow control on each port for each priority Configuring the queue depth for each priority on each port Supports MIB defined in the 802 1Qbb standard and a proprietary M
28. some text describing the error 3 1 5 Using the no Form Several Config mode commands offer the negation form using the keyword no This no form can be used to disable a function to cancel certain command parameters or options or to reset a parameter value to its default To re enable a function or to set cancelled command parameters or options enter the command without the no keyword with parameter values if necessary The following example performs the following 1 Displays the current CLI session options 2 Disables auto logout 3 Displays the new CLI session options auto logout is disabled 4 Re enables auto logout after 15 minutes 5 Displays the final CLI session options auto logout is enabled 1 Display the current CLI session options switch config show cli CLI current session settings Maximum line size 8192 Terminal width 157 columns Terminal length 60 rows Terminal type xterm Auto logout 15 minutes Paging enabled Progress tracking enabled Prefix modes enabled 2 Disable auto logout switch config no cli session auto logout 3 Display the new CLI session options switch 1 standalone master config show cli CLI current session settings Maximum line size 8192 Terminal width 157 columns Mellanox Technologies 23 Mellanox Technologies Confidential 1 6 9 Terminal length 60 rows Terminal type xterm Auto logout disabled Paging en
29. traffic per interface by loading the counter samples into sFlow datagrams The sFlow packets are encapsulated and sent in UDP over IP The UDP port number that is used is the standard 6343 by default Figure 13 sFlow Functionality Overview Collector Collector Pe sFlow agent sFlow agent Statistics Statistics Sampeling sFlow agent Sampeling Statistics Switch Switch Sampeling Switch 5 11 1 Flow Samples The sFlow agent samples the data path packet based Truncation and sampling rate are the two parameters that influence the flow samples In case of congestion the flow samples can be truncated to a predefined size before it is being assigned to the CPU The truncation can be set to any value between 64 to 256 bytes with the default being 128 bytes Furthermore the sampling rate may also be adjust as required Mellanox Technologies 80 Mellanox Technologies Confidential 1 6 9 5 11 2 Statistical Samples The sFlow agent samples interface counters time based Polling interval is configurable to any value between 5 3600 seconds with the default being 20 seconds The following statistics are gathered by the CPU Table 28 List of Statistical Counters Counter Description Total packets The number of packets that pass through sFlow enabled ports Number of flow samples The number of packets that are captured by the sampling mecha nism Number of statistic samples The number
30. two dif ferent sessions 5 LAG The source interface can be a physical interface or a LAG Port mirroring can be configured on a LAG interface but not on a LAG member When a port is added to a mirrored LAG it inherits the LAG s mirror configuration However if port mirroring configuration is set on a port that configuration must be removed prior to adding the port to a LAG interface When a port is removed from a LAG the mirror property is switched off for that port Control Protocols control protocols captured on the mirror port are forwarded to the analyzer port in addition to their normal treatment For example LACP STP and LLDP are forwarded to the analyzer port in addition to their normal treatment by the CPU Exceptions to the behavior above are the packets that are being handled by the MAC layer such as pause frames 5 10 1 2 Destination Interface The destination interface is an analyzer port is one to which mirrored traffic is sent The mirrored packets are duplicated optionally modified and sent to the analyzer port The SwitchX plat form supports up to 7 analyzer ports where any mirror port can be mapped to any analyzer port and more than a single mirror port can be mapped to a single analyzer port Packets can be forwarded to any destination using the command destination interface The analyzer port supports status and statistics as any other port LAG The destination interface cannot be a member o
31. user with greater flexibihty MLNX OS provides a full suite of management options including support for Mellanox s Uni fied Fabric Manager UFM SNMP 1 2 3 and web user interface In addition it incorporates a familiar industry standard CLI which enables administrators to easily configure and manage the system 1 1 MLNX OS Features Table 4 General System Features Feature Description Software Management Dual software image Software and firmware updates File management e FTP TFTP SCP Logging Event history log SysLog support Management Interface DHCP Zeroconf Pv6 Chassis Management Monitoring environmental controls Network Management Inter SNMP vl v2c v3 faces REST interfaces XML Gateway Security SSH Telnet RADIUS e 5 Date and Time NTP Cables amp Transceivers Transceiver info Virtual Port Interconnect VPI Ethernet InfiniBand Table 5 Ethernet Features Feature Description General Jumbo Frames 9K e ACL 24K rules permit deny e Breakout cables Mellanox Technologies 10 Mellanox Technologies Confidential 1 6 9 Table 5 Ethernet Features Feature Description Ethernet support e 48K Unicast MAC addresses e 802 1Q 4K LAG LACP 802 3ad 16 links per LAG 36 LAGs Rapid Spanning Tree
32. 4 3 File Management 4 3 1 Saving a Configuration File To save the current configuration to the active configuration file you can either use the con figuration write command requires running in Config mode or the write memory command requires running in Enable mode e To save the configuration to the active configuration file run switch config configuration write To save the configuration to a user specified file without making the new file the active configuration file run switch config configuration write to myconf no switch e save the configuration to a user specified file and make the new file the active con figuration file run Switch config configuration write to myconf To display the available configuration files and the active file run Switch config show configuration files initial myconf active switch config For further information please refer to Mellanox MLNX OS Command Reference Guide 4 3 2 Loading a Configuration File By default or after a system reset the system loads the default initial configuration file To load a different configuration file and make it the active configuration switch standalone master switch standalone master enable standalone master config configuration switch to myconfig gt gt switch standalone master configure terminal switch d switch standalone master config Fo
33. 4 ACL access list entity switch config mac access list mac acl switch config mac access list mac acl Step 4 Adda MAC IP rules to the appropriate access list switch config mac access list mac acl seq number 10 deny 0a 0a 0a 0a 0a 0a mask WIENS ISIE ie vlan 6 Cos A rococo 800 switch config mac access list mac acl Step 5 Bind the created access list to an interface slot port or port channel switch config switch config interface ethernet 1 1 switch config interface ethernet 1 1 mac port access group mac acl Mellanox Technologies 73 J Mellanox Technologies Confidential 1 6 9 5 9 2 ACL Actions An ACL action is a set of actions can be activated in case the packet hits the ACL rule gt To modify the VLAN tag of the egress traffic as part of the ACL permit rule Step 1 Create access list action profile a Create an action access list profile using the command access list action lt action profile name gt b Add rule to map a VLAN using the command vlan map lt vlan id gt within the action profile configura tion mode Step 2 Create an access list and bind the action rule a Create an access list profile using the command ipv4 mac access list b Add access list rule using the command deny permit action lt action profile name gt Step 3 Bind the access list to an interface using the command ipv4 mac port access group Create an action profile and add vlan
34. 6 Mellanox Technologies Confidential Step 7 Have the new image activate during the next boot Run Step8 Run show images to review your images Run Step 9 Save current configuration Run Step 10 Reboot the switch to run the new image Run Mellanox Technologies 37 Mellanox Technologies Confidential 1 6 9 4 2 3 Deleting Unused Images To delete unused images Step 1 Enter Config mode Run Step2 Get a list of the unused images Run Step3 Delete the unused images Run Mellanox Technologies 38 Mellanox Technologies Confidential 1 6 9 For further information please refer to Mellanox MLNX OS Command Reference Guide 4 2 4 Downgrading MLNX OS Software Prior to downgrading software please make sure the following prerequisites are met Step 1 Log into your switch via the CLI using the console port Step 2 Backup your configuration according to the following steps 1 Change to Config mode Run switch 112094 standalone master gt enable switch 112094 standalone master configure terminal switch 112094 standalone master config 2 Disable paging of CLI output Run switch 112094 standalone master config no cli default paging enable 3 Display commands to recreate current running configuration Run switch 112094 standalone master config show running config 4 Copy the output to a text fi
35. 802 1w Flow control 802 3x IGMP snooping v1 2 LLDP ETS 802 1Qaz PFC 802 1Qbb IP routing e VLAN interface ECMP OSPF Figure 1 Managing an Ethernet Fabric Using MLNX OS Ethernet Subnet Switches Servers with ConnectX and MLNX OFED or MLNX_WinOF Jmm Block Storage 31 mm File Storage Remote Management Node Mellanox Technologies 11 Mellanox Technologies Confidential 2 Getting Started The procedures described in this chapter assume that you have already installed and powered on your switch according to the instructions in the Hardware Installation Guide which was shipped with the product 2 1 Configuring the Switch for the First Time Step 1 Connect the host PC to the console mini USB port of the switch system using the supplied cable Figure 2 Console Ports Management Port Serial Port No remote IP connection is available at this stage via the external management port The internal management port can be accessed currently by the chassis management Step 2 Configure a serial terminal with the settings described below Table 6 Serial Terminal Program Configuration Parameter Setting Baud Rate 9600 Data bits 8 Stop bits 1 Parity None Flow Control None Step3 Login as admin and use admin as password 2 2 Starting the Command Line CLI Step 1 Set up an Ethernet connection betw
36. CP using the following command switch config no interface lt ifname gt dhcp Step 3 Define your interfaces statically using the following command switch config interface lt ifname gt ip address lt IP address gt lt netmask gt For further definitions of the interface please refer to Mellanox MLNX OS Command Refer ence Guide 4 1 2 Configuring IPv6 Address the Management Interface Step 1 Enable IPv6 on this interface switch config interface mgmt0 ipv6 enable Step 2 Set the IPv6 address to be configured automatically switch config interface mgmt0 ipv6 address autoconfig Step3 Verify the IPv6 address is configured correctly switch config Show interfaces mgmt0 brief 4 1 3 Dynamic Host Configuration Protocol DHCP DHCP is used for automatic retrieval of management IP addresses Mellanox Technologies 32 Mellanox Technologies Confidential 1 6 9 For all other systems and software versions DHCP is disabled by default If a user connects through SSH runs the wizard and turns off DHCP the connection 15 9 immediately terminated as the management interface loses its IP address p lt localhost gt ssh admin8192 168 10 101 Mellanox MLNX OS Switch Management Password Mellanox Switch Mellanox configuration wizard Do you want to use the wizard for initial configuration yes Step 1 Hostname switch 6287a4 Step 2 Use DHCP on mgmt0 i
37. Discovery Protocol LLDP DCBX end points exchange request and acknowledgment messages For flexibility parameters are coded in a type length value TLV format DHCP The Dynamic Host Configuration Protocol DHCP is an automatic configura tion protocol used on IP networks DNS Domain Name System A hierarchical naming system for devices in a com puter network ETS ETS provides a common management framework for assignment of band width to traffic classes FTP TFTP sFTP File Transfer Protocol FTP is a standard network protocol used to transfer files from one host to another over a TCP based network such as the Internet Gateway A network node that interfaces with another network using a different network protocol HA High Availability A system design protocol that provides redundancy of system components thus enables overcoming single or multiple failures in minimal downtime Host A computer platform executing an Operating System which may control one or more network adapters LACP Link Aggregation Control Protocol LACP provides a method to control the bundling of several physical ports together to form a single logical channel LACP allows a network device to negotiate an automatic bundling of links by sending LACP packets to the peer directly connected device that also imple ments LACP LDAP The Lightweight Directory Access Protocol is an application protocol for readin
38. Executing Remote Commands 50 48 mDNS Mete d one ee e eed m es red n 50 4 9 User Management and 50 4 9 1 Authentication Authorization and Accounting 50 4 92 55 52 4 9 3 User ACCOUD IS e e eee lg 53 4 10 Network Management 53 4 10 L SNMP x reberipRexebBen stb bU NND EE VETE 53 4 10 2 MLNX OS 59 Chapter 5 Ethernet Switching OO nterface ctas mei eet ERE 60 5 2 Link Aggregation Group 60 5 2 1 Configuring Static Link Aggregation Group 60 5 2 2 Configuring Link Aggregation Control Protocol 61 Dads ted ss etd sodio n RDUM DUNT 61 5 3 1 Configuring Access Mode and Assigning Port VLAN ID PVID 62 5 3 2 Configuring Hybrid Mode and Assigning Port VLAN ID PVID 62 5 3 3 Configuring Trunk Mode VLAN 63 5 3 4 Configuring Hybrid Mode VLAN 63
39. IB for management Provides trace logs for execution upon error conditions and for any event notifications from the hardware or datapath These trace logs are a useful aid in troubleshooting Allows the administrator to configure the minimum and maximum threshold values for flow control These configurations are applied globally on all ports and priorities Priority Based Flow Control PFC provides an enhancement to the existing pause flow control mechanism as described in 802 1X To enable PFC globally Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Enable PFC globally on the switch Run switch config dcb priority flow control enable This action might cause traffic loss while shutting down a port with priority flow con trol mode on Type yes to confirm enable pfc globally yes gt To enable PFC per priority Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Enable PFC globally on the switch Run switch config dcb priority flow control enable dcb priority flow control enable This action might cause traffic loss while shutting down a port with priority flow con trol mode on Type yes to confirm enable pfc globally yes switch config Mellanox Technologies 70 J Mellanox Technologies Confidential 1 6 9 Step 4 Choose the desirable priority you want to enabl
40. Mellanox TECHNOLOGIES Mellanox MLNX OS User Manual for IBM 90Y3474 Rev 1 6 9 Software Version 3 3 4100 www mellanox com Mellanox Technologies Confidential Rev 1 6 9 THIS HARDWARE SOFTWARE TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCTO S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED DOCUMENTATION EVEN
41. able The enable command moves the user to Enable mode This mode offers commands to view all state information and take actions like rebooting the system but it does not allow any con figurations to be changed Its commands are a superset of those in Standard mode Config Theconfigure terminal command moves the user from Enable mode to Config mode Config mode is allowed only for user accounts in the admin role or capabilities This mode has a full unrestricted set of commands to view anything take any action and change any configuration Its commands are a super set of those in Enable mode To return to Enable mode enter exitorno configure Note that moving directly from to Standard mode to from Config mode is not possible Config Interface Management Configuration mode for management interface mgmt0 mgmtl and loopback Config interface ethernet Configuration mode for Ethernet interface Config Interface Port Channel Configuration mode for Port channel LAG Config Vlan Configuration mode for VLAN Any Command Mode Several commands such as show can be applied within any context Mellanox Technologies 20 J Mellanox Technologies Confidential 1 6 9 3 1 2 Syntax Conventions To help you identify the parts of a CLI command this section explains conventions of presenting the syntax of commands Table 9 Syntax Conventions Syntax Convention Description E
42. abled Progress tracking enabled Prefix modes enabled 4 Re enable auto logout after 15 minutes switch config cli session auto logout 15 5 Display the final CLI session options Switch config show cli CLI current session settings Maximum line size 8192 Terminal width 157 columns Terminal length 60 rows Terminal type xterm Auto logout 15 minutes Paging enabled Progress tracking enabled Prefix modes enabled 3 1 6 Parameter Key This section provides a key to the meaning and format of all of the angle bracketed parameters in all the commands that are listed in this document Table 10 Angled Brackets Parameter Description Parameter Description domain A domain name e g hostname A hostname e g switch 1 lt ifname gt An interface name e g mgmt0 mgmt1 lo loopback etc lt index gt A number to be associated with aliased secondary IP addresses lt IP address gt An IPv4 address e g 192 168 0 1 lt log level gt A syslog logging severity level Possible values from least to most severe are debug info notice warning error crit alert emerg lt GUID gt Globally Unique Identifier A number that uniquely identifies a device or compo nent lt MAC address gt A MAC address The segments may be 8 bits or 16 bits at a time and may be delimited by o
43. authenti cation so use scp or sftp for that Note also that if you omit the password part you may be prompted for the password in a follow up prompt where you can type it securely without the characters being echoed This prompt will occur if the cli default prompt empty password setting is true otherwise the CLI will assume you do not want any password If you include the character this will be taken as an explicit declaration that the password is empty and you will not be prompted in any case 3 2 Web Interface MLNX OS package equipped with web interface which is a web GUI that accept input and provide output by generating webpages which can be viewed by the user using a web browser The following web browsers are supported Internet Explorer 8 0 or higher Chrome 18 or higher Mozilla Firefox 12 or higher Safari 5 or higher The web interface makes available the following perspective tabs Setup System Security Ports Status B SM Management Fabric Inspector Mellanox Technologies 25 Mellanox Technologies Confidential Ethernet Management Make sure to save your changes before switching between menus or sub menus Click the Save button to the right of Save Changes Figure 8 WebUI Mellanox 5 SX6036 Management Console TECHNOLOGIES Host switch 6287a4 User admin Logout iFabric Qy Inspctr a
44. ave the configuration to complete the license installation Run 2 42 Installing MLNX OS License Web To install an MLNX OS license via CLI Step 1 Log in as admin Step 2 Click the Setup tab and then Licensing in the left side navigation pane Figure 5 No Licenses Installed 5 SX6506 Management Console pmi C Host Switch 113dc8 User admin Logout Standalone Active node Chassis master ES Subnet Manager is not running a 1i bic ETH Save oem E rom wre Licensing System Serial Number E Z Installed Licenses Routing DNS License Hostname Key LK2 EFM SX 5P26 85G2 3488 A3MG VD3V E7U Hosts Feature EFM SX ARP Valid yes Neighbors Max num ufm ports supported 200 Virtual Switch Mgmt Active yes ET Add New License s Email Alerts XML gateway Please enter one or more licenses each on a separate line Logs Configurations Date and Time NTP Licensing 2 Add Licenses 2009 2012 Mellanox Technologies Inc Mellanox Technologies 16 Mellanox Technologies Confidential 1 6 9 Step3 Enter your license key s in the text box If you have more than one license please enter each license in a separate lin
45. ce ethernet 1 36 exit switch config 5 3 2 Configuring Hybrid Mode and Assigning Port VLAN ID PVID gt To configure Hybrid mode and assign PVID to interfaces Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create a VLAN Run switch config vlan 6 switch config vlan 6 Step 4 Change back to config mode Run switch config vlan 6 exit switch config Mellanox Technologies 62 Mellanox Technologies Confidential 1 6 9 Step 5 Enter the interface context Run Switch config interface ethernet 1 36 Switch config interface ethernet 1 36 Step 6 From within the interface context configure the interface mode to Access Run switch config interface ethernet 1 36 Switchport mode hybrid switch config interface ethernet 1 36 Step 7 From within the interface context configure the Access VLAN membership Run Switch config interface ethernet 1 36 switchport hybrid vlan 6 Switch config interface ethernet 1 36 Step 8 Change to config mode again Run switch config interface ethernet 1 36 exit switch config 5 3 3 Configuring Trunk Mode VLAN Membership gt configure Trunk mode VLAN membership Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create VLAN Run
46. chnologies switch platform is shipped with an Installation Guide document to bring up and initialize the switch platform System Hardware User Manual This document contains hardware descriptions LED assignments and hardware specifications among other things Switch Product Release Notes Please look up the relevant SwitchX based switch system series release note file MLNX OS Command Reference Guide Command Reference Guide for MLNX OS listing all of the commands available through MLNX OS with explanations and examples of these documents can be found on the Mellanox website They are available either through the product pages or through the support page with a login and password Mellanox Technologies 7 Mellanox Technologies Confidential Glossary Table 3 Glossary 1 6 9 Authentication Authorization and Accounting Authentication verifies user credentials username and password Authorization grants or refuses privileges to a user client for accessing spe cific services Accounting tracks network resources consumption by users ARP Address Resolution Protocol A protocol that translates IP addresses into MAC addresses for communication over a local area network LAN CLI Command Line Interface A user interface in which you type commands at the prompt DCB Data Center Bridging DCBX DCBX protocol is an extension of the Link Layer
47. d POST 0x40 Critical RAM memory failure Full POST 0x45 Critical NAND memory failure Full POST 0x80 Non Critical Incorrect firmware version Standard POST Oxff Non Critical POST ended successfully Standard POST 4 7 1 Retrieving Return Codes when Executing Remote Commands To stop the CLI and set the system to send return errors if some commands fail perform the following Step 1 Connect to the system from the host SSH Step2 Add the n parameter after the c1i as shown in the example below to notify the system to halt on failure and pass through the exit code ssh lt username gt lt hostname gt cli h enable show interfaces brief 48 mDNS Multicast DNS mDNS protocol is used by the SM HA to deliver control information between the InfiniBand nodes via the management interface gt In order to block sending mDNS traffic from the management interface use the following command 4 9 User Management and Security 4 9 1 Authentication Authorization and Accounting AAA AAA is a term describing a framework for intelligently controlling access to computer resources enforcing policies auditing usage and providing the information necessary to bill for services These combined processes are considered important for effective network management and secu rity The AAA feature allows you to verify the identity of grant access to and track the actions of users managing the MLNX OS switch The MLNX OS switch supports R
48. downgrade version gt 2010 09 19 16 52 50 Installed images Partition 1 lt current version gt 2010 09 19 03 46 25 Partition 2 lt current version gt 2010 09 19 03 46 25 Last boot partition 1 ext boot partition 1 boot manager password is set switch config Step 4 Install the MLNX OS image Run Switch config image install image name gt Step 1 of 4 Verify Image 100 0 Step 2 of 4 Uncompress Image 100 0 Step 3 of 4 Create Filesystems 100 0 Step 4 of 4 Extract Image 100 0 switch config Step 5 Show all image files on the system Run switch config show images Images available to be installed new image img downgrade version 2010 09 19 16 52 50 Installed images Pentacon die current version 2010 09 19 03 46 25 Partition 2 Mellanox Technologies 40 J Mellanox Technologies Confidential 1 6 9 downgrade version 2010 09 19 16 52 50 Last boot partition 1 Next boot partition 2 No boot manager password is set switch config E Step 6 Set the boot locatio
49. e Click Add Licenses after entering the last license key to install them If you wish to add another license key in the future you can simply enter it in the text box and click Add Licenses to install it Figure 6 Enter Licence Key s in Text Box Mellanox MLNX OS SX6506 Management Console Mellanox ECHNOLOGIES Host switch 113dc8 User admin Logout Standalone EylVirtualIP Active node Chassis master E Subnet Manager is not running 12222 aii titi BSM Perte 3 2122 MGMT Fabric 3 11 Security System System Serial Number HA MEE Routing Installed Licenses DNS License Hostname Key LK2 EFM SX 5P26 85G2 3488 A3MG VD3V E7U Hosts Feature EFM SX Valid yes Neighbors Max num ufm ports supported 200 Virtual Switch Mgmt Active yes Web Remove NES Add New License s Email Alerts XML gateway Please enter one or more licenses each on a separate line f Logs your license key Configurations Date and Time NTP Licensing Add Licenses 2009 2012 Mellanox Technologies Inc All installed licenses should now be displayed Mellanox Technologies 17 Mellanox Technologies Confidential 1 6 9 Figure 7 Installed License Mellanox Mellanox MLNX OS SX6506 Management Console Host switch 113dc8 User admin Logout
50. e tabs to obtain the disambiguation options Thus if you are in Enable mode and wish to learn which commands start with the letter type and click twice on the tab key to get the following switch c lt tab gt clear yl configure switch c There three commands that start with the letter c clear cli and configure 3 1 4 Prompt and Response Conventions The prompt always begins with the hostname of the system What follows depends on what com mand mode the user is in To demonstrate by example assuming the machine name is switch the prompts for each of the modes switch gt Standard mode switch Enable mode switch config Config mode Mellanox Technologies 22 Mellanox Technologies Confidential 1 6 9 The following session shows how to move between command modes switch You start in Standard mode Switch gt enable Move to Enable mode switch You are in Enable mode switch configure terminal Move to Config mode switch config 4 You are in Config mode switch config exit Exit Config mode switch You are back in Enable mode switch disable Exit Enable mode switch gt You are back in Standard mode Commands entered do not print any response and simply show the command prompt after you press lt Enter gt If an error is encountered in executing a command the response will begin with followed by
51. e using the command priority flow control priority lt pri 0 7 gt enable switch config dcb priority flow control priority 5 enable To enable PFC per interface Step 1 Log in as admin Step 2 Change to config mode Run switch gt enable Switch configure terminal Step3 Enable PFC globally on the switch Run Switch config dcb priority flow control enable Step 4 Choose the desirable priority you want to enable using the command priority flow control priority lt pri 0 7 gt enable Switch config dcb priority flow control 5 enable Step 5 Change to Interface mode Run switch config switch config interface ethernet 1 1 switch config interface ethernet 1 1 Step 6 Enable PFC for the specific interface switch config interface ethernet 1 1 dcb priority flow control mode on 5 8 2 Enhanced Transmission Selection ETS Enhanced Transmission Selection ETS provides a common management framework for assign ment of bandwidth to traffic classes for weighted round robin WRR scheduling If a traffic class does not use all the bandwidth allocated to it other traffic classes can use that available bandwidth This allows optimal utilization of the network capacity while prioritizing and provid ing the necessary resources The ETS feature has the following attributes ETS global admin Enable default scheduling mode is WRR according to the configured bandwidth pe
52. ed by configuring mirroring sessions session is an association of a mirror port or more and an analyzer port Figure 10 Mirror to Analyzer Mapping Mirror Port Session Analyzer Port A mirroring session is a monitoring configuration mode that has the following parameters Table 27 Mirroring Parameters Parameter Description Access Source interface s List of source interfaces to be mirrored RW Destination interface A single analyzer port through which all mirrored traffic egress RW Header format The format and encapsulation of the mirrored traffic when sent to analyzer RW Truncation Enabling truncation segments each mirrored packet to 64 bytes RW Congestion control Controls the behavior of the source port when destination port is congested RW Admin state Administrative state of the monitoring session RW 5 10 1 1 Source Interface The source interface mirror port refers to the interface from which the traffic is monitored Port mirroring does not affect the switching of the original traffic The traffic is simply duplicated and sent to the analyzer port Traffic in any direction either ingress egress or both can be mirrored Mellanox Technologies 75 Mellanox Technologies Confidential 1 6 9 There is no limitation on number of source interfaces mapped to a mirroring session Ingress and egress traffic flows of a specific source interface can be mapped to
53. een the switch and a local network machine using a stan dard RJ 45 connector Mellanox Technologies 12 Mellanox Technologies Confidential 1 6 9 Step2 Start a remote secured shell SSH to the switch using the command ssh 1 lt username gt switch ip address gt rem machl gt ssh 1 username ip address gt Mellanox MLNX OS Switch Management Last login Thu Apr 28 11 24 13 2011 from 192 168 10 1 Mellanox Switch switch gt Step 3 Step 4 Login to the switch default username is admin password admin Once you get the prompt you are ready to use the system Refer to MLNX OS Command Refer ence Guide for additional information on the CLI commands 2 3 Starting the Web Interface gt To start a WebUI connection to the switch platform Step 1 Set up an Ethernet connection between the switch and a local network machine using a stan dard RJ 45 connector Step 2 Open web browser Internet Explorer 7 0 Chrome or Mozilla Firefox 3 0 Note Make sure the screen resolution is set to 1024 768 or higher Step3 in the IP address of the switch or its DNS name in the format http switch IP address Step 4 Login to the switch default user name is admin password admin The following figure shows an example of the login window for remote management of the switch Mellanox Technologies 13 J Mellanox Technologies Confidential 1 6 9 Figure 3 MLNX OS Login W
54. eived during a session Accounting is carried out by logging of session statistics and usage information and is used for authorization control billing trend analysis resource utilization and capacity planning activities Authentication authorization and accounting services are often provided by a dedicated AAA server a program that performs these functions Network access servers interface with AAA servers using the Remote Authentication Dial In User Service RADIUS protocol For information on the AAA commands please refer to Mellanox MLNX OS Command Refer ence Guide 4 9 1 1 RADIUS RADIUS Remote Authentication Dial In User Service widely used in network environments is a client server protocol and software that enables remote access servers to communicate with a central server to authenticate dial in users and authorize their access to the requested system or service It is commonly used for embedded network devices such as routers modem servers switches and so on RADIUS is currently the de facto standard for remote authentication It is prevalent in both new and legacy systems It is used for several reasons RADIUS facilitates centralized user administration e RADIUS consistently provides some level of protection against an active attacker For information on the RADIUS commands please refer to Mellanox MLNX OS Command Ref erence Guide 4 9 1 2 TACACS TACACS Terminal Access Controller Access Control System wide
55. emote Access Dial In Mellanox Technologies 50 Mellanox Technologies Confidential 1 6 9 User Service RADIUS Terminal Access Controller Access Control device Plus TACACS protocols Authentication authentication provides the initial method of identifying each individ ual user typically by entering a valid username and password before access is granted The AAA server compares a user s authentication credentials with the user credentials stored in a database If the credentials match the user is granted access to the network or devices If the credentials do not match authentication fails and network access is denied Authorization following the authentication a user must gain authorization for per forming certain tasks After logging into a system for instance the user may try to issue commands The authorization process determines whether the user has the authority to issue such commands Simply put authorization is the process of enforcing policies determining what types or qualities of activities resources or services a user is permit ted Usually authorization occurs within the context of authentication Once you have authenticated a user they may be authorized for different types of access or activity Accounting the last level is accounting which measures the resources a user con sumes during access This includes the amount of system time or the amount of data a user has sent and or rec
56. empReset Check fans and environmental temperature asicOverTemp Check fans and environmental temperature lowPower Add connect power supplies internalBusError N A procCrash Generate SysDump and contact Mellanox support cpuUtilHigh N A procUnexpectedExit Generate SysDump and contact Mellanox support diskSpaceLow Clean images sysDump files using the commands image delete and file debug dump delete systemHealthStatus Refer to Health Status table lowPowerRecover N A insufficientFans Check Fans and environmental conditions insufficientFansRecover N A insufficientPower Add connect power supplies or change power mode using the command power redundancy mode insufficientPowerRecover N A For additional information refer to MELLANOX EFM MIB For event to MIB mapping please refer to Table 19 Supported Event Notifications and MIB Mapping on page 46 Mellanox Technologies 56 J Mellanox Technologies Confidential 1 6 9 4 10 1 4 Configuring SNMP gt To set up the SNMP Step 1 Activate the SNMP server on the MLNX OS switch in configure mode using the following commands 4 10 1 5 Configuring an SNMPv3 User gt To configure SNMP V3 user Step 1 Configure the user using the command where userrole admin auth type md5 or sha e priv type des or aes 128 Step 2 Enter authentication password and its confirmation Step 3 Enter privacy password and its confirmati
57. etherent lt inf gt spanning tree bpdufilter enable disable Configuring BPDU filtering a port connected to a switch can cause bridging loops because the port filters any BPDU it receives and goes to forwarding state aa 5 5 4 Loop Guard Loop guard is a feature that prevents loops in the network When a blocking port in a redundant topology transitions to the forwarding state accidentally an STP loop occurs This happens when BPDUs are no longer received by one of the ports in a physically redundant topology Loop guard is useful in switched networks where devices are connected point to point A desig nated bridge cannot disappear unless it sends an inferior BPDU or brings the link down on a point to point connection The loop guard configuration is only allowed on network port type aa If loop guard is enabled and the port does not receive BPDUs the port is put into an inconsistent state blocking until the port starts to receive BPDUs again A port in the inconsistent state does not transmit BPDUs If BPDUs are received again loop guard alters its inconsistent state condi tion STP converges to a stable topology without the failed link or bridge after loop guard isolates the failure Disabling loop guard moves all loop inconsistent ports to listening state To configure loop guard use the following command switch config interface etherent lt inf gt spanning tree guard loop 5 5 5 Root Guard Con
58. f LAG when the header format is local Control Protocols The destination interface may also operate in part as a standard port receiving and sending out non mirrored traffic When the header format is configured as a local port ingress control proto col packets that are received by the local analyzer port get discarded Advanced MTU Considerations The analyzer port like its counterparts is subject to MTU configuration It does not send packets longer than configured When the analyzer port sends encapsulated traffic the analyzer traffic has additional headers and therefore longer frame The MTU must be configured to support the additional length otherwise the packet is truncated to the configured MTU Mellanox Technologies 76 J Mellanox Technologies Confidential 1 6 9 The system on the receiving end of the analyzer port must be set to handle the egress traffic If it Is not it might discard it and indicate this in its statistics packet too long 5 10 1 3 Header Format Ingress traffic from the source interface can be manipulated in several ways depending on the network layout using the command header format If the analyzer system is directly connected to the destination interface then the only parameters that can be configured on the port are the MTU speed and port based flow control Priority flow control is not supported is this case However if the analyzer system is indirectly connected to the destinatio
59. figuring root guard on a port prevents that port from becoming a root port A port put in root inconsistent blocked state if an STP convergence is triggered by a BPDU that makes that port a root port The port is unblocked after the port stops sending BPDUs To configure loop guard use the following command switch config interface etherent lt inf gt spanning tree guard root 5 6 IGMP Snooping The Internet Group Multicast Protocol IGMP is a communications protocol used by hosts and adjacent routers on IP networks to establish multicast group memberships The host joins a mul Mellanox Technologies 66 J Mellanox Technologies Confidential 1 6 9 ticast group by sending a join request message towards the network router and responds to que ries sent from the network router by dispatching a join report A given port can be either manually configured to be a router port or it can be dynamically man ifested when having received a query hence the network router is connected to this port IGMP Snooping Control packets received from hosts joins leaves are forwarded to the router port and the router port will update its multicast group data base accordingly Each dynamically learnt multicast group will be added to all of the router ports on the switch As many as 5K multicast groups can be created on the switch 5 6 1 Configuring IGMP Snooping You can configure IGMP snooping to establish multicast group members
60. g and editing directories over an IP network MAC A Media Access Control address MAC address is a unique identifier assigned to network interfaces for communications on the physical network segment MAC addresses are used for numerous network technologies and most IEEE 802 network technologies including Ethernet Mellanox Technologies 8 Mellanox Technologies Confidential Table 3 Glossary 1 6 9 Maximum Transfer Unit The maximum size of a packet payload not including headers that can be sent received from a port Network Adapter A hardware device that allows for communication between computers in a network PFC FC Priority Based Flow Control applies pause functionality to traffic classes OR classes of service on the Ethernet link RADIUS Remote Authentication Dial In User Service A networking protocol that enables AAA centralized management for computers to connect and use a net work service RDMA Remote Direct Memory Access Accessing memory in a remote side without involvement of the remote CPU RSTP Rapid Spanning Tree Protocol A spanning tree protocol used to prevent loops in bridge configurations RSTP is not aware of VLANS and blocks ports at the physical level SA Subnet Administrator The interface for querying and manipulating subnet management data SCP Secure Copy or SCP is a means of securely transferring computer files between a
61. gy change trig gered by a local bridge Bridge MIB topolo gyChange Supported for Ethernet unexpected shutdown Unexpected system shut down Mellanox EFM MIB unexpectedShutdown To send use the CLI com mand snmp server notify send test N A Send a testing event Reset occurred due to over heating of ASIC testTrap Mellanox EFM MIB asicOverTempReset Not supported Mellanox Technologies 47 Mellanox Technologies Confidential 1 6 9 Table 19 Supported Event Notifications MIB Mapping Event Name Event Description MIB OID Comments temperature too high Temperature 1s too high Mellanox EFM MIB asicOverTemp 4 6 2 SNMP Trap Notifications To set SNMP notification see Section 4 10 1 6 Configuring an SNMP Notification on page 58 4 6 3 Terminal Notifications To print events to the terminal Set the events you wish to print to the terminal Run switch config logging monitor events notice This command prints system events in the severity notice to the screen For example in case of interface down event the following gets printed to the screen switch config Wed Jul 10 11 30 42 2013 Interface IB1 17 changed state to DOWN Wed Jul 10 11 30 43 2013 Interface IB1 18 changed state to DOWN switch config 4 6 4 Email Notifications gt To configure MLNX OS to send you emails for all configured events and failures
62. he volume of RSTP in Mellanox switches It adds a set of capabilities related to the behavior of ports in different segments of the network For example the required behavior of a port connected to a non switch entity such as Mellanox Technologies 64 Mellanox Technologies Confidential 1 6 9 host is to converge quickly while the required behavior of a port connected to switch entity 15 to converge based on the RSTP parameters Additionally it adds security issues on a port and switch basis allowing the operator to deter mine the state and role of a port or the entire switch should an abnormal event occur For exam ple If a port is configured to be root guard the operator will not allow it to become a root port under any circumstances regardless of any BPDU that will have been received on the port 5 5 1 Port Priority and Cost When two ports on a switch are part of a loop the STP port priority and port path cost configura tion determine which port on the switch is put in the forwarding state and which port is put in the blocking state To configure port priority use the following command switch config interface etherent lt inf gt spanning tree port priority 0 240 To configure port path cost use the following command switch config interface etherent lt inf gt spanning tree cost lt 1 200000000 gt 5 5 2 Port Type Port type has the following configuration options edge is not assumed to
63. hips gt To configure IGMP snooping Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Enable IGMP snooping globally Run switch config ip igmp snooping switch config Step 4 Enable IGMP snooping on a VLAN Run switch config vlan 2 switch config vlan 2 ip igmp snooping 5 6 2 Defining a Multicast Router Port on VLAN You can define a Multicast Router MRouter port on a VLAN in one of the following methods gt To change the Interface Switchport to Trunk Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Enable IGMP snooping globally Run switch config ip igmp snooping switch config Step 4 Change the interface switchport mode of the port the interface is member of VLAN 1 by default Run switch config interface ethernet 1 1 switch config interface ethernet 1 1 switchport mode trunk Step 5 Change back to config mode Run switch config interface ethernet 1 1 exit switch config Mellanox Technologies 67 Mellanox Technologies Confidential 1 6 9 Step 6 Define the MRouter port on the VLAN Run switch config vlan 2 switch config vlan 2 ip igmp mrouter interface ethernet 1 1 switch config vlan 2 gt To change the Interface Switchport to Hybrid Step 1 Log in as admin Step 2 Enter config mode
64. ified premium features This section is relevant only to switch systems with an internal management capability The following licenses are offered with MLNX OS software Table 7 MLNX OS Licenses OPN Valid on product Description UPGR 6012 GW 5 6012 Ethernet L2 L3 Gateway UPGR 6012F 56E 5 6012 56GbE link speed UPGR 1012 GW 5 1012 InfiniBand Ethernet L3 Gateway UPGR 1012 56E SX1012 56GbE link speed UPGR 6018 GW 5 6018 Ethernet L2 L3 Gateway UPGR 6018F 56E 5 6018 56GbE link speed UPGR 6036 GW 5 6036 Ethernet L2 L3 Gateway UPGR 6036F 56E SX6036 56GbE link speed UPGR 1036 GW SX1036 InfiniBand Ethernet L3 Gateway UPGR 1036F 56E SX1036 56GbE link speed UPGR 1024 GW SX1024 InfiniBand Ethernet L3 Gateway UPGR 1024 56E SX1024 56GbE link speed LIC fabric inspector SX6036F T InfiniBand fabric inspector monitoring and health SX65XX 2 4 4 Installing MLNX OS License CLI gt To install an MLNX OS license via CLI Step 1 Login as admin and change to Config mode switch gt enable switch config terminal Step 2 Install the license using the key Run switch config license install lt license key gt Mellanox Technologies 15 J Mellanox Technologies Confidential 1 6 9 Step 3 Display the installed license s using the following command Make sure that the Valid and Active fields both indicate yes Step 4 S
65. in order of appearance from top to bottom Table 15 Status Submenus Submenu Title Description Summary Displays general information about the switch system and the MLNX OS image including current date and time hostname uptime of system system memory CPU load averages etc Profile and Capabilities Displays general information about the switch system capabilities such as the enabled profiles e g IB ETH and their corresponding values Temperature Provides a graphical display of the switch module sensors temperature levels over time 1 hour It is possible to display either the temperature level of one module s sensor or the temperature levels of all the module sensors together Power Supplies Provides a graphical display of one of the switch s power supplies voltage level over time 1 hour Fans Provides a graphical display of fan speeds over time 1 hour The display is per fan unit within a fan module CPU Load Provides a graphical display of the management CPU load over time 1 hour Memory Provides a graphical display of memory utilization over time 1 day Network Provides a graphical display of network usage transmitted and received packets over time 1 day It also provides per interface statistics Logs Displays the system log messages It 1s possible to display either the currently saved system log or a continuous system log Maintenance Used t
66. indow Mellanox MLNX OS Management Console O Host switch Sea580 User not logged in Login Please enter your username and password then click Login Mellanox MLNX OS Switch Management Best viewed using Firefox Chrome IE 7 or higher at 1024x768 resolution or higher 2009 2012 Mellanox Technologies Inc After you log in to MLNX OS a default status summary window will be displayed containing the following information Figure 4 Display After Login Mellanox MLNX OS SX1016 Management Console Host switch SeOaee User admin Logout Standalone L PA sooo IP sytem EE Ei Iove lE E Pj System zzz Ports Status Cpblty c 14 MGMT i Security summary System Capabilitie Date and Time 2012 02 07 16 04 06 J EU Hostname switch Se0aee Power Supplies Uptime 4h 36m 475 Fans Version SX_PPC_M460EX 3 0 0000 dev HA 2012 02 06 08 49 05 ppc CPU Load Model ppc AS Host ID 0002c95e0aee ty System memory 372 MB used 1655 MB free 2027 MB total Network CPU load averages 0 41 0 36 0 25 Logs Maintenance Active alerts Alerts No alerts 2009 2012 Mellanox Technologies Inc Mellanox Technologies 14 Mellanox Technologies Confidential 1 6 9 2 4 Licenses MLNX OS software package can be extended with premium features Installing a license allows you to access the spec
67. info Step 4 Override the log levels on a per class basis Run switch config logging lt IP address gt trap override class lt class name gt For further information please refer to Mellanox MLNX OS Command Reference Guide 4 6 Event Notifications MLNX OS features a variety of supported events Events are printed in the system log file and optionally can be sent to the system administrator via email SNMP trap or directly prompted to the terminal 4 6 1 Supported Events The following table presents the supported events and maps them to their relevant MIB OID Table 19 Supported Event Notifications and MIB Mapping Event Name Event Description MIB OID Comments asic chip down ASIC chip down Mellanox EFM MIB Not supported asicChipDown cpu util high CPU utilization has risen Mellanox EFM MIB too high cpuUtilHigh disk space low File system free space has Mellanox EFM MIB fallen too low diskSpaceLow health module status Health module status Mellanox EFM MIB changed systemHealthStatus insufficient fans Insufficient amount offans Mellanox EFM MIB in system insufficientFans insufficient fans recover Insufficient amount offans Mellanox EFM MIB in system recovered insufficientFansRecover insufficient power Insufficient power supply Mellanox EFM MIB insufficientPower Mellanox Technologies 46 Mellanox Technologies Confidential
68. lc switch standalone master config Step 3 Verify what keys exist in the host Run switch standalone master config show ssh client SSH client Strict Hostkey Checking ask SSH Global Known Hosts Entry 1 myserver erante leu eolit STE 2 loll 535 06 No SSH user identities configured No SSH authorized keys configured switch standalone master config 4 9 3 User Accounts There are two user account types admin and monitor As admin the user is privileged to execute all the available operations As monitor the user can execute operations that display system con figuration and status or set terminal settings Table 21 User Roles Accounts and Default Passwords User Role Default Password admin admin monitor monitor 4 10 Network Management Interfaces 4 10 1 SNMP Simple Network Management Protocol SNMP is a network protocol for the management of a network and the monitoring of network devices and their functions SNMP supports asynchro nous event trap notifications and queries MLNX OS supports SNMP versions vl 2 and v3 SNMP trap notifications Standard MIBs Mellanox private MIBs EHCM MIB Mellanox Technologies 53 J Mellanox Technologies Confidential 1 6 9 4 10 1 1 Standard MIBs Table 22 Standard MIBs Textual Conventions and Conformance MIBs
69. le 4 2 4 1 Downloading Image Step 1 Log into the system to obtain the serial number Run switch 112094 standalone master config show inventory Step 2 Download the requested MLNX OS version from the following link http support mellanox com SupportWeb Step 3 Enter your username and password when prompted Step4 Log into the switch via the CLI using the console port Step 5 Change to Config mode Run switch gt enable switch configure terminal switch config Step 6 Delete all previous images from the Images available to be installed prior to fetching the new image Run switch config image delete image EFM M405EX ppc m405ex 20090531 190132 img Step 7 Fetch the requested software image Run switch config image fetch scp username password 192 168 10 125 var www htm1 lt image_name gt 100 05 AE AE AE AAE EAE AE AE AE E AAE EEE AE AE E AE E EEEE AE E AE E EAE EAE AEE AE E EAE EAEE Mellanox Technologies 39 J Mellanox Technologies Confidential 1 6 9 4 2 4 2 Downgrading Image The procedure below assumes that booting and running is done from Partition 1 and the downgrade procedure is performed on Partition 2 5 Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Show all image files on the system Run switch config show images Images available to be installed new_image img lt
70. local and a remote host or between two remote hosts It is based on the Secure Shell SSH protocol SNMP Simple Network Management Protocol A network protocol for the manage ment of a network and the monitoring of network devices and their functions NTP Network Time Protocol A protocol for synchronizing computer clocks in a network SSH Secure Shell A protocol program for securely logging in to and running pro grams on remote machines across a network The program authenticates access to the remote machine and encrypts the transferred information through the connection syslog A standard for forwarding log messages in an IP network 5 Terminal Access Controller Access Control System Plus networking proto col that enables access to a network of devices via one or more centralized servers TACACS provides separate AAA services XML Gateway Extensible Markup Language Gateway Provides an XML request response protocol for setting and retrieving HW management information Mellanox Technologies 9 Mellanox Technologies Confidential 1 6 9 1 Introduction Mellanox Operating System MLNX OS enables the management and configuration of Mel lanox Technologies SwitchX silicon based switch platforms MLNX OS supports the Virtual Protocol Interconnect VPI technology which enables it to be used for both Ethernet and Infini Band technology providing the
71. lowing components A protocol with a frame format that utilizes TCP over IP Acentralized server that stores all the user authorization information Aclient in this case the switch Each entry in the LDAP server is referenced by its Distinguished Name DN The DN consists of the user account name concatenated with the LDAP domain name If the user account name is John the following is an example DN uid John ou people dc domain dc com For information on the LDAP commands please refer to Mellanox MLNX OS Command Refer ence Guide 4 9 2 Secure Shell SSH t is recommended not to use more than 100 concurrent SSH sessions to the switch 4 9 2 1 Adding Host and Providing SSH Key gt To add entries to the global known hosts configuration file and its SSH value perform the following steps Step 1 Change to Config mode Run Switch standalone master enable Switch standalone master configure terminal switch standalone master config Mellanox Technologies 52 Mellanox Technologies Confidential 1 6 9 Step 2 Add an entry to the global known hosts configuration and its SSH value Run switch standalone master config ssh client global known host myserver ssh rsa AAAAB3NzaClyc2EAAAABIWAAAIEAsXeklqc8TOEN2mnMcVcfhueaRYzIVqt4rVsrERIjmlJh4mkYYIa8hGGikNa t5xw2dRrNxnHYLK51bUsSGlZNwZTlDpme3pAZeMY7G4Z2MgGIW9xOuaXgAA3eBeoUj Fdi6 41BqchWk0nTb 4gMf1 MK heQNns7AtTrvqg O5ry
72. ly used in network environ ments is a client server protocol that enables remote access servers to communicate with a cen tral server to authenticate dial in users and authorize their access to the requested system or service It is commonly used for providing NAS Network Access Security NAS ensures secure access from remotely connected users TACACS implements the TACACS Client and provides the AAA Authentication Authorization and Accounting functionalities Mellanox Technologies 51 Mellanox Technologies Confidential 1 6 9 is used for several reasons e Facilitates centralized user administration Uses TCP for transport to ensure reliable delivery Supports inbound authentication outbound authentication and change password request for the authentication service Provides some level of protection against an active attacker For information on the TACACS commands please refer to Mellanox MLNX OS Command Reference Guide 4 9 1 3 LDAP LDAP Lightweight Directory Access Protocol is an authentication protocol that allows a remote access server to forward a user s logon password to an authentication server to determine whether access can be allowed to a given system LDAP is based on a client server model The switch acts as a client to the LDAP server A remote user the remote administrator interacts only with the switch not the back end server and database LDAP authentication consists of the fol
73. mapping action switch config access list action my action switch config access list action my action vlan map 20 switch config access list action my action exit Create an access list and bind rules switch config mac access list my list switch config mac access list my list permit any any action my action switch config mac access list my list exit Bind an access list to a port Switch config interface ethernet 1 1 Switch config interface ethernet 1 1 mac port access group my list 5 10 Port Mirroring Port mirroring enables data plane monitoring functionality which allows the user to send an entire traffic stream for testing Port mirroring sends a copy of packets of a port s traffic stream called mirrored port into an analyzer port Port mirroring is used for network monitoring It can be used for intrusion detection security breaches latency analysis capacity and performance matters and protocol analysis Figure 9 provides an overview of the mirroring functionality Figure 9 Overview of Mirroring Functionality Analyzer Egress Mirrored Ingress Mirrored Port Copy Port Copy Modify 1 7 opt Mellanox Technologies 74 Mellanox Technologies Confidential 1 6 9 There is no limitation on the number of mirroring sources and more than a single source can be mapped to a single analyzer destination 5 10 1 Mirroring Sessions Port mirroring is perform
74. n both Mellanox Technologies 78 Mellanox Technologies Confidential 1 6 9 Step 3 Add destination interface Run Step 4 Optional Set header format Run Ill Step 5 Optional Truncate the mirrored traffic to 64 byte packets Run Step 6 Optional Set congestion control Run 0 aul Step 7 Enable the session Run 5 10 3 Verifying Mirroring Sessions gt To verify the attributes of a specific mirroring session Mellanox Technologies 79 Mellanox Technologies Confidential 1 6 9 Source interfaces Interface direction 1 1 both To verify the attributes of running mirroring sessions Switch config show monitor session summary Session Admin Status Mode Destination Source 1 Enable Up add eth eth1 2 eth1 1 b 2 Disable Down add vlan eth1 2 eth1 8 i pol B Enable Up add eth eth1 5 eth1 18 e 7 Disable Down local 5 11 sFlow sFlow ver 5 is a procedure for statistical monitoring of traffic in networks MLNX OS supports an sFlow sampling mechanism agent which includes collecting traffic samples and data from counters The sFlow datagrams are then sent to a central collector The sampling mechanism must ensure that any packet going into the system has an equal chance of being sampled irrespective of the flow to which it belongs The sampling mechanism provides the collector with periodical information on the amount and load of
75. n interface there are two options for switching the mirrored data to the analyzer system VLAN tag may be added to the Ethernet header of the mirrored traffic An Ethernet header be added with include a new destination address and VLAN tag It must be taken into account that adding headers increases packet size Figure 11 Header Format Options Source Frame Mirror Frame local Source Frame Mirror Frame add vlan Source Frame Mirror Frame add ethernet header Source Frame Mirror Frame add ethernet header DA sa WE Data DA SA osio ossa DA sa Data 5 10 1 4 Congestion Control The destination ports might receive pause frames that lead to congestion in the switch port In addition too much traffic directed to the analyzer port for example 40GbE mirror port is directed into 10G analyzer port might also lead to congestion In case of congestion When best effort mode is enabled on the analyzer port SwitchX drops excessive traffic headed to the analyzer port using tail drop mechanism however the regular data mir rored data heading to its original port does not suffer from a delay or drops due to the analyzer port congestion When the best effort mode on the analyzer port 1s disabled the SwitchX does not drop the excessive traffic This might lead to buffer exhaustion and data path packet loss Mellanox Technologies 77 Mellanox Technologies Confidential
76. n to be the other partition next Run Switch config image boot next There are two installed images the system Therefore 1f one of the images gets cor rupted due to power interruption for example in the next reboot the image will go up e from the second partition In case you are downloading to an older software version which has never been run yet on the switch use the following command sequence as well switch config no boot next fallback reboot enable switch config configuration write Step 7 Reload the switch Run switch config reload 4 2 4 3 Switching to Partition with Older Software Version The system saves a backup configuration file when upgrading from an older software version to a newer one If the system returns to the older software partition it uses this backup configuration file Note that all configuration changes done with the new software are lost when returning to the older software version There are 2 instances where the backup configuration file does not exist The user has run reset factory command which clears all configuration files in the system The user has run configuration switch to to a configuration file with different name then the backup file Also note that the configuration file becomes empty if the switch 1s downgraded to a software version which has never been installed yet To allow switching partition to the older software version
77. nt Sunday May 01 2011 11 17 AM To lt name gt Subject System event on switch Test email for event notification System information Hostname switch Version lt version gt 2011 05 01 14 56 31 Date 2011 05 01 08 17 29 Uptime 17h 8m 28 0605 This is a test email Done For further information please refer to Mellanox MLNX OS Command Reference Guide 47 Diagnostics Switch Power On Self Test As the switch powers on it begins the Power On Self Test POST a series of tests as part of its power up procedure to ensure that the switch functions properly During the POST the switch logs any errors encountered Some POST errors are criti cal others are not The updated POST diagnostic code will be stored inside the POST Diagnostic Register Mellanox Technologies 49 Mellanox Technologies Confidential 1 6 9 Table 20 lists the POST return codes and their meanings Table 20 POST Return Codes Return Code Severity Meaning POST Section 0 5 Critical System initialization failure Standard POST 0x10 Critical Failure connecting to the main man Standard POST agement process 0x15 Critical VPD initialization failure Standard POST 0x20 Critical CPLD initialization failure Standard POST 0x25 Critical Default IP configuration failure Standard POST 0x30 Critical Temperature sensors failure Extended POST 0x35 Critical Voltage sensors failure Extende
78. nterface yes no lt localhost gt In such case the serial connection should be used 4 1 44 Default Gateway In order to configure manually the default gateway use the ip route command with 0 0 0 0 as prefix and mask The next hop address must be within the range of one of the IP interfaces on the system switch config ip route 0 0 0 0 0 0 0 0 10 209 0 2 switch config show ip route Destination Mask Gateway Interface Source default 018080790 105200902 mgmt 0 static 10 209 0 0 250m 751 0050 0 mgmt0 direct switch config 4 1 5 In Band Management In band management is a management path passing through the data ports In band management be created over one of the VLANs in the systems The in band management feature does not require any license However it works only for system profiles VPI and Ethernet It cannot be enabled with IP Routing or IP Proxy ARP gt To set an in band management channel Step 1 Create a VLAN Run switch config vlan 10 switch config vlan 10 Step 2 Create a VLAN interface Run switch config interface vlan 10 create Step 3 Enter the VLAN interface configuration mode and configure L3 attributes Run switch config interface vlan10 switch config interface vlanl0 ip address 10 10 10 10 24 Mellanox Technologies 33 J Mellanox Technologies Confidential 1 6 9 Step 4 Optional Verify in band managemen
79. o perform specific maintenance operations automatically on a predefined schedule Alerts Used to display a list of the recent health alerts and enables the user to configure health settings IB SM Mgmt The IB SM MGMT menu is not supported in Ethernet systems Mellanox Technologies Confidential 1 6 9 The SM Mgmt menu makes available the following sub menus listed in order of appearance from top to bottom Table 16 IB SM Mgmt Submenus Submenu Title Description Summary Displays the local Subnet Manager SM status running time failures etc Base SM Used to manage basic SM configuration enabling SM priority level and restor ing initial configuration Advanced SM Used to manage basic SM configuration enabling SM priority level and restor ing initial configuration Expert SM Used to configure security and GUID based prefixes m key sm key key etc and to manage special SM attributes that should not be changed except by expert users of the Subnet Manager who understand the risks of manipulating these attributes Compute nodes Used to add compute nodes using network adapter port GUIDs Root nodes Used to add root nodes using switch GUIDs Partitions Manages partition keys sets removes or displays the partition keys Basic Qos Used to configure basic QoS attributes such as default QoS settings and VL arbitration low and high entries It is also used to
80. o print event notifications to the terminal SSH or CONSOLE refer to Section 4 5 1 Monitor on page 45 5 4 10 2 MLNX OS XML MLNX OS XML API is an additional option to manage the system besides SNMP The XML gateway provides an XML request response protocol that can be used by end user tools to get and set management information on the appliance The service can be accessed over HTTP or HTTPS and then it uses the existing web authentication mechanism It can also be accessed over SSH and then it uses the existing CLI authentication mechanism XML Gateway Management information base For further information please contact Mellanox support Mellanox Technologies 59 Mellanox Technologies Confidential 1 6 9 5 Ethernet Switching 5 1 Interface Interface Ethernet have the following physical set of configurable parameters Admin state enabling or disabling the interface Flow control admin state per direction send or receive MTU Maximum Transmission Unit 1518 9216 bytes e Speed 1 10 40 56GbE depends on the interface type and system Description user defined string Module type the type of the module plugged in the interface Step 6 5 2 Link Aggregation Group LAG Link Aggregation protocol describes a network operation in which several same speed links are combined into a single logical entity with the accumulated bandwidth of the originating ports LAG groups
81. of allowed VLANs sent from the Hybrid port is sent tagged while traffic sent with PVID is untagged Mellanox Technologies 61 Mellanox Technologies Confidential 1 6 9 Trunk Trunk port is a port connecting 2 switches It accepts only tagged frames with VLANs of which the port is a member On egress traffic sent from the Trunk port is tagged By default a Trunk port is automatically a member on all current VLANs 5 3 1 Configuring Access Mode and Assigning Port VLAN ID PVID To configure Access mode and assign PVID to interfaces Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Create VLAN Run switch config vlan 6 switch config vlan 6 Step 4 Change back to config mode Run switch config vlan 6 exit switch config Step 5 Enter the interface context Run Switch config interface ethernet 1 36 Switch config interface ethernet 1 36 Step 6 From within the interface context configure the interface mode to Access Run Switch config interface ethernet 1 36 switchport mode access switch config interface ethernet 1 36 Step 7 From within the interface context configure the Access VLAN membership Run switch config interface ethernet 1 36 switchport access vlan 6 switch config interface ethernet 1 36 Step 8 Change back to config mode Run switch config interfa
82. of statistical samples Number of discarded samples The number of samples that were discarded Number of datagrams The number of datagrams that were sent to the collector 5 11 3 sFlow Datagrams The sFlow datagrams contain flow samples and statistical samples The sFlow mechanism uses IP protocol therefore if the packet length is more than the interface MTU it becomes fragmented by the IP stack The MTU may also be set manually to anything in the range of 200 9216 bytes The default is 1400 bytes 5 11 4 Sampled Interfaces sFlow must be enabled on physical or LAG interfaces that require sampling When adding a port to a LAG sFlow must be disabled on the port If a port with enabled sFlow is configured to be added to a LAG the configuration is rejected Removing a port from a LAG disables sFlow on the port regardless of the LAG s sFlow status 5 11 5 Configuring sFlow gt configure the sFlow agent Step 1 Unlock the sFlow commands Run switch config protocol sflow Step 2 Enable sFlow on the system Run Switch config sflow enable Step3 Enter sFlow configuration mode Run Switch config sflow switch config sflow Step 4 Set the central collector s IP Run switch config sflow collector ip 10 10 10 10 Step 5 Set the agent ip used in the sFlow header Run switch config sflow agent ip 20 20 20 20 Mellanox Technologies 81 Mellanox Technologies Confidential
83. on Mellanox Technologies 57 Mellanox Technologies Confidential 1 6 9 retrieve the system table run the following SNMP command snmpwalk v3 l authPriv a MD5 u admin A lt Authentication password gt x DES X lt privacy password system ip SNMPv2 MIB system 4 10 1 6 Configuring an SNMP Notification gt To set up the SNMP Notification traps or informs follow the next steps Step 1 Make sure SNMP and SNMP notification are enable Run switch config snmp server enable Switch config snmp server enable notify switch config Step 2 Configure SNMP host with the desired arguments IP Address SNMP version authentication methods More than one host can be configured Each host may have different attributes Run Switch config snmp server host 10 134 47 3 traps version 3 user my username auth sha my password switch config Step3 Verify the SNMP host configuration Run switch config show snmp host Notifications enabled yes Default notification community public Default notification port 162 Notification sinks 10 134 47 3 1 yes Port 162 default otification type SNMP v3 trap Username my username Authentication type sha Privacy type aes 128 Authentication password set Privacy password set switch config Step 4 Configure the desired event to be sent via SNMP Run switch config snmp server notify even
84. p 3 Create a port channel entity Run switch config interface port channel 1 switch config interface port channel 1 Step 4 Change back to config mode Run switch config interface port channel 1 exit switch config Step 5 Enable LACP in the switch Run switch config lacp switch config Step 6 Add a physical port to the port channel Run switch config interface ethernet 1 4 channel group 1 mode active passive switch config interface ethernet 1 4 5 3 VLANs A Virtual Local Area Network VLAN is an L2 segment of the network which defines a broad cast domain and is identified by a tag added to all Ethernet frames running within the domain This tag is called a VLAN ID VID and can take a value of 1 4094 Each port can have a switch mode of either Access Access port is a port connected to a host It can accept only untagged frames and assigns them a default configured VLAN Port VLAN ID On egress traffic sent from the access port is untagged e Access dcb This mode is Mellanox specific that receives ingress untagged traffic but sends egress priority tag VLAN ID 0 Hybrid Hybrid port is a port connected to either switches or hosts It can receive both tagged and untagged frames and assigns untagged frames a default configured VLAN Port VLAN ID It receives tagged frames with VLANs of which the port is a member these VLANs names are allowed On egress traffic
85. p to bottom Table 13 Security Submenus Submenu Title Description Users Used to manage setting up removing modifying user accounts Admin Password Used to modify the system administrator password SSH Used to display and generate host keys AAA Used to configure AAA Authentication Authorization and Accounting secu rity services such as authentication methods and authorization Login Attempts Used to manage login attempts RADIUS Used to manage Radius client TACACS Used to manage TACACS client LDAP Used to manage LDAP client Certificate Used to manage certificates Ports Menu The Ports menu displays the port state and enables some configuration attributes of a selected port It also enables modification of the port configuration A graphical display of traffic over time last hour or last day through the port is also available Table 14 Ports Submenus Submenu Title Description Ports Manages port attributes counters transceiver info and displays a graphical coun ters histogram Mellanox Technologies Confidential 1 6 9 3 2 5 3 2 6 Mellanox Technologies 29 Table 14 Ports Submenus Submenu Title Description Phy Profile Provides the ability to manage phy profiles Protocol type Manages the link protocol type Status Menu The Status menu makes available the following sub menus listed
86. pdated Section 4 5 Logging on page 45 Updated Table 19 Supported Event Notifications and MIB Mapping on page 46 Rev 1 6 6 Apr 2013 Added Section 4 1 5 In Band Management on page 33 Added Section 4 6 1 Supported Events on page 46 Updated Section 5 5 Spanning Tree on page 64 Rev 1 6 4 Mar 2013 Added Section 5 10 Port Mirroring on page 74 Added Section 5 11 sFlow on page 80 Rev 1 5 4 Sep 2012 Updated Section 4 2 4 Downgrading MLNX OS Software on page 39 Rev 1 5 2 June 2012 Updated Section 4 10 1 SNMP on page 53 Rev 1 5 1 May 2012 Added Section 5 7 Link Layer Discovery Protocol LLDP on page 68 Rev 1 5 May 2012 Initial document Mellanox Technologies 6 J Mellanox Technologies Confidential 1 6 9 About this Manual This manual provides general information concerning the scope and organization of this User s Manual Intended Audience This manual is intended for network administrators who are responsible for configuring and managing Mellanox Technologies SwitchX based Switch Platforms Related Documentation The following table lists the documents referenced in this User s Manual Table 2 Reference Documents Document Name Description InfiniBand Architecture Specification Vol 1 The InfiniBand Architecture Specification that is pro Release 1 2 1 vided by IBTA Switch Installation Guide Each Mellanox Te
87. r So you could say 11 22 33 44 55 66 1122 3344 5566 11 22 33 44 55 66 or 1122 3344 5566 lt netmask gt A netmask e g 255 255 255 0 or mask length prefixed with a slash e g 24 These two express the same information in different formats Mellanox Technologies 24 Mellanox Technologies Confidential 1 6 9 Table 10 Angled Brackets Parameter Description Parameter Description network prefix An IPv4 network prefix specifying a network Used in conjunction with a net mask to determine which bits are significant e g 192 168 0 0 regular expression An extended regular expression as defined by the grep in the man page The value you provide here is passed on to grep E node id gt ID of a node belonging to a cluster This is a numerical value greater than zero cluster id gt A string specifying the name of a cluster lt port gt TCP UDP port number lt TCP port gt A TCP port number in the full allowable range 0 65535 lt URL gt A normal URL using any protocol that wget supports including http https ftp sftp and tftp or a pseudo URL specifying an scp file transfer The scp pseudo URL format is scp username password hostname path filename Note that the path is an absolute path Paths relative to the user s home directory are not currently supported The implementation of ftp does not support
88. r traffic class Disable scheduling mode is Strict Priority SP Bandwidth percentage for each traffic class By default each traffic class gets an equal share The default mapping of priority to traffic classes per interface is as follows Priority 0 1 mapped to tc 0 Priority 2 3 mapped to tc 1 Priority 4 5 mapped to tc 2 Priority 6 7 mapped to tc 3 ETS is enabled by default scheduling is WRR gt To set the scheduling mode to Strict Priority Mellanox Technologies 71 Mellanox Technologies Confidential 1 6 9 Step 1 Log in as admin Step 2 Enter config mode Run switch enable switch configure terminal Step 3 Run the command dcb ets disable switch config no dcb ets enable gt To configure the WRR bandwidth percentage Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Make sure ETS feature is enabled Run switch config dcb ets enable Step 4 Choose the WRR bandwidth rate and distribution By default the WRR distribution function is equal 25 per TC Changing the WRR bandwidth rate will cause a change in the distribution function for example if you wish to schedule more traffic on TC 0 TC 1 TC 2 while reducing the amount of traffic sent on TC 3 run the com dcb ets tc bandwidth switch config dcb ets tc bandwidth 30 30 30 10 show dcb ets ETS enabled TE Bandwidth Number of Traffic
89. r further information please refer to Mellanox MLNX OS Command Reference Guide Mellanox Technologies 43 Mellanox Technologies Confidential 1 6 9 4 3 3 Restoring Factory Default Configuration on a Switch System Single Management Module In cases where the system configuration becomes corrupted it is suggested that you restore the factory default configuration Step 1 Connect to the IBM chassis manager and move into the correct Switch blade context Step 2 Run the command clear cnfg for more assistance please refer to the IBM CMM User Manual Step 3 Wait for the switch blade to reboot itself twice It might take a few minutes between one reboot and the other Please avoid using the system during that time For further information please refer to Mellanox MLNX OS Command Reference Guide 4 4 Managing Configuration Files There are two types of configuration files that can be applied on the switch BIN files binary and text based configuration files 4 4 1 BIN Configuration Files BIN configuration files are not human readable and cannot be edited To create a new BIN configuration file switch config configuration new my filename To upload a BIN configuration file from a switch to an external file server switch config configuration upload my filename scp root my server root tmp my filename To fetch a BIN configuration file switch config configuration fetch scp roo
90. rnet 1 35 Switchport hybrid allowed vlan add 10 Switch config interface ethernet 1 35 Step 8 Change to config mode again Run switch config interface ethernet 1 35 exit switch config 5 4 MAC Address Table 5 41 Configuring Unicast Static MAC Address You can configure static MAC addresses for unicast traffic This feature improves security and reduces unknown unicast flooding gt To configure Unicast Static MAC address Step 1 Log in as admin Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Run the command mac address table static unicast destination mac address gt vlan lt vlan identifier 1 4094 gt interface ethernet lt slot gt port switch config mac address table static unicast 00 11 22 33 44 55 vlan 1 interface ethernet 0 1 5 5 Spanning Tree The operation of Rapid Spanning Tree Protocol RSTP provides for rapid recovery of connectiv ity following the failure of a bridge bridge port or a LAN The RSTP component avoids this delay by calculating an alternate root port and immediately switching over to the alternate port if the root port becomes unavailable Thus using RSTP the switch immediately brings the alternate port to forwarding state without the delays caused by the listening and learning states The RSTP component conforms to IEEE standard 802 1D 2004 RSTP enhancements is a set of functions added to increase t
91. s admin Mellanox Technologies 68 J Mellanox Technologies Confidential 1 6 9 Step 2 Enter config mode Run switch gt enable switch configure terminal Step 3 Enable LLDP globally on the switch Run switch config lldp switch config Step 4 Enable LLDP per interface Run switch config interface ethernet 1 1 lldp receive switch config interface ethernet 1 1 lldp transmit Step 5 Show LLDP local information Run switch config show lldp local LLDP is Enabled Local global configuration Chassis sub type macAddress 4 Chassis id 00 11 22 33 44 55 System Name switch 111111 System Description my system description Supported capabilities B Supported capabilities enabled B Step 6 Show LLDP remote information Run switch config show lldp interfaces ethernet 1 1 remote Ethernet 1 1 Remote Index 1 Remote chassis id 00 11 22 33 44 55 chassis id subtype mac Remote port id ethenret 1 2 port id subtype local Remote port description ethernet 1 2 Remote system name remote system Remote system description remote system description Remote system capabilities supported B B 5 8 Quality of Service QoS 5 8 1 Priority Flow Control and Link Level Flow Control Priority Flow Control PFC provides an enhancement to the existing pause mechanism in Ether net The current Ethernet pause option stops all traffic on a link PFC creates eight
92. t configuration Run switch config show interfaces vlanl0 Interface 1 10 status Comment Admin up yes Link up yes DHCP running no P address 10 10 10 10 Netmask 255 5990 Pv6 enabled no Speed Duplex N A nterface type ethernet nterface source vlan MTU 1500 HW address 00 02 C9 75 24 58 RX bytes 0 TX bytes 250 RX packets 0 TX packets 3 RX mcast packets 0 TX discards 0 RX discards 0 TX errors 0 RX errors 0 TX overruns 0 RX overruns 0 TX carrier 0 RX frame 0 TX collisions 0 TX queue len 0 switch config Step 5 6lISorappropriate 4 22 Software Management 4 2 1 Upgrading MLNX OS Software Preconditions Prior to upgrading MLNX OS software from version 3 2 0100 and lower please remove any old configuration from your system To remove old configuration Step 1 Clear your system of any old configuration Run from CMM system switch 2 gt clear cnfg OK system switch 2 gt Step2 Follow the steps described in Section 4 2 2 Upgrading MLNX OS Software on page 35 Mellanox Technologies 34 Mellanox Technologies Confidential 1 6 9 4 2 2 Upgrading MLNX OS Software When upgrading from a software version older than 3 2 0100 to software version 3 3 0000 or higher the upgrade procedure must be done in two steps First update the software to 3 2 0300 100 for InfiniBand platforms or 3 2 0506 for Ethernet plat forms then update to the desired software
93. t interface up switch config This particular event is used as an example only gt Mellanox Technologies 58 J Mellanox Technologies Confidential 1 6 9 Step 5 Verify the list of traps and informs being sent to out of the system Run switch config show snmp events Events for which traps will be sent asic chip down ASIC Chip Down cpu util high CPU utilization has risen too high disk space low Filesystem free space has fallen too low health module status Health module Status insufficient fans Insufficient amount of fans in system insufficient fans recover Insufficient amount of fans in system recovered insufficient power Insufficient power supply interface down An interface s link state has changed to down interface up An interface s link state has changed to up internal bus error Internal bus I2C Error liveness failure A process in the system was detected as hung low power Low power supply low power recover Low power supply Recover new root local bridge became a root bridge paging high Paging activity has risen too high power redundancy mismatch Power redundancy mismatch process crash process in the system has crashed process exit process in the system unexpectedly exited snmp authtrap An SNMP v3 request has failed authentication topology change local bridge trigerred a topology change unexpected shutdown Unexpected system shutdown switch config order t
94. t my server root tmp my filename gt To see the available configuration files Switch config show configuration files initial active my filename Active configuration initial Unsaved changes no switch config gt To load a BIN configuration file switch config configuration switch to my filename 4 4 2 Text Configuration Files Text configuration files are text based and editable Mellanox Technologies 44 Mellanox Technologies Confidential 1 6 9 gt To create new text based configuration file switch config configuration text generate active running save my filename To apply a text based configuration file Switch config configuration text file my filename apply may result in unpredictable behavior It is therefore suggested to first clear the switch s configuration by applying a specific configuration file following the procedure in Sec tion 4 4 1 or by resetting the switch back to factory default 3 Applying text based configuration file to an existing running data port configuration A To upload a text based configuration file from a switch to an external file server switch config configuration text file my filename upload scp root my server root tmp my filename To fetch a text based configuration file from an external file server to a switch switch config configuration text fetch scp root my server root tmp my filename To apply a text
95. ups are not sup ported qBridgeVlanStatisticsGroup qBridgeVlanStatisticsOverflowGroup e qBridgeVlanHCStatisticsGroup qBridgeLearningConstraintsGroup The following SNMP tables are not sup ported dotlqTpFdbTable dynamic UC MAC addresses e dotlqTpGroupTable dynamic MC MAC addresses dotlqForwardAllTable GMRP dotlqForwardUnregisteredTable GMRP dotlqVlanCurrentTable GVRP This MIB is not relevant to InfiniBand RSTP MIB RFC 4318 This MIB is not relevant to InfiniBand LLDP MIB 802 1 AB 2005 This MIB is not relevant to InfiniBand 4 10 1 2 Private MIB Table 25 Private MIBs Supported MIB Comments MELLANOX SMI MIB Mellanox Private MIB main structure no objects MELLANOX PRODUCTS MIB List of OID per managed system sysObjID MELLANOX IF VPI MIB IfTable extensions MELLANOX EFM MIB Deprecated MIB based on Mellanox MIB Traps definitions are supported MELLANOX ENTITY MIB Enhances the standard ENTITY MIB contains GUID and ASIC revision Mellanox private MIBs can be downloaded from the Mellanox Support webpage Mellanox Technologies 55 J Mellanox Technologies Confidential 1 6 9 4 10 1 3 Mellanox Private Traps The following private traps are supported by MLNX OS Table 26 SNMP Traps Trap Action Required asicChipDown Reboot the system asicOverT
96. version To upgrade MLNX OS software on your system perform the following steps Step 1 Change to Config mode switch enable Switch configure terminal switch config Step 2 Obtain the previously available image img file You must delete this image in the next step to make room for fetching the new image switch config show images Installed images Partition 1 SX PPC 460 SX 3 3 3130 2013 03 20 21 32 25 ppc Partition 2 SX 460 SX 3 3 3130 2013 03 20 21 32 25 ppc Images available to be installed image PPC 460 5 3 3 3256 i1mg SX 460 SX 3 3 3256 2013 03 20 21 32 25 ppc Serve image files via HTTP HTTPS no No image install currently in progress Boot manager password is set No image install currently in progress Require trusted signature in image being installed yes default switch config Step 3 Delete the old image that is listed under Images available to be installed prior to fetching the new image Use the command image delete for this purpose switch config image delete image PPC_M460EX SX_3 0 1224 img switch config Step 4 Fetch the new software image switch config image fetch scp username password 192 168 10 125 var www htm1 lt image_name gt Mellanox Technologies 35 J Mellanox Technologies Confidential 1 6 9 Step 5 Display the available images Step 6 Install the new image Mellanox Technologies 3
97. xample 7 Angled brackets Indicate a value variable that lt 1 65535 gt or switch inter must be replaced face Square brackets Enclose optional parameters destination ip destination port However only one parameter out destination mac of the list of parameters listed can be used The user cannot have a combination of the parameters unless stated otherwise Braces Enclose alternatives or variables mode active on passive that are required for the parameter in square brackets Vertical bars Identify mutually exclusive active on passive choices Do not type the angled or square brackets vertical bar or braces in command lines This guide uses these symbols only to show the types of entries CLI commands and options in lowercase and are case sensitive For example when you enter the enable command enter it all in lowercase It cannot ae be ENABLE or Enable Text entries you create are also case sensitive 3 1 3 Getting Help You may request context sensitive help at any time by pressing 2 on the command line This will show a list of choices for the word you are on or a list of top level commands if you have not typed anything yet For example if you are in Standard mode and you type at the command line then you will get the following list of available commands switch gt e Configure CLI shell options enable Enter enable mode exit
98. y the link type switch to switch switch to HCA etc and by the link rate speed or width System Names Allows the mapping of System Names to GUIDs to ease system identification ETH Mgmt Eth Mgmt menu is not applicable when the switch profile is not ethernet The ETH Mgmt menu makes available the following sub menus listed in order of appearance from top to bottom Table 18 ETH Mgmt Submenus Submenu Title Description Spanning Tree Used for configuring and monitoring spanning tree protocol MAC Table Used for configuring static mac addresses in the switch and displaying the mac address table Link Aggregation Used for configuring and monitoring aggregated Ethernet links LAG as well as configuring LACP VLAN Used for managing the switch VLAN table IGMP Snooping Used for managing IGMP snooping in the switch ACL Used for managing Access Control in the switch Mellanox Technologies Confidential 1 6 9 4 System Management 4 1 Management Interface 4 1 1 Configuring Management Interfaces with Static IP Addresses If your switch system was set during initialization to obtain dynamic IP addresses through DHCP and you wish to switch to static assignments perform the following steps Step 1 Change to Config mode Run switch gt switch enable switch configure terminal Step 2 Disable setting IP addresses using the DH

Mellanox MLNX-OS® User Manual for IBM 90Y3474

Contents

Download Pdf Manuals

Related Search

Related Contents

Mellanox MLNX-OS&reg; User Manual for IBM 90Y3474

Contents

Download Pdf Manuals

Related Search

Related Contents

Mellanox MLNX-OS® User Manual for IBM 90Y3474