Supermicro AOC-CIBF-m1
Contents
1. Powered connectors for optical and active cable support InfiniBand e IBTA Specification 1 2 1 compliant e Hardware based congestion control e 16 million I O channels e 256 to 4Kbyte MTU 1Gbyte messages Enhanced InfiniBand e Hardware based reliable transport Oollective operations offloads GPU communication acceleration Hardware based reliable multicast Extended Reliable Connected transport Enhanced Atomic operations Ethernet EEE Std 802 3ae 10 Gigabit Ethernet EEE Std 802 3ba 40 Gigabit Ethernet EEE Std 802 3ad Link Aggregation and Failover EEE Std 802 3az Energy Efficient Ethernet EEE Std 802 1Q 1p VLAN tags and priority 1 2 Chapter 1 Overview IEEE Std 802 1Qau Congestion Notification IEEE P802 1Qaz D0 2 ETS IEEE P802 1Qbb D1 0 Priority based Flow Control Jumbo frame support 9 6KB Hardware based I O Virtualization Single Root IOV Address translation and protection Dedicated adapter resources Multiple queues per virtual machine Enhanced QoS for vNICs VMware NetQueue support Manageability Features Additional CPU Offloads RDMA over Converged Ethernet TCP UDP IP stateless offload Intelligent interrupt coalescence Flexboot Technology Remote boot over InfiniBand Remote boot over Ethernet Remote boot over iSCI 1 3 SUPERO AOC CIBF m1 Add on Card User s Guide Protocol Support Open MPI OSU MVAPICH Intel MPE MS MPI PLATFORM MPI e TCP UDP2. 3 Dual USB 2 0 Ports USB Ports Two USB 2 0 ports are located on the add on card Refer to the add on card layout on page 2 1 for the the location of the USB ports QSFP Port A single QSFP port is located on the add on card Connect a QSFP cable to the port to provide Infiniband 56Gb s and Ethernet 40Gbp s communica tion Refer to the add on card layout on Page 2 1 for the location of the QSPF port QSFP PCB LEDs There are two LEDs located on the PCB between the QSFP port and the Mellanox controller refer to the add on card layout on Page 2 1 for the location of the LEDs A blinking yellow LED indicates activity A solid green yellow or off LED indicates the type of link up connection See the table to the right for more information 2 2 SUPERO AOC CIBF m1 Add on Card User s Guide 2 3 Connectors and PCB LEDs QSFP PCB LEDs LED Color Definition Activity Yellow Port activity Blinking Link Yellow Logical Link up Green Physical Link up Off No Connection Chapter 3 Installation 3 1 Chapter 3 Installation Static Sensitive Devices Electrostatic Discharge ESD can damage electronic components To avoid dam aging your add on card it is important to handle it very carefully The following measures are generally sufficient to protect your equipment from ESD Precautions Use a grounded wrist strap designed to prevent static discharge Touch a
3. 598EB 2 CX4 10Gb port 6 6 16 7cm x 3 9 9 8cm UIBF m1 UIO LP PCI E x8 Mellanox ConnectX 3 1 QSFP 56Gb port 5 6 14 0cm x 2 5 6 4cm UINF m2 UIO LP PCI E x8 Mellanox ConnectX 2 2 CX4 20Gb port 5 5 14 0cm x 2 5 6 4cm UIBQ m1 UIO LP PCI E x8 Mellanox ConnectX 2 1 QSFP 40Gb port 5 6 14 3cm x 2 5 UIBQ m2 UIO LP PCI E x8 Mellanox ConnectX 2 2 QSFP 40Gb port 5 6 14 3cm x 2 5 6 4cm CGP i2 MicroLP PCI E x4 Intel amp i350 2 RJ45 1Gb port 4 5 11 3cm x 1 3 3 4cm CG i2 MicroLP PCI E x4 Intel 82580 2 RJ45 1Gb port 4 5 11 3cm x 1 3 4cm CIBF m1 MicroLP PCI E x8 Mellanox ConnectX 3 1 QSFP 56Gb port 4 5 11 3cm x 1 3 3 4cm CTG i1S MicroLP PCI E x8 Intel 82599EN 1 SFP 10Gb port 4 5 11 3cm x 1 3 3 4cm Headquarters Address Tel Fax Email Website Europe Address Tel Fax Email Asia Pacific Address Tel Fax Website Email Tel Preface Contacting Supermicro Super Micro Computer Inc 980 Rock Ave San Jose CA 95131 U S A 1 408 503 8000 1 408 503 8008 marketing supermicro com General Information support supermicro com Technical Support www supermicro com Super Micro Computer B V Het Sterrenbeeld 28 5215 ML s Hertogenbosch The Netherlands 31 0 73 6400390 31 0 73 6416525 sales supermicro nl General Information support supermicro nl Technical Support rma gsupermicro nl
4. Customer Support Super Micro Computer Inc 4F No 232 1 Liancheng Rd Chung Ho Dist New Taipei City 235 Taiwan R O C 886 2 8226 3990 886 2 8226 3991 www supermicro com tw support supermicro com tw Technical Support 886 2 8226 5990 Technical Support SUPERO AOC CIBF M1 Add on Card User s Guide Table of Contents Preface Chapter 1 Overview 1 1 Overview 12 Key Feature Sirc ied ei deep da du b na tum deco ERR Han aya da d k 1 1 3 SpeciiCatlonS 5 eet otn EC Gore beg ties 1 1 Chapter 2 Hardware Components 2 1 Add On Card Image and LaAyoOUt EEEE kk kk kk kk 2 1 2 2 Major Compol lliS y na err aly kan an aya rb yana tee Xel e rd perta 2 1 2 9 Connectors and PCB LEDS sya c sal3 s l yar xan ban eH edi haero RS ERE Wela 2 2 USB Ports QSFP ce l WEN 2 2 OSFP PCB WEDS ys xuna ek akinesia nannies 2 2 Chapter 3 Installation 3 1 Static SerisitiveDeVviC8s nie ipd ner n dada Hurke nay aka diker kek ene ok kh daka 3 1 PRECAUTIONS J 3 1 l nore le po RNN DU ea cess cae tonsa eae steve E E eseeeeeaee een 3 1 3 2 Add On Card Brackets s 3 2 3 3 Be rore nstallatiOn 5 ct rri ete per ene stead cen EE Or cedere A YEN EA PET Sn 3 2 3 4 Installing the Add onm Gard iain re ette er tice eus 3 3 3 5 Installing the Windows Operating System ii ee 3 3 3 6 Ins
5. EoIB IPoIB SDP RDS e SRP iSER NFS RDMA uDAPL Operating Systems Distributions Novell SLES Red Hat Enterprise Linux RHEL and other Linux distributions Microsoft Windows Server 2008 CCS 2003 HPC Server 2008 e OpenFabrics Enterprise Distribution OFED e OpenFabrics Windows Distribution WinOF VMware ESX Server 3 5 vSphere 4 0 4 1 Physical Dimensions Card PCB dimensions without end brackets 12 32cm 4 85in x 3 90cm 1 54in LxW Operating Condition Operating temperature 0 C to 55 C 32 F to 131 F Compliance Environmental RoHS RoHS Compliant 6 6 Pb Free QD 1 4 Chapter 1 Overview Supported Platforms Supermicro Twin Server Systems with microLP expansion slot see table below MicroLP Add On Card AOC CG i2 AOC CGP i2 AOC CIBF m1 AOC CTG i1S f Notes 1 1U Bracket 2 2U Bracket This product is only available as an integrated solution with Supermicro server systems For the most current product information visit www supermicro com 1 5 SUPERO AOC CIBF m1 Add on Card User s Guide Notes Chapter 2 Hardware Components Chapter 2 Hardware Components 2 1 Add On Card Image and Layout The AOC CIBF m1 Image Mellanox nnectX 3 Controller 2 2 Major Components The following major components are installed on the AOC CIBF m1 1 Mellanox ConnectX 3 FDR Controller 2 QSFP Quad Small Form Factor Pluggable Port
6. SUPERO AOC CIBF M1 User s Guide Revision 1 0 The information in this User s Guide has been carefully reviewed and is believed to be accurate The vendor assumes no responsibility for any inaccuracies that may be contained in this document and makes no commitment to update or to keep current the information in this user s guide or to notify any person or organization of the updates Please Note For the most up to date version of this user s guide please see our Website at www supermicro com Super Micro Computer Inc Supermicro reserves the right to make changes to the product described in this user s guide at any time and without notice This product including software and documentation is the property of Supermicro and or its licensors and is supplied only under a license Any use or reproduction of this product is not allowed except as expressly permitted by the terms of said license IN NO EVENT WILL SUPER MICRO COMPUTER INC BE LIABLE FOR DIRECT INDIRECT SPECIAL INCIDENTAL SPECULATIVE OR CONSEQUENTIAL DAMAGES ARISING FROM THE USE OR INABILITY TO USE THIS PRODUCT OR DOCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES IN PARTICULAR SUPER MICRO COMPUTER INC SHALL NOT HAVE LIABILITY FOR ANY HARDWARE SOFTWARE OR DATA STORED OR USED WITH THE PRODUCT INCLUDING THE COSTS OF REPAIRING REPLACING INTEGRATING INSTALLING OR RECOVERING SUCH HARDWARE SOFTWARE OR DATA Any disputes arising between the manuf
7. acturer and the customer shall be governed by the laws of Santa Clara County in the State of California USA The State of California County of Santa Clara shall be the exclusive venue for the resolution of any such disputes Supermicro s total liability for all claims will not exceed the price paid for the hardware product FCC Statement This equipment has been tested and found to comply with the limits for a Class A digital device pursuant to Part 15 of the FCC Rules These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment This equipment generates uses and can radiate radio frequency energy and if not installed and used in accordance with the manufacturer s instruction manual may cause harmful interference with radio communications Operation of this equipment in a residential area is likely to cause harmful interference in which case you will be required to correct the interference at your own expense California Best Management Practices Regulations for Perchlorate Materials This Perchlorate warning applies only to products containing CR Manganese Dioxide Lithium coin cells Perchlorate Material special handling may apply See www dtsc ca gov hazardouswaste perchlorate WARNING Handling of lead solder materials used in this product may expose you to lead a chemical known to the State of California to cause birth defects and other reproductive h
8. ard you have received may or may not look exactly the same as the graphics shown in this user s guide Returning Merchandise for Service A receipt or copy of your invoice marked with the date of purchase is required before any warranty service will be rendered You can obtain service by calling your ven dor for a Returned Merchandise Authorization RMA number When returning the motherboard to the manufacturer the RMA number should be prominently displayed on the outside of the shipping carton and the shipping package is mailed prepaid or hand carried Shipping and handling charges will be applied for all orders that must be mailed when service is complete For faster service You can also request a RMA authorization online http www supermicro com This warranty only covers normal consumer use and does not cover damages in curred in shipping or from failure due to the alternation misuse abuse or improper maintenance of products During the warranty period contact your distributor first for any product problems SUPER AOC CIBF M1 Add on Card User s Guide Conventions Used in the User s Guide Pay special attention to the following symbols for proper system installation and to prevent damage to the system or injury to yourself Warning Important information given to ensure proper system installation or to prevent damage to the components or injury to yourself Note Additional information given to differentiate between var
9. arm User s Guide Revision 1 0 Release Date July 11 2012 Unless you request and receive written permission from Super Micro Computer Inc you may not copy any part of this document Information in this document is subject to change without notice Other products and companies referred to herein are trademarks or registered trademarks of their respective companies or mark holders Copyright 2012 by Super Micro Computer Inc All rights reserved Printed in the United States of America Preface Preface About this User s Guide This user s guide is written for system integrators PC technicians and knowledgeable PC users It provides information for the installation and use of the AOC CIBF M1 add on card About this Add on Card The AOC CIBF m1 is the most compact powerful InfiniBand adapter on the market Based on Mellanox ConnectX 3 with Virtual Protocol Interconnect VPI it provides the most high performance flexible interconnect solution for servers used in enter prise data centers and high performance computing The AOC CIBF m1 simplifies system development by providing both InfiniBand 56Gb s and Ethernet 40Gb s support in one hardware design The AOC CIBF m1 is designed in a small microLP form factor to fit within Supermicro Twin server systems An Important Note to the User All images and layouts shown in this user s guide are based upon the latest PCB Revision available at the time of publishing The c
10. ccordingly Supermicro disclaims any and all liability and should buyer use or sell such products for use in such ultra hazardous applications it does so entirely at its own risk Furthermore buyer agrees to fully indemnify defend and hold Supermicro harmless for and against any and all claims demands actions litigation and proceedings of any kind arising out of or related to such ultra hazardous use or sale
11. ctions below 1 Power down the system 2 Remove the power cord from the wall socket 3 Use industry standard anti static equipment such as gloves or wrist strap and follow the instructions listed on Page 3 1 to avoid damage caused by ESD 4 Familiarize yourself with the server motherboard and or chassis documenta tion 5 Confirm that your operating system includes the latest updates and hotfixes 3 2 Chapter 3 Installation 3 4 Installing the Add on Card Follow the steps below to install the add on card into your system 1 3 5 Remove the server cover and if necessary set aside any screws for later use Remove the add on card slot cover If the case requires a screw place the screw aside for later use Position the add on card in the slot directly over the connector and gently push down on both sides of the card until it slides into the PCI connector Secure the add on card to the chassis If required use the screw that you previously removed Attach any necessary external cables to the add on card Replace the chassis cover Plug the power cord into the wall socket and power up the system Installing the Windows Operating System Follow the steps below to install the drivers needed for your Windows OS support The controller comes with a driver on the CD ROM CDR NIC 1 Run the CDR NIC If you do not have a product CD ROM download drivers from the Supermicro Support Website and th
12. en transfer them to your system When the SUPERMICRO window appears click on the computer icon next to the product model S SUPERMICRO Add On Card Drivers and Tools WinXP xi 10GbE Adapters 2 UTG32 STG 42 STGNH2S STG 2T SUPERMICR 1B Adapters i a UINF m2 UIBQ m1 UIBQ m2 R1UIG BQ CIBF m1 UIBF mt IB drivers Drivers amp Tools ES 1 IB Adapters LTL UINFm2 UIBQ m1 UIBO m2 RIUG BQ CIBF m1 UIBF m1 10GbE drivere D GbE Adapters q gt B UG 4 PG i2 SG i2 SG i4 CG 2 SGP i4 CGP 2 ng TS Red Pott Desain and Manual Ss 40 SUPERMICRO Computer Inc For mote information please visit SUPERMICRO s web site Note If the FOUND NEW HARDWARE WIZARD screen displays on your system click CANCEL 3 3 SUPERO AOC CIBF m1 Add on Card User s Guide 3 Click on INSTALL DRIVERS AND SOFTWARE 4 Follow the prompts to complete the installation 3 6 Installing Drivers and Firmware Use the procedures below to install both drivers and firmware for the AOC CIBF m1 add on card for both Linux and Windows Linux Drivers Use the following procedures for installing AOC CIBF m1 drivers for the Linux operating system Installing InfiniBand Drivers for the Linux Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox gt InfiniBand gt Linux 2 Choose the nfiniBand Linux driver package file 3 Install the driver by entering the
13. following commands tar xzvf OFED lt ver gt tgz cd OFED ver install pl This installs the Linux drivers to your system Installing 10G Drivers for the Linux Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox gt ConnectX EN gt Linux 2 Choose the 10G Linux driver package file 3 4 Chapter 3 Installation 3 Install the driver by entering the following commands tar xzvf mlnx en ver tgz cd mlinx en ver install sh This installs the 10G drivers to your system Windows Drivers Use the following procedures for installing AOC CIBF m1 add on card drivers for the Windows operating system Installing InfiniBand Drivers for the Windows Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox gt InfiniBand gt Windows 2 Choose the desired InfiniBand Windows driver package file 3 Double click to run and install the driver package file Installing 10G Drivers for the Windows Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox gt ConnectX EN gt Windows 2 Choose the desired 10G Windows driver package file 3 Double click to run and install the driver package file 3 5 SUPERO AOC CIBF m1 Add on Card User s Guide Firmware Update Procedures Use the following procedures for updating the AOC CIBF m1 add on card firmware for both
14. grounded metal object before removing the add on card from the antistatic bag Handle the add on card by its edges only do not touch its components or peripheral chips Put the add on card back into the antistatic bags when not in use For grounding purposes make sure that your system chassis provides excellent conductivity between the power supply the case the mounting fasteners and the add on card Unpacking The add on card is shipped in antistatic packaging to avoid static damage When unpacking your component or your system make sure that the person handling it is static protected Warning To avoid damaging your components and to ensure proper installation be sure to always connect the power cord last and always remove it before adding removing or changing any hardware components 3 1 SUPERO AOC CIBF m1 Add on Card User s Guide 3 2 Add On Card Brackets The add on card ships with a PCB bracket that is pre installed at the factory For 1U systems an I O bracket is also pre installed on the card For 2U systems the NO bracket is bundled with the card and must be installed by the customer See the drawings below for bracket details BKT 0051L Factory installed MicroLP PCB bracket For 2U System For 1U System BKT 0053L Customer installed BKT 0052L Factory I O bracket for 2U system installed I O bracket 3 3 Before Installation To install the add on card properly be sure to follow the instru
15. ious models or provides information for correct system setup Naming Convention Character Representation Options 1st Product Family AOC Add On Card Form Factor U UIO S Standard P Proprietary C MicroLP Product Type Speed G GbE 1Gb s TG 10GbE 10Gb s IBF IB FDR 56Gb s IBQ IB QDR 40Gb s INF InfiniBand DDR 20Gb s Chipset Model Optional N Niantec 82599ES P Powerville i350 Chipset Manufacturer i Intel m Mellanox Number of Ports 1 1 port 2 2 ports 4 4 ports Connector Type Optional S SFP T 10GBase T SMC Networking Add on Cards Form Factor Interface Controller Connection Dimension without Brackets Hx L SG i2 Standard LP PCI E x4 Intel 82575EB 2 RJ45 1Gb port 5 2 13 2cm x 2 5 6 4cm SG i4 Standard LP PCI E x8 Intel 82576EB 4 RJ45 1Gb port 5 8 14 7cm x 2 5 6 4cm SGP i4 Standard LP PCI E x4 Intel i350 4 RJ45 1Gb port 3 9 9 9cm x 2 5 6 4cm STG i2T Standard LP PCI E x8 Intel X540 2 RJ45 10Gb port 5 4 13 7cm x 2 5 6 4cm STGN i2S Standard LP PCI E x8 Intel 82599ES 2 SFP 10Gb port 5 4 13 7cm x 2 5 6 4cm STG i2 Standard LP PCI E x8 Intel 82598EB 2 CX4 10Gb port 5 6 14 1cm x 2 5 6 4cm PG i2 Proprietary LP PCI Ex4 Intel 82576EB 2 RJ45 1Gb port 3 8 9 6cm x 2 5 6 4cm UG i4 UIO FH PCI E x8 Intel 82571EB 4 RJ45 1Gb port 6 6 16 7cm x 3 9 9 8cm UTG i2 UIO FH PCI E x8 Intel 82
16. talling Drivers ard Firmware crecen rre 3 4 MID DIVET re ETT 3 4 Windows Drivers 9 5 Firmware Update Procedures ls x as ksayasd nee salada diay gesinne 3 6 vi Chapter 1 Overview Chapter 1 Overview 1 1 Overview Congratulations on purchasing your add on card from an acknowledged leader in the industry Supermicro products are designed with the utmost attention to detail to provide you with the highest standards in quality and performance For product support and updates please refer to our website at http www supermicro com products nfo networking cfm adapter 1 2 Key Features The key features of this add on card include the following Single QSFP Quad Small Form Factor Pluggable Connector MicroLP Form Factor PCI Express 3 0 up to 8GT s Virtual Protocol Interconnect VPI Up to 56Gbps InfiniBand or 40Gbps Ethernet CPU offload of transport operations Application offload GPU communication acceleration End to end QoS and congestion control e Hardware based l O virtualization Ethernet encapsulation EoIB RoHS compliant 6 6 1 3 Specifications General Mellanox ConnectX 3 FDR controller Compact size microLP form factor Single QSFP port and dual USB 2 0 ports PCI E 3 0 x8 8GT s interface 1 1 SUPERO AOC CIBF m1 Add on Card User s Guide Connectivity Interoperable with InfiniBand or 10 40GbE switches Passive copper cable with ESD protection
17. the Linux and Windows operating systems Updating Firmware for the Linux Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox Firmware Tool Linux 2 Choose the MFT Linux package file and untar the package file 3 Install the package by entering the following command cd mft ver install sh 4 Install the firmware by the following commands mst start mst status write down the pci cr0 device name flint d pci cr0 device name i firmware file name bin b Updating Firmware for the Windows Operating System 1 From the CDR NIC LAN driver CD or FTP site go to the following directory Mellanox Firmware Tool Windows 2 Choose the desired MFT Windows package file 3 Double click to run and install the MFT package file 4 Open the DOS command prompt and locate the MFT folder 5 Install the firmware by entering the following commands mst start mst status write down the pci cr0 device name flint d pci cr0 device name i firmware file name bin b 3 6 Disclaimer Continued The products sold by Supermicro are not intended for and will not be used in life support systems medical equipment nuclear facilities or systems aircraft aircraft devices aircraft emergency communication devices or other critical systems whose failure to perform be reasonably expected to result in significant injury or loss of life or catastrophic property damage A
Download Pdf Manuals
Related Search