User Manual - Advanced HPC
Contents
1. Valid values http host port debug_level Verbosity level of fca manager for integer between 0 7 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 logfile name can contain printf like tokens which are substituted during log file creation e hostname where FCA Manager is running e D current date in format DDMMYYYY e current thread ID Parameter Description Values debugging The default is 3 The debug levels are e 0 fatal e error e 2 warn e 3 info e 4 debug e 5 7 detailed debug info log file FCA Manager log filename The string representing log file name The default is fmm_ H_ D log log_file_max_size This is a critical size in MB of the log_ file to which the file will be rolled If set to zero rolling is disabled and file size is unlimited any positive valid number The default is 10 log_file_max_backu p_files Denotes the number of backup files to be created Effective only when log_file_max_size parameter has value greater than zero integer The default is 20 enable stdout Determines whether the log should also be written to the standard output Valid values e yor l enable e nor 0 disable character The default is enable The following parameters may be changed in the INI file under section ib on the matching device If2. entes 21 6 1 Specifying FCA Parameters as mpirun Command Line 21 6 2 Specifying FCA Parameters in the INI File 21 Configuring Rules for 24 7 1 Enabling Dynamic Rules Mechanism 24 7 2 estne nn 24 8 OpenMPI MCA Parameters to Control FCA Offload eese 26 3 Mellanox Technologies Confidential Version 2 5 Contents List of Figures Figure d FGA Architecture si o et dat itty Abad He t d ti ed ts 9 Figure 2 FCA Components tap edere Pe E Cadre Dos GE RUD ee 10 o Mellanox Technologies Confidential Contents Version 2 5 List of Tables Table 1 FCA Related Packages Table 2 System 14 Table 3 Paths for FCA Manager INI File E atra et ee ue 18 Table 4 FCA Manager INI File 18 Table 5 FCA Parameters in Open MPI at Run 21 N 5 Me
3. in the fabric 1 5 FCA Software Components The FCA related software components are listed in the following table Table 1 FCA Related Packages Package Description FCA Manager The FCA Manager is server software that responds to requests from the MPI application to set up new communicators FCA MPI The FCA MPI Runtime library is a user level shared library which is integrated Runtime with specific MPI distributions IBM PE OpenMPI Platforms MPI IntelMPI Libraries MVAPICH2 that is responsible for offloading MPI collective operations into Fabric 11 Mellanox Technologies Confidential Version 2 5 Introduction to Mellanox Fabric Collective Accelerator 1 6 FCA Installation Package Content The FCA installation package includes the following items FCA Mellanox Fabric Collector Accelerator Installation files fca lt version gt x86_64 lt OS gt rpm fca lt version gt x86_64 lt OS gt tar gz where lt version gt is the version of this release and lt OS gt is one of the supported Linux distributions listed in Prerequisites on page 14 Mellanox Fabric Collective Accelerator FCA Software End User License Agreement FCA Manager software FCA MPI runtime libraries Mellanox Fabric Collective Accelerator FCA User Manual Mellanox Fabric Collective Accelerator FCA Release Notes Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Ma
4. not set or zero the first active port is used dev_name If set the specified IB device will be string representing active IB used for communication device name The name as appears in The default is set sys class infiniband directory If not set the first device with ACTIVE port is used port_num If set the selected port number is used positive integer The default is unset use auto discover service_level Quality of Service QoS is offered in IB as a means to offer some guarantees minimum requirements for certain applications on the fabric SL2VL mapping should be configured integer The default is 0 19 Mellanox Technologies Confidential Version 2 5 Configuring Parameter Description Values in OpenSM Valid values 0 15 Note that OpenSM works by default with QoS values of 0 7 20 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 6 FCA MPI Runtime Library Configuration Parameters 6 1 Specifying FCA Parameters as mpirun Command Line Arguments The FCA runtime library is used by MPI to offload collective operations into IB switches You can supply configuration parameters to the FCA runtime library The configuration parameters may be passed to the FCA library by either loading the parameters from a configuration INI file or entering the parameters in a command line to the MPI job or setti
5. CA support Download OpenMPI 1 6 x from the OpenMPI site Enter the following commands mkdir p S HOME openmpi 5 cd SHOME openmpi wget http www open mpi org software ompi v1 6 x downloads openmpi 1 6 3 tar gz S Bee Gero 1 tar Cw cd openmpi 1 6 3 4 2 Verifying the FCA Installation gt To verify that OpenMPI is working with the FCA installation e Enter the following command SMPI HOME bin ompi info param coll fca grep fca enable The list of FCA parameters should be displayed as a command output 4 3 Running MPI Jobs with FCA Make sure that the FCA manager tarball is unpacked and available from all cluster nodes Its opened location is referenced below as 1 Use the following script examples with the information provided on how to run MPI jobs with FCA for different MPI vendors For OpenMPI MPI HOME scripts run ompi fca sh For Platforms MPI HOME scripts run pmpi fca sh For Intel MPI SFCA HOME scripts run impi fca sh For MVAPICH2 FCA HOME scripts run mvapich2 fca sh 2 Check the FCA HOME etc fca mpi spec ini file for various FCA tuning options N 17 Mellanox Technologies Confidential Version 2 5 Configuring 5 1 Configuring FCA FCA Manager Configuration Parameters The fca manager spec ini file is a configuration file containing FCA related parameters which you can change or overwrite usin
6. Mellanox TECHNOLOGIES Fabric Collective Accelerator FCA User Manual Version 2 5 Last Modified on January 17 2012 www mellanox com Mellanox Technologies Confidential Version 2 5 Contents NOTE THIS HARDWARE SOFTWARE OR TEST SUITE PRODUCT PRODUCT S AND ITS RELATED DOCUMENTATION ARE PROVIDED BY MELLANOX TECHNOLOGIES AS IS WITH ALL FAULTS OF ANY KIND AND SOLELY FOR THE PURPOSE OF AIDING THE CUSTOMER IN TESTING APPLICATIONS THAT USE THE PRODUCTS IN DESIGNATED SOLUTIONS THE CUSTOMER S MANUFACTURING TEST ENVIRONMENT HAS NOT MET THE STANDARDS SET BY MELLANOX TECHNOLOGIES TO FULLY QUALIFY THE PRODUCT S AND OR THE SYSTEM USING IT THEREFORE MELLANOX TECHNOLOGIES CANNOT AND DOES NOT GUARANTEE OR WARRANT THAT THE PRODUCTS WILL OPERATE WITH THE HIGHEST QUALITY ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT ARE DISCLAIMED IN NO EVENT SHALL MELLANOX BE LIABLE TO CUSTOMER OR ANY THIRD PARTIES FOR ANY DIRECT INDIRECT SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES OF ANY KIND INCLUDING BUT NOT LIMITED TO PAYMENT FOR PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY FROM THE USE OF THE PRODUCT S AND RELATED D
7. OCUMENTATION EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE Mellannx Mellanox Technologies Inc Mellanox Technologies Ltd 350 Oakmead Parkway Suite 100 Beit Mellanox Sunnyvale CA 94085 PO Box 586 Yokneam 20692 U S A Israel www mellanox com www mellanox com Tel 408 970 3400 Tel 972 0 4 909 7200 972 0 74 723 7200 Fax 408 970 3403 Fax 972 0 4 959 3245 Copyright 2012 Mellanox Technologies Inc All Rights Reserved Mellanox Technologies rights reserved Mellanox Mellanox logo BridgeX ConnectX CORE Direct InfiniBridge InfiniHost InfiniScale PhyX SwitchX Virtual Protocol Interconnect and Voltaire are registered trademarks of Mellanox Technologies Ltd Connect IB FabricIT MLNX os MetroX Unbreakable Link UFM and Unified Fabric Manager are trademarks of Mellanox Technologies Ltd All other trademarks are property of their respective owners CERAM Mellanox Technologies Confidential Contents Version 2 5 Contents nba 6 m ERR 7 1 Introduction to Mellanox Fabric Collective Accelerator eere 9 XM S EA Ii en 9 1 2 Supported nennen sinn sn nennen ns 10 1 9 Topologies ite tne iet Pe eb epe tds 11 1 4 Planning the Server Configuratio
8. OFED v1 5 3 3 1 0 1 Enter the following commands mkdir p usr local mellanox cd usr local mellanox ipee 4 2 Run the following post install scrip on all hosts cd fca 2 5 xxxx x86 64 scripts udev update sh 3 Set the environment variable pointing to the extracted location of FCA in the user login profile export FCA MGR HOME usr local mellanox fca 2 5 xxxx x86 64 15 Mellanox Technologies Confidential Version 2 5 Installing 3 2 3 Starting the FCA Manager gt To start the FCA Manager e Enter the following command SFCA MGR HOME scripts fca managerd start e For RPM setup only enter etc init d fca managerd start e Toconfigure FCA manager to start automatically after boot run etc init d fca manager install service Mou Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 4 Installing FCA MPI Support Libraries You can install the FCA MPI support libraries from either an RPM or from Tarball on all cluster nodes or to the shared NFS location using Tarball For further information see Installing the FCA Manager from RPM on page 15 and Installing the FCA Manager from Tarball on page 15 4 1 Building OpenMPI 1 6 x with FCA Support NOTE If you use OpenMPI 1 6 x or later no patch is required OpenMPI 1 6 x supports FCA natively gt To build OpenMPI 1 6 x with F
9. brics Enterprise Distribution for Linux MLNX_OFED 3 Mellanox ConnectX 2 or ConnectX 3 HCA with firmware version 2 9 1000 or later To download the latest ConnectX HCA firmware version go to Firmware Downloads The minimum system requirements for installing and running FCA are listed in the following table E NOTE Mellanox OFED 1 5 3 3 1 0 includes FCA 2 2 and OpenMPI which is compiled with FCA v2 2 Both packages should be removed prior to installing FCA v2 5 To remove them run f rpm e fca rpm e openmpi Table 2 System Hequirements Item Requirement FCA 2 5 Supported switches Mellanox IB QDR FDR switches Linux distributions OS RHEL 6 2 Supported HCAs Mellanox ConnectX 2 HCA with firmware version 2 9 1000 or later Mellanox ConnectX 3 HCA with firmware version 2 10 0000 or later Open Message Passing Open MPI 1 6 3 or later Interface MPI Project Open Fabrics Enterprise 1 5 3 3 1 0 or later Distribution Root permission The installer should have root permissions for post installation tasks InfiniBand Subnet All InfiniBand Subnet Management based software is supported in Management FCA version 2 5 14 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 3 2 Installing the FCA Manager on a Dedicated Node FCA Manager must be installed on a dedicated machine which is not a
10. bytes e communicator size in ranks e offloading method CD CoreDirect UD MPI native e Operation for Reduce AllReduce e Data type for Reduce AllReduce Enabling Dynamic Rules Mechanism gt To enable dynamic rules mechanism e Enter the following command in fca_mpi_ spec ini file section called rules enable 0 1 Example enable 1 Configuring a Specific Rule User defined offloading rules are added and enumerated in fca_mpi_spec ini file Every user defined rule is represented by a new INI file section named in the following format rule coll name gt lt SN gt coll name can be one of the following values reduce allreduce beast barrier allgather allgatherv e SNisarule serial number for given coll name The default value for min max params is 1 no limit Valid offload types are e ud use FCA in UD mode e cd default use FCA in COREDirect mode e none do not use FCA 24 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Rules are applied by the first match If none of the rules match the default is to use FCA with COREDirect mode The following is a list of valid rules parameters for FCA Version 2 5 Parameter Description Default msg_size_min lt int gt Minimum message size No limit msg size max int Maximum message size No limit comm size min int Min
11. g the command line during runtime The FCA Manager process reads its configuration on startup from the FCA MGR HOME etc fca manager spec ini file Depending on the method used to install FCA the Mellanox provided parameter file fca manager spec ini will be located in the path described in the following table Table 3 Paths for FCA Manager INI File Installation Method Path to fca manager spec ini From RPM opt mellanox fca etc From Tarball FCA_HOME etc The FCA Manager configuration file is in INI format and contains two sections fmm and ib To set parameter values in the fca_manager_spec ini file edit the file as necessary using the following format variable value Example fmm debucmlc 5 log file leg Table 4 FCA Manager INI File Parameters Parameter Description Values The following parameters may be changed in the INI file under the fmm section osm_type Select subnet manager service opensmlufmlautodetect provider The default is autodetect Valid values ufm Use Mellanox s e opensm Use OpenSM library e autodetect Detect automatically from fabric When embedded OpenSM is used in the switch the FCA module should be disabled in that specific switch ufm_url URL of OpenSM service The default string port is 8081 You can replace The default is localhost with the IP or hostname of the machine on which OpenSM runs
12. i stats file name File name in which to keep collected statistics collect stats must be enabled for this parameter to take effect string The default is fca stats xml The following parameters may be changed in the INI file under section ib fca ib dev name If set the specified IB device will be used for communication The name as appears in sys class infiniband directory If not set the first device with an ACTIVE port is used string representing active IB device name default Leave empty or commented then auto discovery will be used 22 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 INI File Parameter Description Values fca_ib_port_num If set the selected port number is used on positive integer the matching device The default is unset If not set or zero the first active port is use auto discover used Example opt openmpi 1 6 3 bin mpirun np 32 machinefile hostfile mca btl sm self openib x fca mpi debug level 4 mpi hello world 23 Mellanox Technologies Confidential Version 2 5 Configuring Rules for Offloading 7 1 7 2 Configuring Rules for Offloading The FCA system is provided with user defined rules to select the most suitable offloading method for MPI communication The used defined rules consider the following MPI Communicator parameters e message size range in
13. imum communicator size No limit comm size int Maximum communicator size No limit offload type lt string gt FCA offload type cd COREDirect mode data type string Data type given as a parameter Applicable for reduce allreduce reduce op string Reduce operation type requested Applicable for reduce allreduce Examples of reduce rules rules enable rule reduce 1 msg size min msg size max Size mum Comm Size mss offload type data_type reduce op rule reduce 2 msg size min meg SILAS comm size min Size offload type 256 1024 30 39 ud MPI CHAR MPI LXOR 2024 10 none N 25 Mellanox Technologies Confidential Version 2 5 OpenMPI MCA Parameters to Control FCA Offload 8 OpenMPI MCA Parameters to Control FCA Offload The complete list of OpenMPI FCA related parameters can be extracted using the ompi info command gt To extract the complete list of OpenMPI FCA related parameters SMPI HOME bin ompi info param coll fca gt To provide MCA parameters to the OpenMPI mpirun command use the following format SMPI HOME bin mpirun mca param value Example RE The following is a list of MCA parameters for FCA Parameter Description Default coll_fca_priority lt int gt P
14. isolated from the rest of the traffic in the fabric using a private virtual network VLane eliminating contention with other types of traffic The following diagram summarizes the FCA architecture Figure 1 FCA Architecture Inter core communication optimized Use of IB multicast for result Collective tree amp Rank placement optimized to the topology Mellanox Technologies Confidential Version 2 5 Introduction to Mellanox Fabric Collective Accelerator The following diagram shows the FCA components and the role that each plays in the acceleration process Figure 2 FCA Components Job UFM GD 4700 4200 Scheduler FCA Manager Mellanox UFM FCA Manager Orchestrating fabric wide collectives GD 4036 4036E CPUs a a H H A A A A A offload collective nt EJ EJ EJ Ed eS Ed EE E E computations Intra node EJ ES Ed Ed E E d collective EI EI EI computation Az ew ew CU ue uei Compute nodes 1 2 Supported MPI Collectives FCA addresses a wide range of applications with out of the box integration with leading MPI implementations such as Platform MPI and Open MPI and requires no changes to the application The following MPI collectives are currently supported by FCA and accelerated e MPI Reduce e MPI Allreduce e MPI Barrier e MPI Bcast e MPI_AllGather e MPI_AllGatherv FCA supports an unlimited message
15. lerator FCA User Manual is the MPI implementer and the network administrator responsible for managing FCA on Mellanox InfiniBand switches It is assumed that the administrator is familiar with advanced concepts in network management Related Documentation The following document is part of the library for network administrators and installers supporting the Mellanox FCA Document Name Part Number Mellanox Fabric Collective Accelerator Release Notes DOC 00984 Typographical Conventions Before you start using this guide it is important to understand the terms and typographical conventions used in the documentation The following kinds of formatting in the text identify special information Formatting convention Type of Information Special Bold Items you must select such as menu options command buttons or items in a list Emphasis Use to emphasize the importance of a point or for variable expressions such as parameters CAPITALS Names of keys on the keyboard for example SHIFT CTRL or ALT KEY KEY Key combinations for which the user must press and hold down one key and then press another for example CTRL P or ALT F4 7 Mellanox Technologies Confidential Version 2 5 Preface Document Conventions gt NOTE Identifies important information that contains helpful suggestions CAUTION Alerts you to risk of personal injury system damage or loss of da
16. llanox Technologies Confidential Version 2 5 Revision History Revision History Version 2 5 Dec 2012 e Removed section Upgrading from FCA 2 0 or Later e Updated the following sections FCA Installation Package Content on page 12 Downloading the FCA Software on page 13 Prerequisites on page 14 Building OpenMPI 1 6 x with FCA Support on page 17 Installing the FCA Manager on a Dedicated Node on page 15 Installing the FCA Manager from RPM on page 15 Version 2 2 May 2012 e Removed section Activating the Software License e Updated the following sections Prerequisites on page 14 Installing the FCA Manager on a Dedicated Node on page 15 Configuring a Specific Rule on page 24 Upgrading from FCA 2 0 or Later Starting the FCA Manager on page 16 Version 2 1 1 December 2011 e Updated the following sections to reflect offloading collective operations onto HCA Overview text and graphics Supported Topologies FCA Installation Package Content e Updated Prerequisites and Installation sections for 2 1 1 e Removed section on configuring Grid Director switches to enable FCA e Added note for OpenMPI 1 5 x in section Building OpenMPI 1 4 x with FCA Support P AM Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 Preface Audience The intended audience for the Mellanox Fabric Collective Acce
17. n enne nnns 11 15 FCA Software Components sess enne nennt nnne nnns 11 1 6 FCA Installation Package 12 2 installation and Initial 13 2 1 Overview of Installation and Initial Configuration sese 13 2 1 4 Downloading the FCA 13 Bis we 14 Prerequisites xeu etate amete 14 3 2 Installing the FCA Manager on a Dedicated 15 3 2 1 Installing the FCA Manager from RPM sse nene 15 3 2 2 Installing the FCA Manager from Tarball seen 15 3 2 9 Starting the FCA Manager onte rc ea e DRE Hte be ce PE ek need 16 4 Installing FCA MPI Support Libraries 17 4 1 Building OpenMPI 1 6 x with FCA 17 4 2 Verifying the FCA Installation sess enne 17 4 3 Running MPI Jobs 17 5 Configuring ata ede es secede To cde 18 5 1 FCA Manager Configuration Parameters sess 18 6 FCA MPI Runtime Library Configuration Parameters
18. ng them from the shell environment e The default configuration file for the FCA MPI runtime parameter is located at HOME etc fca_mpi spec ini e The FCA parameters can be entered as command line parameters as part of the mpirun command To pass FCA parameter from shell environment e Enter the following command export fca ini section name ini section param name value Example export fea m MEO lewel 5 or provide the parameter to OpenMPI as command line argument MOLI x rea msi Ceoug rever Ec moirun pasamei e oso 6 2 Specifying FCA Parameters in the INI File At runtime use mpirun s command switch to overwrite FCA parameters set in the fca mpi spec ini file Set the FCA parameters in an MPI job at runtime with the following syntax SMPI HOME bin mpirun np 32 machinefile hostfile mea coll oma fca library path HOME lib libfca so mca btl sm self openib lt param gt value other mpi options mpi hello world where e MPI HOME represents the path to the MPI installation directory e FCA_HOME represents the path to the FCA software directory NOTE The x command switch is used as follows x lt param gt lt value gt Example x fca mpi collect stats y Table 5 FCA Parameters in Open MPI at Run Time INI File Parameter Description Values The following parameters may be changed in the INI file under section mpi N 21 Mellanox Technol
19. nual Version 2 5 2 Installation and Initial Configuration 2 1 Overview of Installation and Initial Configuration FCA software includes the FCA Manager and the FCA MPI runtime support libraries FCA Manager software should be installed on a central management node For optimal performance and to minimize interference with other applications it is recommended to use a dedicated server for the FCA Manager installation The following sections provide step by step instructions for installing the FCA Server software and installing the FCA Agent 2 1 1 Downloading the FCA Software NOTE Mellanox OFED 1 5 3 3 1 0 includes FCA 2 5 which is installed under opt mellanox fca If you have installed OFED 1 5 3 310 you do not need to download and install FCA This software download process applies to software updates as well as for first time installation gt To download the FCA software Go to the Mellanox website 1 2 Click the Downloads tab and select the relevant version of the software to download 3 Save the file on your local drive A Click Close N 13 Mellanox Technologies Confidential Version 2 5 Installing 3 Installing FCA 3 1 Prerequisites Before you begin be certain that 1 InfiniBand Subnet Management is installed and running on a dedicated node in the fabric 2 Mellanox OFED 1 5 3 3 1 0 or later is installed To download the latest MLNX_OFED version go to Mellanox OpenFa
20. ogies Confidential Version 2 5 FCA MPI Runtime Library Configuration Parameters INI File Parameter Description Values fca_mpi_debug_level Verbosity level for MPI FCA debugging The debug levels are e 0 fatal e error e 2 warn e 3 info e 4 debug e 5 7 detailed debug info integer between 0 7 The default is 2 fca_mpi_log_file FCA log filename The logfile name can contain printf like tokens which are substituted during log file creation e hostname of the process e u current time in ms e current thread ID e s time in sec e Cbt time in ticks String representing log file name or empty for none The default is none fca mpi enable stdout Determines whether the FCA log should also be written to the standard output Valid values lt gt character The default is y fca_mpi_fp_sum_fixedpoi nt Use fixed point math when performing floating point summation to keep a consistent result regardless of the order of operations Valid values lt gt character The default is n fca_mpi_collect_stat Collect MPI application performance statistics Valid values lt gt character The default is n fca_mpi_stats_max_ops Max number of different MPI collective operations for which to collect statistics This option is effective when collect stat y positive integer The default is 1000 fca mp
21. part of the cluster nodes with only a single instance of the FCA Manager in operation per fabric NOTE We recommend that the FCA Manager be installed on the same node as OpenSM There are two options for installing the FCA Manager when using MLNX OFED v1 5 3 3 1 0 e From an RPM on page 15 Select this option if you want to install the FCA Manager on the machine s local disk and let the RPM package handle all post install tasks e From Tarball on page 15 Select this option if you wish to install FCA Manager in any location user s home directory NFS shared folder etc There are number of post install tasks that need to be applied as root on every cluster node after you install FCA from a tarball Select one of the installation options according to your site s installation policy 3 2 1 Installing the FCA Manager from RPM To install the FCA Manager on all cluster nodes from an RPM as root when using MLNX OFED v1 5 3 3 1 0 Enter the following command rpm e fca rpm e openmpi rpm ihv fca 2 5 x86 64 rpm 2 Set the environment variable pointing to the installed location of FCA in the user login profile export FCA MGR HOME opt mellanox fca 3 Optional Configure the FCA Manager to start automatically after boot etc init d fca managerd install service 3 2 2 Installing the FCA Manager from Tarball To install the FCA Manager from Tarball in the shared NFS location when using MLNX
22. riority of the fca coll component 80 coll fca verbose int Verbose level of the fca coll 0 component coll enable lt 011 gt Enable Disable Fabric Collective 1 Accelerator coll fca spec file lt string gt Path to the FCA configuration file fca_mpi_spec ini _mpi_spec ini coll_fca_library_path lt string gt Path to FCA runtime library FCA HOMH lib libf ca so coll_fca_np lt int gt Minimal allowed job s NP to activate 64 FCA coll fca enable barrier lt 011 gt Enable Disable FCA Barrier support 1 coll fca enable bcast lt 011 gt Enable Disable FCA Bcast support 1 coll fca enable reduce lt 011 gt Enable Disable FCA Reduce support 1 coll fca enable allreduce Enable Disable FCA Allreduce 1 0l support coll fca enable allgather lt 011 gt Enable Disable FCA Allgather 1 support coll fca enable allgatherv Enable Disable FCA Allgatherv 1 0l support 26 Mellanox Technologies Confidential
23. size and advanced optimizations for Torus topologies It can work with any InfiniBand Subnet Management based software OpenSM Embedded SM Host SM FCA supports the following data types for Reduce and Allreduce operations e data types for C language bindings except MPI_LLONG_DOUBLE e All data types for C reduction functions C reduction types e The following data types for FORTRAN language bindings MPI INTEGER MPI INTEGER2 Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 MPI INTEGERA MPI MPI REAL MPI REALA MPI_REAL8 FCA does not support data types for Fortran reduction functions Fortran reduction types 1 3 Supported Topologies e FCA supports almost all fabric topologies Fat Tree HyperScale Torus e FCA requires a Mellanox based Infiniband network 1 4 Planning the Server Configuration Following are points to consider when planning on which server to install the FCA Manager e The FCA Manager should be installed on a different server than the one where MPI jobs will run e If you do not have two servers running UFM for redundancy you should install FCA Manager on the UFM server e If you do have two servers running UFM for redundancy it is best to install FCA Manager on a non UFM server and provide it with the virtual address of the UFM machine e Only a single instance of FCA Manager should be running
24. ta WARNING Warns you that failure to take or avoid a specific action might result in 4 personal injury or a malfunction of the hardware or software Be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents before you work on any equipment ei looo y Mellanox Technologies Confidential Fabric Collective Accelerator FCA User Manual Version 2 5 1 1 Introduction to Mellanox Fabric Collective Accelerator Overview The Mellanox Fabric Collective Accelerator FCA is a unique solution for offloading collective operations from the Message Passing Interface MPI process to the server CPUs AS a system wide solution FCA does not require any additional hardware The FCA manager creates a topology based collective tree and orchestrates an efficient collective operation using the CPUs in the servers that are part of the collective operation FCA accelerates MPI collective operation performance by up to 100 times providing a reduction in the overall job runtime Implementation is simple and transparent during the job runtime FCA is built on the following main principles e Topology aware Orchestration The MPI collective logical tree is matched to the physical topology The collective logical tree is constructed to assure Maximum utilization of fast inter core communication Distribution of the results e Communication Isolation Collective communications are
Download Pdf Manuals
Related Search