Home

Scali System Guide

Contents

1. LO links L1 links Figure 3 11 Example of 2x4 two dimensional system cabling in a 1x8 cabinet Scali System Guide Version 3 0 3 4 Dolphin SCI interconnect installation 29 Chapter 3 Hardware installation 3 4 3 4 Three dimensional SCI torus cabling With the 3D three dimensional SCI torus topology cabling becomes more of a challenge The 3D topology is harder to merge with the natural row column way of arranging node hardware More effort is needed for optimal node placement and it can be hard to avoid long cables Cabling of 3D systems follows the guidelines set up for rings and 2D torus topologies with one addition e Connect the third dimension on SCI link 2 L2 In the example below we will explain how to connect a small 18 node 3x3x2 3D torus The structure of the torus is shown in Figure 3 12 for readability the SCI network is drawn without rings and node names are only represented by the numeric part only m 223 ee 233 213 a 123 133 i 212 112 211 LA 122 132 LO 221 231 L2 AS 111 121 131 Figure 3 12 A 3x3x2 3D SCI
2. OK ae Testing MPI performance __ OK Testing MPICH communication OK Testing MPICH performance OK ze Testing OpenPBS OK Gas Install complete In the Epilogue section some information will be printed before the installation is complete and the installation program will exit Epilogue e oes Scali Software Getting Started Guide Most of the Scali software is accessible through the graphical desktop ScaDesktop To start the desktop issue this command on the command line opt scali bin scadesktop Please note that the desktop will be different for root than for ordinary users Please also note that the desktop is not available for frontend only installations For help on specific issues regarding the installation and operation of the Scali software please refer to the Scali System Guide document version Id GettingStarted v 1 3 2001 11 29 13 47 55 ae Exp Clean up and exit O EE EE Removing recorded information 58 Scali System Guide Version 3 0 4 3 Scali software installation 4 3 3 SSP Uninstall If you wish to remove the SSP at a later stage you can run the uninstall program directly on the frontend as root opt scali sbin SSPuninstall You also have the p
3. Do you want to read the HW installation guide before continuing O Read it now n Scali System Guide Version 3 0 45 Chapter 4 Software installation 46 Read OS documentation 2 ES Do you want to read the OS installation guide before continuing Q Read it now n Specifying node names ___ 5 5 2 Please specify nodenames for your system You may use expression with Brace Expansion see bash man pages Include the frontend if it is supposed to be used as a processing node Press lt return gt on an empty line after adding your node s Q Please enter node name s 1 scalil 1 2 1 2 Q Please enter node name s 5 The following nodes have been defined scalil 11 scalil 12 scalil 21 scalil 22 Q Do you accept the above configuration yl y Specifying repository 9 DEFAULT Preparing repository om Creating non existant path opt scali repository Linux2 i86pc Copying packages to repository ScaSH Removing old versions and copying files to repository ScaPkg Removing old versions and copying files to repository ScaDiag Removing old versions and copying files to repository ScaLM Removing old versions and copying files to repository Preparing ScaSH config um ae Installing installation tools Sec ScaSH Preparing package confi
4. Scali System Guide Version 3 0 10 4 MPI problems 10 4 MPI problems Common problems encountered running MPI programs with Scali s MPI implementation ScaMPI is handled in chapter 4 4 When things don t work troubleshooting in the Scali Library User s Guide accompanying the product You shuld also take a look at the on line ScaMP FAQ in the support section at Scali s web site http www scali com support 10 5 SCI Interconnect problems TheLEDs on the SCl card can give hints about the condition of the SCI interconnect refer to Table 10 2 This table is only valid for D33x cards Colour Behaviour Interpretation Yellow Steady Power on D33x card but no cable installed correctly remote com puter not powered or erroneous cable Green Steady Cable installed OK Yellow Short period Self test after power on after installation of cable or after soft ware reset Green Blinking Cable installed and OK and SCI traffic on B Link Table 10 2 LED behaviour on D33x cards Scali uses its own highly optimized driver ScaSCI when utilising the ultra high speed SCI interconnect in a Scali system The LEDs on the SCI card will not be steady green until this software is loaded and all cables are inserted correctly 10 5 1 SCI error messages The SCI status error messages reported by opt scali bin mpimon are explained by the program opt scali bin sciemsg Taking as input a SCI status error message numbe
5. done in different stages configuration package installation and test One may restart theinstallation at any of these stages without the need to run the previous stage again 4 3 2 1 Introduction Theintroductory part of theinstallation mainly deals with giving the user information about practical issues for the installation The license terms for the different software modules have to be accepted for the installation to continue The following example was run after the initial installation was aborted Questions about using savestate information will only be addressed when there has been an incomplete installation prior to the current installation run rootfscalil 11 root tmp SSP install Welcome to the Scali Software Platform SSP installation launcher This program will install the Scali SSP installation package and launch the installer program This program should only be started on the machine you want to be the frontend The frontend is among other responsible of installing Scali software and upgrades on the cluster running the configuration and monitoring servers In small systems lt 8 nodes this is usually one of the compute nodes In larger systems a separate node is preferred The name of this machine scalil 11 Q Are you sure you want to use this machine as an frontend y y Installing ScaSSP package ScaSSP is not installed launching opt scali sbin SSPinstall Scali System Guide Version 3 0 41 C
6. 5 3 4 2 Running ScaMPI MPI programs From the Scali MPI monitor window execution of MPI programs at selected nodes may be controlled To invoke the Scali MPI monitor window select MPI Program from the Run menu Scali MP monitor ies MPI program Browse MPI monitor options J Processes per node 1 Figure 5 10 Scali MPI monitor window From this window a MPI program may bestarted or stopped The following options are available e MPI Program Specify MPI program to be started Any options to the program is also specified here The Browse button may be used for selecting program e MPI monitor options Specify options for mpimon see ScaMPI User s Guide for more information e Processes per node The number of instances of the program to be started on each node Scali System Guide Version 3 0 71 Chapter 5 Scali Universe Run program Start the selected program by pressing this button The program will be started on the nodes selected in the main desktop window Output to stdout and stderr is printed in the centre window While a program is running the button becomes the Stop Program button Stop program Only available when a program is running By pressing this button the program will be aborted All instances of the program at all nodes will be stopped Only MPI programs started from the Scali MPI monitor window may be stopped MPI program Scali MPI monitor opt scali co
7. SCD te EE 144 SCL SWOOP NEE 144 145 Ree EE 169 SCLIN A A T ETTETATATT 166 CR Le 169 196 Scali System Guide Version 3 0 SCICMISO EE 166 elei ltee WEE 166 SCINODGs oa EEN 167 Elle DEE 144 165 A A A G 146 Software installation problem 139 Software Menu ia wiht acini akin nen Eli 75 SSP EE TN 12 CD ROM EE 151 elle ege EEN 152 installation PEO Aia da 35 Uninstall program sisi daniel Eike 59 SSP ua een Gd ae aan rs anna dE NEEN 140 status MUA Oo 78 SUPPORE Herne ns A rer 149 System Configuration Window nennen nnnn nennen ran nnn 65 T Task Force on Cluster Computimg EE 187 terminal Broadcasting mises ea Bein d e EEN deed 74 e 810 0 e ET EE 145 LOS eege eege ee ee ee 145 T ORUS TOPOLOGY EEN 107 TEOUDIESNOGE NG u ae een Rn 139 U USER Oe UE 63 V VAM PIRtrace Installation and User s Guide 187 NM LEW MCN EE 69 W WIKI EPM aa a aida id ia apoco 150 Scali System Guide Version 3 0 197 198 Scali System Guide Version 3 0
8. The configuration files in opt scali etc 156 Scali System Guide Version 3 0 Appendix B Scali Software Licensing B 1 Introduction All necessary components of the Scali software licensing system will be installed and configured during SSP installation by the install program If you for some reason should run into problems or need to alter licenses manually this appendix explains the Scali licensing system in more detail B 2 Requesting licenses Y our first experience with the Scali licensing system will be when you need the license to enable the initial SSP installation Normally this is handled using the procedure described in B 2 1 Automated demo licenses generation from customer ID tag Please notethat different E mail addresses are used for the different types of license requests B 2 1 Automated demo licenses generation from customer ID tag To enable license generation outside office hours and to speed up license generation in general Scali has set up an automated demo license generation service This service provides our customers with demo licenses of 10 days validity 24 hours a day 7 days a week Using this service requires your private customer identification tag which can be found on your package list or invoice The format of the tag is like this SDxxxxxxx where xxxxxxx may be any letter or number combination To request a demo license using your customer ID tag just create an E mail where the subject field
9. Usage opt scali bin scimonitor a lt adapter gt Options a lt adapter gt Adapter number to monitor C 1 3 sciemsg The sciemsg script maps a return value from a call tothe SCI API or a symbolic error typetothe corresponding error message See Table Table 10 3 on page 142 for some examples of symbolic SCI error types Usage opt scali bin sciemsg lt SCI status error message number gt C 1 4 scidbx The scidbx program is a text based command line tool for inspection and manipulation of the SCI hardware through the SCI driver When started scidbx will prompt you with scidbx gt ready to accept commands Type help for further information about available commands in scidbx The scidbx program is only available to root Usage opt scali sbin scidbx a lt adapter gt f Options cor e lt string gt Execute lt string gt as scidbx command in batch mode f Force start even when adapter sanity checks fails h print this text C 1 5 scideb Thescideb program will set the debug level of the SCI driver This setting can change the amount and character of SCI driver information printed in the system log files Usage opt scali sbin scideb V lt new debug level gt Options V E nable verbose mode 166 Scali System Guide Version 3 0 Possible levels in any combination SSCI_INFO SSCI_ICM SSCI_CFG SSCI_UE SSCI_CR SSCI_ENT SSCI_CB SSCI_FABRIC SSCI_PKT SSCI_AEH SSCI_MSEG SSCI_MMAP SSCI_CH SSCI_K
10. You are permitted to print and distribute paper copies of the unmodified online documentation freely In this case you may not charge a fee for any such distribution 8 Export Requirements You may not export or re export the Software or any copy or adaptation in violation of any applicable laws or regulations 9 Termination Scali AS may terminate your license upon notice for failure to comply with any of these License Terms Upon termination you must immediately destroy the Software together with all copies adaptations and merged portions in any form version Id LICENSE_TERMS v 1 2 2001 06 14 09 11 56 taa Exp This is licensed software Before installation can proceed you must accept the following license terms for Scali and integrated third party software Scali AS Software License terms GNU General Public License terms Console Server License terms MPICH License terms OpenPBS Software License terms DG aM Q Select the license terms to be read or type yes to confirm that you have read and do accept the license terms above yes 44 Scali System Guide Version 3 0 4 3 Scali software installation Expert mode e 2 This program tries to reduce user interaction to a minimum in standard mode However if you would like to be able to control the process in more detail Use secure shell ssh scp for installation Modify default install set Install ScaShMem ScaIP or ScaFPDisp Modify node config
11. adapters found check if the SCI hardware is recognised by the OS Are your motherboards supported Check the ScaSCl release notes Check if the SCI hardware are recog nised by the OS Solaris prtconf pv grep PCI 11c8 model PCI 11c 8 658 class Bridge dev model PCI 11c8 0 class Bridge device model PCI 11c8 658 class Bridge dev Linux cat proc pci grep Vendor id 11c8 Vendor id 11c8 Device id 658 LEDs on SCI cards are red yellow 1 One or more of the LEDs on the SCI card are red yellow and the instal lation process is completed Check that cabling is according to the guidelines in section 3 4 3 Check that all cables run from IN con nector to OUT connector and that all links are connected to the same link number on a neighboring node Link O on node A is connected to link O on node B etc Correct the cabeling if errone ous Otherwise take a look at the next issue Red yellow LEDs on SCI cards 2 Oneor more of the LEDs on the SCI card is red yellow and the cabling is checked and found ok Check that all the cables are properly inserted into the connector and tighten all screws Table 10 4 ScaSCI troubleshooting Scali System Guide Version 3 0 143 Chapter 10 Troubleshooting Problem Description Solution Red yellow LEDs on Checking cabling and tightening Check that the SCI driver is loaded on SCI card screws di
12. connection closed 8 3 9 Node definition using Virtual Processors Default setup of node definition is that all nodes are E xclusive Nodes i e only one job will be allowed to run on a node at a time Multiple MPI processes may be started on a node by using the npn option to scasub Node specification may be changed to use Virtual Processors by e unconfiguring of ScaOPBS and making backup of current configuration opt scali libexec scaopbs rmconfig b lt backup dir gt e modifying the file lt backup dir gt nodes Setting np lt no of CPU s in SMP gt for each node e reconfiguring ScaOPBS for use of the modified files opt scali libexec scaopbs config p lt backup dir gt nodes s lt backup dir gt pbs_server conf OpenPBS will then allocate one job pr CPU instead of one job pr node However to make a single job run multiple processes pr node you still have to use the npn scasub option Future versions of ScaOPBS will include an automated option to scaopbs config for configuring of ScaOPBS to use Virtual Processors Please see section 3 2 2 at page 21 in the Portable Batch System Administrator Guide available at the frontend as opt scali contrib pbs doc pbs_admin_guide pdf for further description of virtual processors 120 Scali System Guide Version 3 0 8 4 Example of ScaOPBS usage 8 3 10 xpbs and xpbsmon The standard graphical tools xpbs and xpbsmon are not included in ScaOPBS distribution if you wa
13. file reserved nodes PBS_JOBID at the users home directory Scali Universe scalll Job submission File Options Number of nodes 4 Processes per node A Reserve nodes only Minutes to reserve Job Name Scali Universe scalll Job submission Eile Options Number of nodes 4 Processes per node J 1 Reserve nodes only _ Command Line Popvscalscontrib mpich bin mpirun opt scali examples mpich cpi Browse Job Name Figure 8 3 E xample of submitting a MPICH job from Scali U niverse to 4 nodes one process pr node Scali System Guide Version 3 0 125 Chapter 8 Batch system ScaOPBS Input file Output file Error file E mail address Start time MMDDhhmm se Figure 8 4 Options Advanced window for J ob Submission The Options gt Advanced menu is used for additional parameters for jobs All these entries are optional Input file is the filename used for stdin to the job Output file is the filename for stdout from the job Error file is the filename for stderr from the job E mail address is the email address used for job completion notification Note that sendmail must be operational at frontend for this to work e Start time specifies when the job will be started 126 Scali System Guide Version 3 0 8 8 Using ScaOPBS from Scali Universe 8 8 2 Monitoring jobs from Scali Universe The Scali Universe graphical user interface lets you monitor the queue status for OpenP
14. newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test remote shell access 9 5 scaps process information The command scaps prints processes on nodes in a Scali cluster Usage scaps options options h u user s string V V P d b W Print help text Print help text Only list those processes that match the given username Only list those processes that contains the given string Print machine name and command before each output block Print version Print machine name before each line in each output block Dryrun only print command do not execute Run in background Wait until command has finished on all hosts n n1 n2 Nodenames separated by space characters 134 Scali System Guide Version 3 0 9 6 scakill parallel kill command f lt file gt Use node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test remote shell access 9 6 scakill parallel kill command The scakill program allow you to kill or send a specific signal to processes running on nodes in a Scali cluster Usage scakill options options Print help text h Print help text V Print machine name and command before each output block V Print version p Print machine name before each line in each output block
15. than the associated server process One examples of node processes is the Scali SNMP daemon scasnmpd Communication between modules of control software is over the Ethernet on the local LAN Management and use of the system through Scali Universe can be done from any workstation on the local LAN Through Scali Secure Socket Communication SSSC clusters may even be completely managed over the Internet using Scali Universe clients from a remote workstation 16 Scali System Guide Version 3 0 2 2 Scali system examples 2 2 Scali system examples 2 2 1 Rackmount systems Powerful rackmount computing nodes and easy integration of infrastructure hardware has made 19 racks the favoured solution for professional cluster based computers Scali s turn key systems can be tailored towards customer requirements fig 2 2 Scali systems from our OEM partners has a different visual appearance fig 2 3 Figure 2 2 Scali system cabinet examples Standard half size T erarack left anda ruggedized version for portable supercomputing right Figure 2 3 OEM partner packaging Fujitsu Siemens hpcLine left and Dell right Scali System Guide Version 3 0 17 Chapter 2 Scali system overview 2 2 2 Scali system cabinet For systems built from mini miditowers or other non rack mountable cases the Scali system cabinet provides a practical solution for organizing hardware and infrastructure Each node is put in a stackable enclosure with interc
16. 3 2 C3 routing in a 3D torus TheC3 algorithmis a fault tolerant routing algorithm availablefor systems with a C3 routing license The C3 algorithm is capable of maintaining full connectivity between the remaining nodes even if more than one node have failed 7 4 4 Routing partitions A routing partition partition from here is a collection of nodes which has a route to all other nodes in the same partition via the SCI network Hence all nodes in the same partition can communicate via the SCI interconnect Nodes not part of the same partition does not have a route between them and can not communicate via the SCI network Partitions are reported as a positive integer called partition number for each node If a negative number or 0 is reported this has a special meaning 0 Thenodeis unreachable Power off or considered dead in some sense No MPI programs or anything else may run on nodes with partition number 0 1 Partition number for this node is not known initial state Before routing is set all nodes will have this partition number Scali System Guide Version 3 0 109 Chapter 7 Interconnect configuration A partition may contain from one to all nodes in the system The routing algorithms computes which nodes that are part of what partition If a node is OK but due to routing strategy it is not possible to connect this node to any other node on the SCI network it will receive an exclusive partition number Exclusive partition numbe
17. 3 4 4 Running Parallel Shell commande AAA 73 5 3 4 5 Node terminal SESSIONS nrsnsennennnnernnennnnnnnnnennnnnnnnnennnnnn ann 73 5 3 4 6 Frontend terminal session urnsrsneernennnnnennnnnnnn nennen nn 74 5 3 5 Management EE 74 5 3 6 Software installation Software menu 75 5 3 6 1 Software Installation Window nersesseensnnersnnnnnnnnnnnennnnnn nenn 76 Chapter 6 System Monitoring unusnnnunnnenonnunnnnunnunnnnunnnnnnnnnnnnnnnnnnannnnnnnnannnnannnn ann 77 OLEO VI Wicca deeg nun Sehnen it nn ee 77 DCL Architecture enee degen EEN 77 6 2 Using monitoring from Scali Universe 78 6 2 T The StatuS MENU serisini nante ne id nein 78 6 3 Graphical monttoring EEN 79 6 3 1 COMMON functionality ensernernsennennnnersnnnnnnnnnnennnnnnnnnnnnnen nenn ennnnnnnnnn 80 O23 A E Elle EL WEE 80 6 3 1 2 3D view camera controlS uenseensennneernnennennnnnennnnnnnnnnnnnennnnn nn 80 03 2 COMPA VIEWS RIND 81 6 3 3 Two dimensional 2D bar view 81 6 3 4 3D o7 PEVA E natin EE Ee EE SE ee 82 6 3 5 2D E WA 82 6 3 6 3D ETC 83 6 3 7 3D System View Compound view 83 6 4 USING All AN EE 84 6 4 1 The Alarm window A 84 6 4 2 The AlarmEditO miii aa 85 6 4 3 Alarm log viewer nnensensssnsssnnnnannnnsnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnannnnsnnnnn 87 6 4 4 Example Defining a new aarm 87 6 5 The monitoring daemon SCAMONG AAA 88 6 5 1 Manual Start StOp vs ccd chester Seele ine ng 88 6 5 2 Se ale eg
18. 4 UNK 000 000 000 I I I I 000 000 0 000 The nodeis OK The nodeis UNREACHABLE 0 The node has no node daemon NO_DAEMON 00 The SCI driver is not loaded NO_SCI_DEV INV This is an invalid state and should never occur UNK The node is in state UNKNOWN Example Controller link output with display codes 1 1 0 0 0 0 1 1 0 0 1 1 RG ME E WE OS ee DT 0 0 1 1 0 disabled down 1 enabled up 178 Scali System Guide Version 3 0 D 2 scaconftool command reference unknown unknown D 2 17 ConfTool gt link 5 Thelink command disables or enables links Usage link lt disable enable gt lt controller number gt lt list gt Example link disable 0 all Disable controller O on all nodes D 2 18 ConfTool gt update The update command ask the configuration server for updated system information Usage update lt list gt system Example update scali2 13 scali2 23 Ask for updated information on nodes scali2 13 and scali2 23 Example update system Ask for new information on whole system This command is more or less obsolete as updated system information is received each time the server is contacted that is for nearly every command performed from scaconftool D 2 19 ConfTool gt restart The restart command does an on line restart of the configuration server Usage restart D 2 20 ConfTool gt sciping The sci ping command
19. Adding a queue qmgr is used for adding extra queues In this example a queue called fluent is added The queue is configured to run a single job at a time to handle software license restrictions opt scali contrib pbs bin qmgr create queue fluent set queue fluent queue_type Execution set queue fluent max_running 1 set queue fluent enabled True set queue fluent started True 8 3 5 Removing all ScaOPBS configuration files All ScaOPBS configuration files may be removed by running the script opt scali libexec scaopbs rmconfig 118 Scali System Guide Version 3 0 8 3 OpenPBS configuration 8 3 6 ScaOPBS file locations The ScaOPBS package includes all files for Open PBS for the frontend running the PBS server and scheduler and for the nodes running pbs_mom only ScaOPBS files are installed in opt scali contrib pbs One exception is the script opt scali bin scasub scasub s described in scasub on page 122 ScaOPBS configuration and spool files are located in var spool PBS All configuration and spool files are removed when running opt scali libexec scaopbs rmconfig 8 3 7 Using aliases for host names Note for installation on nodes where hostname aliases are used If hostname aliases are used in any of the SSP configuration files mentioned in Manual reconfiguration of ScaOP BS on page 115 thealiases must beincluded in the etc hosts file i e hostname a must report the hostname aliases Oth
20. EE 104 7 3 5 Setting SCI nodelD manually ersenssennnnersnnnnnnnnnnnnnnnn nennen nn 104 7 3 6 Reloading SCI driver uusnensrsssnnsnnnnnennnnnnnnnnnannnnsnnnnnnnnnnnsnnnnennnnnn nn 105 7 3 7 Access to console on one noe 105 7 3 8 Hot restart of the configuration server ueerseensnennnnennnnnnnnnnnnen nennen 105 RECH Ce Le Dale EE 106 7 3 10 Daemon control with scaconftool EN 106 7 3 11 Setting server options sidras siais 106 KE D Lu EE 107 FALL O o o A seduces ea ieiveceedutemeedss 107 7 4 2 2D Torus tOPOlOOY AAAA E non on cnn nn nro nr ron rnnn nr nenn nennen nn 107 7 4 2 1 Dimensional routing in 2D torus E 108 7 4 2 2 Maxcy routing algorithm for 2D tous 108 7 4 3 3D Re ee e OT 109 7 4 3 1 Dimensional routing in a 3D torus E 109 7 4 3 2 C3 routing iN a 3D torus ueerseensnennnnersnnnnnnnnnnnen nenn nnennnnn nn 109 LAA Routing partitions e eignen rn 109 6 Scali System Guide Version 3 0 7 4 5 Testing the FOULING EEN 110 7 5 The configuration Server daemon eerssssennennnnennnnnnnnnennnnnnnnnnnen nn rn rr 110 7 6 The ScaConf node daemon nn nennen nn 111 Chapter 8 Batch system Sca0PBS nnuenonnunnnuunnunannunnnunnnnunnnnunnnnunnunnnnnunnnnannnnnnnnn 113 BT OVE VIEW i ernannt DR tt 113 8 2 SCAOPBS Installation enan inn rani errea ara aA nn 114 8 2 1 Installing OpenPBS during SSP installation 114 8 2 2 Installing ScaOPBS after SSP installation esseere 115 8 2 3 Running the ScaOPBS
21. MPICH window to control the execution of M PICH jobs on the system Submit Job If a queue system is installed this will open the job submission window of the queue system Its use is described in detail in section 8 8 1 on page 124 Parallel Shell Opens theparallel shell window torun shell commands on all selected nodes Node Terminal Opens terminal window sessions on all selected nodes Frontend Terminal Opens a terminal session to the frontend of the system Table 5 3 Main window Run menu items explained 5 3 4 1 Working with runsets TheScaMPI monitor window MPICH launcher and queue submission window all supports runsets A runset is a collection of all information needed torun the program program name program command line options and in case of ScaMPI MPI monitor options The idea is that you can save time and effort by creating runsets for programs you use frequently You will find the runset functionality under the File menu where available Menu Item Description Open Runset Opens a runset replacing current contents of window Save Runset Saves current contents of window as a runset 70 Scali System Guide Version 3 0 5 3 Using Scali Universe Note N ode selections are not part of the runset This means that when opening a runset you ll still have to select nodes to run on It also means that you can use the same runset on a different set of nodes than it was created for
22. can get tremendous speed up from the faster 64 66 PCI bus operation a few things to remember here are 1 Makesure your SCI card does not share the PCI bus with a slow device This will force the entire PCI bus to operate at the lower speed 2 If used make sure the raiser card supports full 64 66 operation For 1U and 2U rack mount cabinets it is common to use a raiser card to enable one or two full size PCI cards to be mounted horizontally Scali System Guide Version 3 0 25 Chapter 3 Hardware installation 3 4 3 SCI Cabling The number of nodes in the system and the available SCI hardware determines your options for selecting an SCI topology Our recommendations can be found in table 3 1 Cabling suggestions for the different topologies will be described in the following sections With Scali turn key systems you can find a detailed description of the SCI topology the interconnect section of the Scali System Configuration Description Topology nodes links node Supported SCI cards Ring 2 8 1 Any D31x or D33x 2D Torus 4 128 2 D311 D312 D315 D316 D335 D336 D337 D339 3D Torus 64 256 3 D336 D337 Table 3 1 Topology vs SCI hardware compatibility 3 4 3 1 General rules Interconnect cabling of simple topologies are somewhat complicated by system cabinets with another size and shapethan the interconnect topology The need to avoid long SCI cables adds another level of complexity to the
23. changes colour from white to red You may reset the link error counters from the SCI link status window pop up menu 7 2 2 2 SCI link status window options Thelink status window provides a pop up menu for setting options E xcept for the Reset Counters option these are standard options for any monitoring window Right click inside the window to bring up the menu 98 Scali System Guide Version 3 0 7 3 Command line interface scaconftool Menu Item Description Large Icons Select the default view with large icons and information as described above Small Icons Select compact view with only status LED indicators and node names this allows for compact monitoring Reset Counters Will reset the link error counters for all links but only for this instance of the link status window Close Close the window Table 7 2 SCI link status window pop up menu options Note that resetting link error counters only affects this instance of the link status window You will not interfere with other people monitoring the same system 7 3 Command line interface scaconftool The scaconftool ASCII based command line interface is a fast and efficient way to access the ScaConf functionality An on line plain text manual for scaconftool may be found i the file opt scali doc ScaConfC scaconf users instruction txt YOU may want to check out this for last minute information which may not have made it into the manual 7 3 1 St
24. command The fix command attempts to bring all specified nodes in a system back to an OK state by re establishing connections and restarting daemons as necessary The following algorithm is used e For all of the nodes that are in state UNREACHABLE Attempt to reconnect nodes For all nodes still in state UNREACHABLE If the node answers to ping restart the communication daemon Wait for some time Attempt to reconnect the nodes again e Then for all of the nodes which arein stateNO DAEMON Attempt to reconnect nodes For all nodes still in state NO_DAEMON Restart configuration daemon on the node Wait for some time Attempt to reconnect the nodes again e Then for all of the nodes which arein state NO_SCI_DEV Attempt to reconnect nodes For all nodes still in state NO_SCI_DEV Reload SCI driver on the node Wait for some time Attempt to reconnect the node It may be necessary to run the ix command more than once if the process of restarting daemons takes a long time due to heavy load on the system When a daemon is started this is done with the scash utility scaconftool will wait for a given amount of time By default this is Number_of_nodes WaitConst seconds WaitConst is 0 7 by default causing a system with 8 nodes to wait for approx 6 seconds before it will try to reconnect a node WaitConst may be tuned when scaconftool is started with the s option frontend gt scaconftool s0 5
25. configuration 7 3 9 Failing nodes Used mainly for testing purposes the fai1 command allows you to soft fail a node regardless of whether the node is alive or not This will force the node into the UNREACHABLE state in the configuration server database but it will not affect the node apart from the fact that this node will be considered failed if the cluster is rerouted The fail command is useful for generating and setting routing patterns that normally only will occur if the node actually has failed A node failed with the fail command may be brought back to its correct state with the reconnect Or restart commands In order for the soft failing of a node to last more than a few seconds the AUTO_REPAIR option in the server must be turned off prior to failing the node 7 3 10 Daemon control with scaconftool The two daemons scaconfnd and scacomd may be started stopped or restarted on all nodes from the tool This is done with the daemon command If you want to stop and start the communication daemon on all nodes type this in the tool ConfTool gt daemon scacomd restart all If you want to stop and start the configuration daemon on all nodes in state NO DAEMON no configuration daemon you type this in the tool ConfTool gt daemon scaconfnd restart NO_DAEMON The daemon command uses scash to access all nodes and the ScaSH package must be installed on the machine running the tool in order for the daemon command to work from the t
26. default values is available These default values have very limited use since they will only work if you run the desktop locally on the frontend of a Scali system Hence you should configure Scali U niverse to enable remote access to this and all other available Scali systems Select EditConfiguration from the main menu to open the system configuration window Scali system System name Scali system Custer type generic Frontend flocalhost Scali directory fopt scali _Add Remove Save Restore Ses en Re Figure 5 4 Configuration window showing system defaults Inthe System Configuration window select the Scali system and edit System name and Frontend to the correct values for your system U sually default values for Cluster type and Scali directory are correct and can be left unchanged To make the changes take effect and save the newly edited configuration press Save More Scali Systems may be added to your configuration like this Press the Add button and you will be prompted for a new system name Enter a unique system name edit the Frontend name and press Save again System name lpha C Frontend scalio 1 1 Scali directory fopt scali _Add Remove save Restore Figure 5 5 Example System configuring Scali System Guide Version 3 0 65 Chapter 5 Scali Universe Removal of undesired systems is done by selecting a system and then pressing the Remove button TheRestore button wil
27. ethernet connection serial switch Scali System Guide Version 3 0 49 Chapter 4 Software installation 9 Finish configuration Q Please enter your selection 1 1 Q Please enter hostname for power switch number 01 scali pow3 Please select what type of switch the power switch number 02 is Select switch type by entering a number from the list below 1 Baytech RPC 3 with ethernet connection 2 Pulizzi with ethernet connection serial switch 9 Finish configuration Q Please enter your selection 1 9 The following power switch es have been defined Power switch Type Configuration powsw01 baytech scali pow3 Q Do you accept the above configuration y Specifying power switch map _ 2 Please specify the power mappings Please specify which power switch node scalil 11 is connected to Power switch Type Configuration powsw01 baytech scali pow3 Q Please enter power switch for host scalil 11 powsw01 Q Please enter power switch port outlet number for host scalil 11 1 1 Please specify which power switch node scalil 12 is connected to Power switch Type Configuration powsw01 baytech scali pow3 Q Please enter power switch for host scalil 12 powsw01 Q Please enter power switch port outlet number for host scalil 12 2 Please specify which power switch node scalil 21 is connected to Power switch Type Configuration powsw01 baytech scali pow3 Q Please ente
28. existing frontend oe OK The following node will be used scalil 11 Checking Frontend OK Find existing nodes ___ OK The following nodes will be used scalil 11 scalil 12 scalil 21 scalil 22 Find existing repository __ 77 oK The following repository path will be used opt scali repository Checking access to nodes OK Remove installation Confirmation ee 2 You will now remove the SSP from all involved nodes Q Please type yes to confirm no to decline no yes Will you also remove the repository on the frontend opt scali repository Q Please type yes to confirm no to decline no yes Remove packages and files __ Removing packages on nodes please wait Removing config files on nodes Removing references to opt scali Removing references to opt scali on frontend Removing packages on frontend please wait Removing config files on frontend Removing repository on frontend Uninstall complete II pe IE pe IE 2 I de IE 24 I I 4 I 2 2 I 2 tr 2 r gt Scali System Guide Version 3 0 Chapter 5 Scali Universe Aimed at all users of Scali systems this chapter explains the organisation of and how to use the Scali Universe graphical user interface Although Scali Universe consists of a great number of other software components workin
29. installation tests manually cece 115 8 3 OpenPBS configuration een id 115 8 3 1 Manual reconfiguration of SCaAOPBS nnsnsersennnnnnnsnennnnnnnennnnnn nn nn 115 8 3 2 Backup of local configuration les 116 8 3 3 The node property file uneersennnnnnnnnnnnnnnnnnnnnnnnnanannnnnnnnnnnnnnnnannnn nn 117 8 3 4 Server configuration omg 117 8 3 4 1 Listing the current server configuration 117 8 3 4 2 Adding a TI TC 118 8 3 5 Removing all ScaOPBS configuration files A 118 8 3 6 SCAOPBS file locations unnerseennnennnnernnnnnnnnnnnnnnnnnnennnnnnnnnnnen nn 119 8 3 7 Using aliases for host names nnennnennnennnnennnnnnnennnnnen nennen nenn 119 8 3 8 PBS ut GC Cl 119 8 3 9 Node definition using Virtual Proceeseors AA 120 8310 Xpbs ANA POSI EE 121 8 4 Example of SCAOPBS usage nnnesnennnensnnennnnnnnnennnnnnnnnnnnennnnnnnnnnnennnnnnnnnannn anna 121 8 5 Open PBS comMaAnGs E 122 SiO SCASUD EE 122 8 7 Starting MPI jobs from within scripts submitted to OpenPBS AAA 123 8 8 Using ScaOPBS from Scali Universe ENN 124 8 8 1 Submitting jobs using Scali Universe A 124 8 8 2 Monitoring jobs from Scali Universe 127 8 8 3 Enable and disable OpenPBS queue enforcement A 128 B D SOURCE COAG Ae eegen NEEN Aen E ee A Een Eu de 128 8 10 Software EEN 129 Chapter 9 ScaSH parallel shell tools uannruonnunnnnunnnnunnunnnnnunnunannunnnnunnnnnnnnnnun 131 9 1 scash the parallel shell commande 131 9 2
30. interconnect cabling pattern Some general rules apply though when connecting the cables e Always connect the SCI cables from the OUT connector to the IN connector Cables are colour coded to help you getting it right e The basic component in any SCI network is a ring Scali system rings must always be connected between the same SCI link on all nodes i e LO outputs to LO inputs and L loutputs to L1 inputs e Interleave cabling to avoid long cables see figure 3 8 e Keep the number of nodes on the SCI rings in each dimension of the torus as even as possible Many short SCI rings is better than a few long ones A 16 node system should therefore be connected as 4x4 not as 8x2 26 Scali System Guide Version 3 0 3 4 Dolphin SCI interconnect installation 1 IN 1 HS OUT 1 NM mw 1 OUT y NA ARI 6 OUT 1 HS Ga ae CW 1 OUT 1 242252228 3 OUT 1 LI leg H2 OUT 1 ee ccm a 1 OUT OG Figure 3 8 Avoid long cables by interleaving interconnect in each ring 3 4 3 2 Single SCI ring cabling In small systems a single ring can be efficient Figure 3 9 shows how to connect an 8 node SCI ring based on the table below From OUT To IN Node 1 Node 8 Node 8 Node 5 Node 5 Node 7 Node 7 Node 8 Node 8 Node 6 Node 6 Node 4 Node 4 No
31. is set to the tag number only the SDxxxxxxx part and send it to demolicense scali com Within a few minutes you should receive a new demo license Note that you must send this E mail from a working E mail account B 2 2 Demo licenses general requests Scali will also provide demo licenses for evaluation purposes Since demo licenses are valid for all hosts until the expiration date no system information is required Y ou may send a simple E mail request or use the License Request form in the support section of the Scali web site General demo license requests should always be directed to sales scali com If your request is granted you will most probably receive a customer ID tag which can be used to get demo licenses following the procedure in B 2 1 Scali System Guide Version 3 0 157 Appendix B Scali Software Licensing B 2 3 Permanent licenses Permanent licenses for Scali software are node locked therefore in order to receive a permanent license you will need to supply Scali with the hostid for all nodes in the system including the hostname and hostid of the front end and some other technical details You must also provide 1 Resellers company name if applicable 2 License owner s company name 3 E mail address to recipient of license keys 4 Your order reference number Requests for permanent licenses should always be sent to license scali com B 2 3 1 Permanent license request format example Here is a
32. maxtime lt mt gt Max time for mpi job in minutes ScaMPI only m lt addr gt e mail lt addr gt when job completed N lt job name gt Specify name for job s lt system gt submit job to given system resources ns submit job to any system independent of arch r lt minutes gt reserve nodes for lt minutes gt nodes listed in file reserved_nodes PBS_JOBID env lt env gt export list of environment variables format varl var2 122 Scali System Guide Version 3 0 8 7 Starting MPI jobs from within scripts submitted to OpenPBS o lt filename gt specify filename for stdout e lt filename gt specify filename for stderr q Quiet no echo of mpimon mpirun command debug Debug print debug output Q lt queue gt Destination queue for job 8 7 Starting MPI jobs from within scripts submitted to OpenPBS Sometimes it is not convenient to use the scasub script for starting ScaM P applications directly e g when the application has its own start up script The user may then submit the application start up script using scasub and specifying the number of processes by the np option and possibly the number of processes pr node with the npn option When the application is started by the queue system the script is started at the first node allocated by the scheduler This script may then determine the nodes allocated for the job from the file specified by the environment variable PBS _NODEFILE This fi
33. reason would like to have the console output routed to the serial port 4 2 5 1 Configuring serial console on Linux Edit etc inittab to set up console over serial port RH 6 x 7 0 S0 2345 respawn sbin getty ttyS0 9600 vt100 RH 7 X S0 2345 respawn sbin agetty ttySO 9600 vt100 SuSE Run yast choose System administration Change configuration file and set the entry SERIAL_CONSOLE to ttyS0 9600 Add ttyso to etc securetty Scali System Guide Version 3 0 37 Chapter 4 Software installation Edit etc lilo conf to enable 1i1o to operate over serial port and inform kernel to use console over serial port Include these lines in etc lilo conf serial 0 9600n8 append console ttyS0 9600n8 and run sbin lilo to make the settings active Reboot the system 4 2 5 2 Configuring serial console on Solaris Configure with eeprom as root eeprom ouput device ttya eeprom input device ttya 4 2 6 M Use a common file system for MPI applications You should have a common file system for MPI applications Usually this means mounting home to a common NFS server 4 2 7 O Use NIS We recommend using NIS on the cluster to ease management Hereis a short description of how to enable it 4 2 7 1 Red Hat Linux NIS setup On the NIS master usually the frontend e set NIS domainname run nisdomainname lt yourdomain gt e edit etc yp conf e edit etc nsswitch conf e edit var yp securenets e run sh etc rc d init d y
34. remove the pam_securetty check or add hosts to pam authentication RH 7 x Add login to the file etc securetty SuSE Run yast choose System administration Change configuration file and set the entry ROOT_LOGIN_REMOTE to yes 4 2 4 M Ensure the at service is working You must have a working at daemon atd running on all nodes Verify that it works 36 Scali System Guide Version 3 0 4 2 Operating system preparation echo touch tmp atfile at now test r tmp atfile amp amp echo OK test r tmp atfile amp amp echo ERROR If not please do the following steps Linux Check that package is installed rpm q at Ensure that the at daemon will be started at boot RH chkconfig atd on SuSE Run yast choose System administration Change configuration file and set the entry sTART_ATD to yes Check if the daemon is running RH etc rc d init d atd status Start the daemon if not etc rc d init d atd start SuSE ps aux grep atd grep v grep Start the daemon if not Solaris Ensure that the at cron daemon will be started at boot test r etc rc2 d S75cron amp amp In s etc init d cron etc rc2 d S75cron Check if the cron daemon is running ps ef grep cron grep v grep Start the cron daemon if not etc init d cron start 4 2 5 O Set up console over the serial port This is only necessary if you want to make use of console switching hardware or for any other
35. scahosts nodes availability check AA 132 9 3 scacp Local file COPY icon tal diri aida 133 9 4 scarcp remote file e UN 133 9 5 scaps Process information ooconnccncccnacinnnncnancnonnnnn cnn nc rro n nn nn rr rr arca rre ran 134 9 6 scakill parallel kill COMMANG 00 ANN 135 eege ne Ee DEE 136 Scali System Guide Version 3 0 7 9 8 scawho user Ipformatlon AAR 137 9 9 SCASH configuration fil Esisi aasre anii aapi nna Eip aange 137 Chapter 10 Troubleshooting s s sssssssssunsnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn annann rre 139 10 1 Hardware installation problems AEN 139 10 2 Software installation PrODleMS ANE 139 10 3 License e e TE EE 140 MER OM E EE 141 10 5 SCI Interconnect problems AAA 141 10 5 1 SCI error MESSAGES ueenneennannnnnnnnnnnnnnnnnnsnnnnnnnnnnnsnnnannnnnnnnnnnnennnnnnnann 141 10 5 2 SCI Troubleshooting EE 143 10 53 ce link EE 144 10 931 50 te DEE 144 10 5 3 2 S amp i NWROD DEE 145 Ee e EC EE 146 10 6 Technical SUPPO eise aaa sn bin bahn 149 10 6 1 SUPport ee ul le CR 149 10 6 2 How to report the problem AA 149 10 6 3 Other Heedbadde cisco ic aab heat dabas 150 10 6 4 Keeping informell 150 Appendix A Scali software details unusonnunnnnunnnunnnnunnnunnnunnnnnunnunnnnunnnnnannnnnnnnnnnn 151 A 1 General information EE 151 A 1 1 Scali software platform CD ROM 151 A 1 1 1 Installation for the impatient AAA 151 Lake SR Il Le EE 152 A 1 2 Scali software directory strucure nen
36. the server is started from opt scali sbin scaconfsd binary The server maintains a database with the current state of the system The database contains among others e Theinterconnect configuration of the machine For each node e Nodenameand SCI nodelD e Routing tables e Number of adapters and controllers in the system 110 Scali System Guide Version 3 0 7 6 The ScaConf node daemon e Link status This status can be either UP or DOWN indicating if the connection is ok or not e Status for each controller A controller can be either enabled or disabled e Status for each node regarding the communication between configuration server and configuration daemon e Whether a SCI driver is available or not on each node e Routing partition number for each node When started the server will poll every node for status information If the AUTO_REROUTE server option is enabled it will reroute the cluster upon start and at every event that demands rerouting Such events are node or link failure or change of SCI nodelD For further information about server options read section 7 3 11 7 6 The ScaConf node daemon For the configuration server to collect information from the nodes a configuration daemon must run on all nodes in the machine The configuration daemon will be started upon reboot on each node when installed It may also be started manually with the standard opt scali init d scaconfnd start stop restart script on e
37. the front end with an updated license file in the default location you may copy the license file to the same location on all nodes with the command opt scali bin scarcp opt scali etc license dat After the new license file has been installed you must also restart the daemons that depends on licensing Assuming that you re still on the front end you can restart the ScaMon daemon with the command opt scali init d scamond restart If you have SCI installed you must also restart the ScaConf server daemon with the command opt scali init d scaconfsd restart Scali System Guide Version 3 0 161 Appendix B Scali Software Licensing B 5 Meta licensing Starting with SSP 3 0 0 Scali introduced the concept of meta licensing to simplify license handling Using meta licensing each Scali core product Universe Universe XE ClusterE dge will need only one FEATURE linein the license file This single FEATURE line will unlock all individually licensed sub components of the product This is differs form SSP 2 X releases where you would need a FEATURE line for every component Thefile opt scali etc metalicense dat controls the different meta licenses Please note that there is an unchangeable one to one relationship between SSP releases and the meta licensing file End users should never attempt to do anything with this file Changing the metalicense dat file will make your system useless B 6 Troubleshooting If your program fails to
38. the nodes Usage reload lt list gt Example reload scali2 22 Reload SCI driver on node scali2 22 Please note that a reload of the SCI driver also implies that several other daemons are restarted e g scid mpid scaconfnd scasnmpd D 2 15 ConfTool gt reroute The reroute command does reroute the cluster with specified algorithm Use info routalg to get a list of available routing algorithms for your topology Usage reroute routing algorithms Example Scali System Guide Version 3 0 177 Appendix D ScaConf Reference reroute maxcy Use the maxcy routing algorithm to route the 2d torus D 2 16 ConfTool gt status The scaconftool command status prints information about nodes or links according tothe cluster configuration The output looks like the list configuration command but instead of node name a three letter code is indicating the status of the node when nodes is the argument E nabled disabl ed or up down information is displayed if controllers or links is the argument In 3D torus topology the torus is stacked into layers The lt stacked Link parameter specify which link that is not shown in the output Default is link 2 Usage status nodes controllers links lt stacked link gt Example status nodes Print status information for each node Example status controllers Print information about enabled disabled links Example output for nodeargument which display three letter status codes 1 2 3
39. the state of the system resources and the scheduler to learn about the availability of jobs to execute The pbs_sched daemon is running at the frontend node 8 2 ScaOPBS Installation 8 2 1 Installing OpenPBS during SSP installation ScaOPBS can be installed by the Scali Software Platform SSP installation During the installation the user will get the following question You may install the free queue system OpenPBS www openpbs org The ScaOPBS package is a binary distribution of OpenPBS with easy installation and configuration of OpenPBS and job submission script for ScaMPI and MPICH applications Q Do you want to install OpenPBS queue system n y Requesting username 2 2 2 Testing of OpenPBS requires a non root username the username must be defined at all nodes Use empty username default to skip the OpenPBS test Q Please give username lt testuser gt 114 Scali System Guide Version 3 0 8 3 OpenPBS configuration OpenPBS does not allow root to submit jobs hence to be able to test the installation a username is required The user must be properly defined at all nodes in the cluster for the test to pass If a username is provided the installation script will also test OpenPBS at the end of the installation Testing Cep AAA A OK 8 2 2 Installing ScaOPBS after SSP installation If ScaOPBS was not installed during the SSP installation you may run an upgrade to install ScaOPBS at y
40. this node is part of Hereis an example nodelist defining four nodes node 1 node 2 node 3 node 4 and a frontend machine boss Format node lt name gt lt categories gt node node node node 184 node 1 sci_node node 2 sci_node node 3 sci_node node 4 sci_node eth_node eth_node eth_node eth_node Scali System Guide Version 3 0 E 2 Configuration node boss sci_frontend eth_frontend smp_node E 2 6 Package response files If a file with the name lt package gt response is found in the package repository without architecture extension this file will be read when installing lt package gt Example is ScaSCI response No user changes should be necessary for these files Scali System Guide Version 3 0 185 Appendix E ScaPkg Scali Software installation program 186 Scali System Guide Version 3 0 Appendix F Related Documentation F 1 References 1 MPI A Message Passing Interface Standard The Message Passing Interface Forum Version 1 1 June 12 1995 http www mpi forum org 2 MPI The complete Reference Volume 1 The MPI Core Marc Snir Steve W Otto Steven Huss Lederman David W Walker Jack Dongarra 2e 1998 The MIT Press 3 MPI The complete Reference Volume 2 The MPI Extension William Grop Steven Huss Lederman Ewing Lusk Bill Nitzberg William Saphir Marc Snir 1998 The MIT Press 4 Scali Library Guide Scali AS http www scali com 5 ScaM
41. through 4 If you decide to use another connection scheme this presents no problem for the system but you will spend moretime with the console server configuration without the benefit of intelligent default values Note f you use a private network with your system we strongly recommend that the serial switches are connected to the public network If not you may have severe problems reaching the consoles of your nodes if the frontend gateway goes down With Scali turn key systems TeraRack console switching is preconfigured A description can be found in the Scali Configuration Description delivery document 32 Scali System Guide Version 3 0 3 6 Power switching 3 6 Power switching Like with console switching centralized and remote power switching is an obvious advantage for cluster systems management The SSP products UniverseXE and ClusterE dge both supports remote console switching and console broadcasting provided that you use one of the qualified remote power switches Please refer to the SSP release notes for an updated list of supported hardware Cabling of the remote power switches is really simple Power cords must be connected between the power switch power outlets and the nodes power input If you follow a simple sequential connection scheme you should be able to get sensible default values also when configuring the power server during SSP installation later on An explanation of what we mean by sequ
42. topology rings and positions Default is print mode The print and write modes will find the connection list and topology used in the lopt scali etc ScaConf conf file and print them to screen or write it directly to the file Only unbroken rings will be printed in theconnection list The topology will be one TOP_RING _CR TOP_TORUS_ 23 D or TOP_FREE_ D Thering and torus topology are the ones supported Still no error messages will be printed if the cluster is not connected in a non supported topology ie TOP_FREE_ D The topology will be TOP_FREE_ D if some nodes or link controllers are down The connection mode checks if the nodes are connected according to the connection list in the opt scali etc ScaConf conf file If the connection list is correct O will be returned If not 1 will be returned and error messages indicating what might be wrong will be printed The same goes for the topology mode 0 will bereturned if the cluster is connected in a supported topology that is a ring or torus topology If not 1 will be returned and error messages will be printed The rings mode prints the rings in a more reader friendly manner with one ring per line and the positions will print the nodes positions in a ring 2D or 3D topology with respect to an origin node There are several options to change the behaviour of sci_hwtop With f lt ScaC onf conf file gt sci_hwtop will get the rings from this file rather than perform hardwar
43. type of SCI cards D33x 10 5 3 SCI link errors For systems with an explicit support license a special test programs for the SCI network is available in the ScaDiagSC package ScaDiagSC is intended for use at troubleshooting problems related to SCI hardware and consists of three programs sci_hwid sci_hwtop and sci_hwdiag They are all ment to be used when the system is operational and may be used to find failing cards and cables that are not detected by hardware or topology miss configurations 10 5 3 1 sci_hwid The sci_hwid program checks that every link controller on the nodes have a unique SCI IDs If every ID sin fact unique O will be returned to the calling environment If not 1 will be returned and an error message will be printed The 1 lt le list gt input parameter determines which link controllers to check Default is to check link controller 0 In a 2D torus there are two link controllers To check them both use opt scali sbin sci_hwid 1 o 1 If the program discovers any non unique SCI IDs please contact your SCI hardware vendor to sort this out 144 Scali System Guide Version 3 0 10 5 SCI Interconnect problems 10 5 3 2 sci_hwtop The sci_hwtop program performs a series of link controller resets to figure out how a set of nodes are connected topology sci_hwdiag will get the node names from opt scali bin scahosts z You may run sci_hwtop in one of several modes print write connections
44. types of errors may show up during the run You should pay special attention to large amount of link errors nodel 1 LC O link errors 27 148 Scali System Guide Version 3 0 10 6 Technical Support A bad cablein the cluster may show up as link errors when running either the IcO_rings or the c1_rings The bad cable may bein connection with the node on which the error occurred but not necessarily Due to the nature of the SCI ring errors are not always discovered where they are generated but may propagate to several nodes in thering If you get errors when running rectangles but none when running IcO_rings nor Ic1_ rings this may indicate a bad SCI board Errors evenly distributed all over the cluster may bea sign of some other problem not related to bad SCI boards or cables Not all counters reported are due to errors The message nodel 1 Times handler called and interrupt not claimed 90302 may indicate that the SCI card share interrupt with another device typically your NIC Thisis not an error but may in certain circumstances affect system performance 10 6 Technical Support Scali is proud to provide the highest level of technical support Normal first response time for a support request is less than 8 working hours Y ou will then be appointed a personal contact and a serial number for your particular problem 10 6 1 Support Contracts The duration and level of your support contract or warranty should be obvi
45. was run if you run b_link sh off twice b_link sh on will have no effect since it sets blink _retry count tozero Default value for b link retry is 64 Make sure routing is OK after turning b link on off The way of the program Before the test application is started the cluster is set in diagnostics mode This involves turning of all automatic modes in ScaConf server daemon scaconfsd and it may also imply reconfiguration of the ScaSCI driver Default ScaSCI driver configuration for diagnostic mode is max_trans_size 64 Ic_ec_period 0 and scilink_freq 200 These parameters will be reconfigured with scireconf F or the reconfiguration to take effect the ScaSCI driver is reloaded on all nodes before the MPI test application starts When the tests are finished these parameters will be set back to their original values If the value originally was not the same on all nodes the parameter will be commented out from the file opt scali kernel ssci conf and default value will be used when the ScaSCI driver is reloaded With the S flag you may add your own parameters to be reconfigured or override the default Do not use this flag if you don t Know what you are doing The flag is followed by a string with format lt parameter name gt lt value gt The parameter you specify with the S flag will be set when the diagnostic mode is applied When the MPI test application is run the cluster will be divided into rings and rectangles th
46. 1 2 0 2002 6 13 12 0 1 0 2 0 1 3 6 1 2 1 25 1 3 0 1536 1 3 6 1 2 1 25 1 4 0 auto BOOT_IMAGE linux2417 ro root 4806 BOOT_FILE boot vmlinuz 2 4 17 2smp console ttyS0 a 3 6 1 2 1 25 1 5 0 Gauge32 4 3 2 1 25 1 6 0 Gauge32 133 735 6 1 6 1 2 1 25 1 7 0 0 Scali System Guide Version 3 0 91 Chapter 6 System Monitoring This example queries the Scali SNMP daemon scasnmpd which runs on a separate UDP port 32016 and on the community birka You re likely to use a separate SNMP daemon that has a different port and community The regular community tend to be public As a hint many SNMP daemons only support SN MP version 1 and you ll need to give the v 1 option tosnmpget Once you areabletoread the SNMP variables OI Ds you wish to monitor make a note of the options to snmpget you ll need them in later when editing the config file 6 7 2 Editing the ScaMon configuration file On the frontend node there is a file opt scali etc ScaMond conf where all the necessary configuration is done This file has a number of sections where as four are of interest and will require some editing 6 7 2 1 Editing the class definition The first section of interest is the class definition class name type format cmdline class scasnmp internal s s snmpaget p 32016 s birka s Where there is a scasnmp class for the Scali SNMP daemon The snmpaget is an internal snmpget that does a parallel query to many
47. AL SSCI_SAL SSCI_ICME SSCI_INTR SSCI_SOFT SSCI_MEM SSCI_MCTX SSCI_PG SSCI_CH_V SSCI_RCHUNK SSCI_LCHUNK C 1 6 scinode 0x1 0x2 0x4 0x8 0x20 0x40 0x80 0x100 0x200 0x400 0x800 0x1000 0x2000 0x4000 0x8000 0x10000 0x20000 0x40000 0x80000 C 1 SCI hardware status programs Various infrequent info Use of KalLog in OS independent code Configuration info Details about user errors Connect and release details Entry and exit of entry points Entry and exit of callbacks Interconnect fabric related conditions SCI software packet interface Asynchronous error handling Memory segment details not too verbose Entry in mmap Channel interface Enter and exit of Kal funs except mutexes Suballocator Use of KdPrint in OS independent code User interrupt processing DDI_soft Memory allocation and export process 0x100000 M map context adm 0x200000 Page pool adm info 0x800000 Verbose printout for channel interface 0x2000000 P Remote connect disconnect details 0x4000000 Local chunk maintenance details The scinode program prints sets nodeid of the specified adapter Default is to print the nodeid of adapter number 0 on standard output Only root may set nodeid The scinode options are listed below Usage opt scali sbin scinode h v s lt new nodeid gt a lt adap
48. BS Select Status gt Queue Status to open queue monitoring window Scali Universe scalll Queue status Max Tot Ena Str Que Run Hid Wat Trn Ext Type NDS Cpu Time Wall Time S Queue all2all 4 00 00 00 00 00 16 R scali_exec all2a11_2 4 0 0 Q scali_exec cpi 4 0 0 Q scali_exec Figure 8 5 E xample Queue Status display in Scali Universe When you select a running job in the Queue Status window the nodes used by a running job will be marked in the corresponding Main Window of Scali Universe Use the left mouse button to select job If you select a job and press the right mouse button you may select actions to perform on the job from a pop up menu e details detail view of job attributes see example below e delete delete job only allowed by root or job owner e hold put the job on hold only allowed by root or job owner e release release the job only allowed by root or job owner e order order job only allowed by root or job owner Scali System Guide Version 3 0 127 Chapter 8 Batch system ScaOPBS Figure 8 6 E xample Queue detail window in Scali Universe 8 8 3 Enable and disable OpenPBS queue enforcement In administrator mode it is possible to enable and disable OpenPBS queue enforcement Select Management gt Queue E nforcement gt E nable Disable For further information about OpenPBS queue enforcement please see PBS enforcement on page 119 8 9 Source code The full OpenPBS source cod
49. Ctrl gt H eft click Selected nodes will appear with a grey background in the window An operational node Node for which the running the Linux status and operating A operating system system has not yet been determined An operational node Unreachable node running the Solaris operating system Table 5 1 System M ainWindow node symbols 5 3 2 MainWindow menu overview Thecontents of the main menu will vary slightly depending on whether you are logged into the system as root user or not At the top level it is only the Software menu which is used for software installation that disappears completely for normal users Menu Item Description File The file menu only contains a Close button which will terminate the session to this Scali system Run The Run menu contains the different application launcher interfaces for the system including queue system submission View Opens the view menu from which different presentations for the nodes in the main window can be selected Status The Status menu contains functions from the Scali Monitoring system ScaMon including alarms Table 5 2 System MainWindow menu overview 68 Scali System Guide Version 3 0 5 3 Using Scali Universe Menu Item Description Management The Management menu contains hard system management functions like power console shutdown Interconnect Only available on systems with high speed inter
50. E 84 AOS eege eegene ir Ae ic ee ideen get eta 84 ACL VAN ie Mein eine E EE 88 GEFININg NN 87 B le ead nil pee el AG eon ea Ce a A 148 C losal e ECNE EE 80 CCS resource management sOtware nennen 187 channel DONANO aise ario aa a a Vido a Aaa Eaa A N a enti 22 COMPACE VIEW nee EE 81 es Aler i pa BC EE ER configuration files ER 156 S CAC OCDE deer A thoes EE de 107 110 SCADESK EE 63 console switching CODING VE 32 ee le UL Cl RE 49 OS preparatl Oni eege Eege EE Eeer 37 ee A EREECHEN 157 D EA 146 REEL 24 Scali System Guide Version 3 0 193 EES EE 24 26 daemons e RTE 155 SCA COMPA A 111 SCACOMES WENN 110 SEENEN 88 e ebe Ehe Eed ocnentditg at E 90 E environment variable SCAEM LICENSE FILE in ie i er 160 Ethernet SCLUP EEN 21 F Frontend Ter Mina nia da ad ENEE EN 74 H Hardware installation pDroblemme AAA 139 IntericonnectM Usan aia iria E 97 Ueleg leie A Ae 143 L LED A A A ada 141 Bleu 140 license le A e BEE 157 TiC geg gd Een e erer See deet AE eet Eege Set et t den ge 159 PEF IMAN CNG ee ice sieeve Bad hip a a eadto rate et 158 POQUCSEING DEE 159 INKE ge CR 144 M SA een ar ehe inserieren ragen ehe 150 Management Menus wicca geed nr edd 74 Management Window s irian tn piata a a nr 63 Master Terminal Window ERAN 74 monitoring e 77 Variables default 2 aia ds ria adi 89 194 Scali System Guide Version 3 0 o A aaa E Aaa EOE EENE 141 N VE 38 NOES Sel gt VTT 67 101 Node Terminal dei Aide 73 O
51. EE 30 Example cabling of a 3x3x2 3D torus please note that SCI links have been drawn mirrored for Clarity seenernnennnennnnennnnnnnnnnnnnnnnnnnnnnnnnn nennen 31 Scali Universe overvieW nnneenssenssnnnnannnnsnnnnnnannnnsnnnnnnnnsnnnnnnennnnnnnsnnnnennannnnn anna 61 Initial unconfigured Scali System Management window sesser 63 An example Sali Universe session with multiple monitoring views 64 Configuration window showing system defaults AA 65 Example System configuring A 65 Scali System Management window with five Scali Systems eese 66 LOGIN dialog e boX una teccagcceesdeeterieedie castes ic ae ana denies ah 66 Scali Universe system main window with four selected nodes A 67 Main Window views Detail View left and Small I cons right 69 Scali MPI Monitor WINdOW TEE 71 Scali MPI monitor window with output from an MPI program 72 Example of a Parallel shell command AAA 73 The Master window for terminal broadcasting nen 74 The Management menu eusseennsnnnnnennnnnnnnnnnennnnnnnnnnnnnnnsnnnnnnnannnsnnnnennnannnnnnnn 74 Software installation window unnneerseenssennnnersnnnnnnnnnnnnnnnnnnennnnnnnn nn nn nenn nn 76 ScaM on components in Contest 77 TAG SEAEUS MU incitar erario 78 Scali System Guide Version 3 0 191 8 1 8 2 8 3 8 4 8 6 10 1 192 3D View camera Control S iaa ae a a a 80 Compact view examples with 4and 132 noden 81 2D Bar small icons view of system
52. ICH Monitoring System performance and health ScaM on Managing batch queue systems ScaOPBS Scali System Guide Version 3 0 15 Chapter 2 Scali system overview 2 1 1 Hardware system architecture A Scali system is usually set up with a dedicated frontend sometimes called master node The frontend is used for compiling monitoring application launching and other management tasks If the system is set up with a private network the fortntend acts as a gateway between the private network and the local LAN figure 2 1 For small clusters 4 to 8 nodes the frontend node will often double as processing node LAN Internet server daemon m Frontend 0 M Remote workstation Master node Universe client i P ESN 0707 Local Workstation Universe client Scali Cluster Compute Nodes node daemons Figure 2 1 System architecture of a 4x4 Scali system 2 1 2 Software system architecture Scali cluster software systems often follows the same architecture as the hardware with the frontend running a master process daemon or main process while the compute nodes all running the similar node processes The main process is the server for Scali Universe clients Examples aretheScali monitoring daemon scamond or the interconnect configuration server daemon scaconfsd The node processes daemons and are typically lighter
53. N NENNEN NEEN 21 3 3 2 Using a privatenework AA 22 3 3 3 Using channel bondtng AAA 22 3 4 Dolphin SCI interconnect installation arar 23 3 4 1 SCI hardware Ooerview rro nan rre 23 3 4 2 SCI adapter installation A 24 3 4 2 1 64bit 66 MHz PCI bus operation sssrinin 25 3 4 3 SCI ee Lu EE 26 3 4 321 General rula ees gd isn 26 Scali System Guide Version 3 0 3 3 4 3 2 Single SCI ring cabling oe eee cece este tere eneee ete nennen nenn nennen 27 3 4 3 3 Two dimensional SCI torus cabling uunsrsserneennneenenenn nenn 28 3 4 3 4 Three dimensional SCI torus cabiltng nern 30 3 5 EE ee ll Le DEE 32 3 6 Power SWIECHING WEE 33 Chapter 4 Software installation uusunnneonnnunnnuunnunnnnnunnnunannunnnnnunnnnannnnnnnannnnnnnnnnnnnn 35 4 1 Software installation OVErViCW ernennen 35 4 2 Operating system Preparation nennen nnnennnn nn 35 4 2 1 M E nable rsh root access from to all boete EEN 36 4 2 2 O Enable rsh access from to all hosts for common userS nnnsseseeenen 36 4 2 3 O Enable rlogin and login as root from Tront end 36 4 2 4 M Ensure the at service is WOKING nennen nn 36 4 2 5 O Set up console over the serial port 37 4 2 5 1 Configuring serial console on Linus 37 4 2 5 2 Configuring serial console on Garten 38 4 2 6 M Use a common file system for MPI appltcattons AE 38 4 2 7 QO ee NIE ees Eeer EE REES 38 4 2 7 1 Red Hat Linux NIS SOlUP cconnonncccocnnnnninnnncnnnnnannnncn cnn cnn arra
54. NA d A EE 20 D Hack age response vo civic ern ss wien ei elle 185 Pallas benchmark EE 148 Parallel Srl dao 73 power switching COMTI QU MAU WEE 49 private worms tailandia happens 22 R REFERENCES E 187 ER OUT O itera na tate 144 Reset COMUNES cr BE Ee e Rehn 98 RING de Ce Lo e VE 107 e UE EE 36 IR let DEET 107 G3 EE 109 CIMENSI OM CN 108 MaA E 108 partiti E 109 ENEE 36 A Die HUE 70 run PerMUtated DIGIC ECE EE 148 E DE eebe eaten tease EE EEN 70 S a BEE 111 SE AE 110 140 146 setting ZvrC ge dl EE 106 SCACONICOO EE 140 144 SCACONFT ool COMMMANAS EE 171 Cla EE 133 Scali System Guide Version 3 0 195 A A dee 135 Sali software pl attor Mii oia EE EE ech 131 Scali system IT 11 O EAEE T NE A A RN 15 Scali Universe getting Started aneio e EE 62 EE EA E T NN 61 AA A E AT E A E EA 63 USING EE 67 SCAMONG DEE 88 SCAM Plis eier ert seen eege ees Eed ee Ee 141 SAMPLE EE 141 SCOPKG vir td caida 140 OAAS ER EHRE ELSE TOTESRLEIENSE SIETEOLLTEBELTTOERTLLTERFELPETTETRFRLOETTOESTHIERSLTEFFNILTRREETSRSEGERTTHERTEENACHTELER 134 SOP CP EE 133 SCOR UP EE 135 SEaasSCl error MESSAGES TEEN 141 SCASCI beg IEE e d glle EE 143 SCAS CAMA WEE 140 SO MEET 131 SE 131 SCASAMPA EEN 90 SCASUD EE 122 SCI Macao lod bai 23 hardware overview anne ne na nn 23 reloading driver nn neeee EE 105 SEEDING Ee 1 D WT ED 104 standard Cables een a ee Tee 23 NT WEE 26 SE WEE EE 28 SD Ree 20 NN 27 SCT AWd ooo ota 145 147 148 Genk elle DE 144
55. Nodenames separated by space characters f lt file gt Use node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test remote shell access 9 4 scarcp remote file copy The program scarcp copies file s between local machine and nodes in a Scali cluster One may specify the command to be used when copying files with the c option Usage scarcp options lt from gt lt to gt Scali System Guide Version 3 0 133 Chapter 9 ScaSH parallel shell tools options g R V V p d b W r key S lt cmd gt F lt n gt Print this text Print this text Copy from remote nodes to local node default is local to remote usein combination with r option to avoid overwriting file Note that use of the g option will force fanout to be O zero Copy recursively Print machine name and command before each output block Print version Print machine name before each line in each output block Dryrun only print command do not execute Run in background Wait until command has finished on all hosts Replace key with machinename if key is found in command Which remote execution copy command to use Fanout value Default is 8 Use 0 or 1 to force no fanout n n1 n2 Nodenames separated by space characters f lt file gt Use node names from file containing nodenames separated by
56. On a machine with 8 nodes this will cause the tool to wait 4 seconds between restart of a daemon on a node and an attempt to reconnect the node 7 3 5 Setting SCI nodelD manually An SCI nodal D is the identification for a nodein an SCI network It is stored in NVRAM in the SCI hardware card The configuration server scaconfsd will set check and correct SCI nodel D s on all nodes in the system unless this functionality has 104 Scali System Guide Version 3 0 7 3 Command line interface scaconftool been explicitly turned off by setting the AUTO_NODE_ID server option toOF F When automatic nodel D checking and setting has been turned off the nodeid command in scaconftool allows you to set SCI nodel Ds manually Do not use this command if you do not know what you are doing The nodeid command is only available in admin mode When run without parameters the nodeid command will simply activate the check and correct cycle of the scaconfsd This sets nodel Ds sequentially starting at 0x100 and increasing in steps of 0x100 following the order in which nodes are listed in the configuration file opt scali etc ScaConf conf Toset the nodel D toa specific value you must specify both node name and the new nodelD like this ConfTool gt nodeid scali2 24 0x2400 Make sure this new nodel D is legal The nodel D must be unique in the range 0x100 to Oxff00 and two least significant bytes in the nodel D must always be 00 7 3 6 Reloading SCI dri
57. PI release notes Scali AS http www scali com 6 The Scali parallel tools environment Draft 1999 Scali AS http www scali com 7 CCS Computing Center Software resource management for networked high perform ance computers Paderborn Center of Parallel Computing http www uni paderborn de pc2 8 VAMPIRtrace for Solarisx86 ScaMPI Installation and User s Guide Pallas GmbH Release 1 0 for VAMPIRtrace version 1 5 1998 http www pallas de 9 Review of Performance Analysis Tools for MPI Parallel Programs http www cs utk edu browne perftools review 10 High Performance Debugger Forum http www ptools org hpdf 11 NHSE National HPCC Software Exchange Parallel Tools Library http www nhse org ptlib 12 TFCC IEEE CS Task Force on Cluster Computing http www dgs monash edu au rajkumar tfcc 13 The Extreme Linux Organisation http www extremelinux org Scali System Guide Version 3 0 187 Appendix F Related Documentation 14 IEEE Standard for Scalable Coherent Interface SCI EEE Std 1596 1992 Additional information on SCI is available from http www scizzl com 15 UCD SNMP Home page http ucd snmp ucdavis edu 188 Scali System Guide Version 3 0 List of tables Acronyms and abbrevattons NN 12 E an TEE 13 Typographic Copventions A 13 Topology vs SCI hardware compatibility erseenseennneesnnnnnnennnnnnnnnnnnnennnnn nenn 26 Co
58. SCI hardware status programs C 1 10 scidle The scidle utility returns status code 0 if SCI links are ok and idle Usage opt scali sbin scidle a lt adapter no gt Options an Use adapter number n default 0 C 1 11 scicards The scicards script prints the number of SCI adapter cards installed and recognised by a node Usage opt scali sbin scicards Scali System Guide Version 3 0 169 Appendix C SCI Utility Programs 170 Scali System Guide Version 3 0 Appendix D ScaConf Reference D 1 Installation and package dependencies Installation of the Scali interconnect configuration system is normally handled by the SSP install program but the table below gives you a list of what to look for in case of problems Note that the ScaSH package must be installed on the frontend Name Description ScaComd Communication daemon node ScaConfNd Configuration daemon node ScaConfSd Configuration server frontend ScaConfC Scaconftool ASCII based configuration tool frontend ScaSH Parallel remote shell interface to nodes frontend Table 10 8 ScaConf packages D 2 scaconftool command reference This section describes all available scaconftool commands and their syntax The lt list gt parameter is essential lt list gt is to be replaced with a set of nodes For commands where a list of nodes are required it is at least five different ways to state these nodes For further information about
59. Scali System Guide Copyright 1999 2002 Scali AS All rights reserved Table of contents Chapter 1 Introduction usunsonnnnonnunnnnunnnnnonnunnnnunnnnnnnnunnnnannnnannnnnnnannnnnnnnannnnannn ann 11 1 1 Purpose of the Scali System GUI AEN 11 1 2 Overview of the Scali System Guide 11 1 3 HOw tir Gad this QUe cnica ns dienen 12 1 4 Acronyms and abbreviations AEN 12 1 5 T EFMS ANG CONVENCION Ss acciona ca 13 1 6 Typographic CONVE miii tee ru BER Ren 13 Chapter 2 Scali system OVerVie Weno 15 2 1 Description of a Scali system unnssnnssennnennnnnnnnnnnannnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnannn nn 15 2 1 1 Hardware system architecture EE 16 2 1 2 Software system architecture A 16 KE examp EE 17 2 2 1 Rackmount Systems ANA 17 2 2 2 Scali system cabinet uusesseensnsnnnnnnnennannnnsnnnnnnnnnnnnsnnnennnnnnnnnnannnnsnnnnn 18 Chapter 3 Hardware installation usnunennnnonnunnnnunnunnnnunnnnnunnnnnnnnnnnnunnnnnannnnannnn ann 19 3 1 Hardware oroantzattom NEE 19 3 1 1 Node naming scheme u usnnerssnnnnsnnnnnnnnnnnnnnnnennnnnnnnnennnnnnnnnnnannnnnnnnn 19 ES ZU ADSM a Le ET 20 ERC ee E tr Ri re 20 3 2 Node BIOS preparation iiia Aaa aa ET aag 20 3 2 7 Set MP lV ME 20 3 2 2 Interrupt level Mapping nerennnnnneensnnnnnnnnnennnnnnnnnnen ernennen 20 3 2 3 Power on startUPp WEE 20 32 4 Enable aan 21 3 2 5 BIOS redirection to serial port 21 ER E DI EE 21 33 1 Vsingthesitel AN ek NEE
60. Scali System Guide Version 3 0 75 Chapter 5 Scali Universe 5 3 6 1 Software Installation Window fare installation Figure 5 15 Software installation window The software installations window provides the folloving options The Select packages scrolled list at the left edge of the window lists possible packages to install on the System You may select all packages with the Select All button Alternatively packages may be toggeled between selected and unselected state by lt Ctrl gt Heft dick hold the Ctrl key down while clickling with the left mouse button Similarly the Select nodes lit to the right specifies which nodes to participate in the software installation When software packages and nodes have been selected you may start int all at ion with the Install button 76 Scali System Guide Version 3 0 Chapter 6 System Monitoring Aimed at all users of Scali Systems this chapter explains how to use and configure the Scali cluster monitoring system ScaM on Except for configuration ScaMon is almost exclusively used through the graphical environment in Scali Universe 6 1 Overview ScaMon provides flexible and powerful monitoring of clustered computer systems including user definable alarms on selected monitoring events Based on the industry standard SNMP a number of system and interconnect parameters may be monitored Thegraphical monitoring plug in for Scali Universe provides a number of presentation options ranging
61. Small Icons view of the presentation Small icons allows for compact monitoring but has less detailed information Log to file Enables you to select a file to which the monitoring data is logged The logging interval is the same as the update interval specified in the monitoring options Close Close the window Table 6 2 Monitoring view window options menu 6 3 1 2 3D view camera controls All the 3D presentation view windows have scroll bars right and bottom to control the camera angle for the presentation The right scrollbar controls the elevation up down while the bottom scrollbar controls rotation left right In addition we have the mouse key bindings as explained below zoom is only available as a mouse binding Shift scrollbars Left button Left button sg up up zoom in left right down down zoom out left d gt right Figure 6 3 3D view camera controls 80 Scali System Guide Version 3 0 6 3 Graphical monitoring 6 3 2 Compact View Sg Stall waivers Upsand CFS Unnr time ES Figure 6 4 Compact view examples with 4and 132 nodes Due to its flexible resizing behaviour the compact view is great both for monitoring large systems and for making icon sized monitors for your desktop The compact view provides dynamic scaling and labelling which means labels are removed when the window is resized below a certain limit Still by moving the mouse pointer over one of the val
62. This file is formatted for on line reading and can be viewed with any text file viewer like more 1 Or less 1 7 3 3 Status check of nodes and daemons Normally the first thing you want to do is to check the status of all the nodes in your system To help scaconftool provides the list and status commands Here is an example using list with the options nodeid status all which will show SCI node identification and status for all nodes ConfTool gt list nodeid status all Name Nodeld Status scali2 24 0x2400 OK scali2 23 0x2300 OK scali2 22 0x2200 OK scali2 21 0x2100 OK scali2 14 0x1400 OK scali2 13 0x1300 OK scali2 12 0x1200 OK scali2 11 0x1100 UNREACHABLE Have a look at the Status column if some of the nodes are not OK state you should investigate this further Table 7 3 contains a list of all possible node states with descriptions and some hints in how toresolvea possible problem Try using these hints to get the node back toan OK state It may be that some daemons are not running and needs restarting on some of the nodes The fix command described in the next section attempts to do this automatically 102 Scali System Guide Version 3 0 7 3 Command line interface scaconftool Node state Description How to resolve OK SCl driver is loaded and This is the state of a healthy node communication with dae and need not be resolved mons on this node is ok NO_SCI_DEV TheSCI driver onthenode You can relo
63. a 3x3x2 3D torus please note that SCI links have been drawn mirrored for clarity Scali System Guide Version 3 0 31 Chapter 3 Hardware installation 3 5 Console switching The advantages of centralized and remote access to node consoles are obvious when performing tasks like boot monitoring BIOS configuration OS installation and so on The SSP products UniverseXE and ClusterE dge both supports remote console switching and console broadcasting provided that you e Usea qualified remote serial switch see SSP release notes Enable console redirection to the serial port in the nodes Cabling of a remote serial switch is quite simple Cables must be connected from the ports of the serial switch to the serial port on each node possibly by the aid of a RJ 45 to serial port adapter If you connect the nodes sequentially you should also get sensible default values when configuring the console server during SSP installation later on What we mean by sequential connection is best illustrated by an example Say that you have 3 x 8 port serial switches and 20 nodes named node 1 through node 20 If nodes are defined in this sequence during SSP installation this is the order in which the console server expects them to be connected to the switches Y ou should therefore e Connect nodes 1 through 8 to switch 1 ports 1 through 8 e Connect nodes 9 through 16 to switch 2 ports 1 through 8 e Connect nodes 17 through 20 to switch 3 ports 1
64. aSNMPd Removing old versions and copying files to repository ScaSNMPt Removing old versions and copying files to repository ScaSSP Removing old versions and copying files to repository ScaSensor Removing old versions and copying files to repository ScaShMem Removing old versions and copying files to repository Copying config to repository _ Preparing ScaSH config Modifying ScaPkg config Some pre installation tests will be run in the last part of the installation configuration part Checking hostnames 7772777777772 OK Checking some package versions OK Se Checking for glibc memory leak OK Checking required applications OK Testing AT daemon e OK Checking bigphysarea patch _ OK Checking FPU control word bug OK Checking kernel versions OK Checking HW support OK Checking OS support o OK Checking SCI boards OK Checking MI service 7772772727272 OK Next is an example showing how to change the default node category suggestion The Determine node categories sub section is part of the Installtion configuration section The example starts with the de
65. ach node or with the daemon command in scaconftool Scali System Guide Version 3 0 111 Chapter 7 Interconnect configuration 112 Scali System Guide Version 3 0 Chapter 8 Batch system ScaOPBS The Scali binary distribution of Portable Batch System ScaOPBS allows jobs to be submitted to a batch queue directly from the Scali Universe graphical cluster management interface ScaOPBS also provides an interface for ScaMPI jobs to be submittet to the batchqueue system 8 1 Overview The Portable Batch System PBS is a batch job and computer system resource management package It will accept batch jobs a shell script and control attributes preserve and protect the job until it is run run the job and deliver the output back to the submitter Portable Batch System PBS was developed at NASA Ames Research Center and later by Veridian System where it was split into a commercial version called PBS Pro and an open source version called OpenPBS License and support for PBS Pro may be obtained from Veridian at http www pbspro com The full source and other resources for OpenPBS is found at http www openpbs org Additional information and patches for OpenPBS are available from OpenPBS Public Home at http www unix mcs anl gov openpbs ScaOPBS is the Scali binary packaging of Open PBS It includes e scasub Script for easy submission of ScaM PI mpich applications e automatic node server scheduler configuration with Scali Unive
66. ad the driver with the is not loaded reload command or usethefix com mand If this did not help the state must be resolved outside scaconftool Please check that the SCI card is properly fitted in the node Also make sure that your hardware and or kernel version is supported by the ScaSCI driver NO_DAEMON The configuration server Make sure that the has lost contact with the AUTO_RECONNECT and node daemon scaconfnd AUTO_REPAIR options are ON on the node This usually Then you may try the fix command indicates that the daemon to resolve this state is not running or not con nected to the server UNREACHABLE The configuration server Make sure that the has lost or never gained AUTO_RECONNECT and contact with this node AUTO_REPAIR options are ON If This may indicate that the it isa communication failure this communication daemon will be resolved with the fix lt node gt scacomd on this node is command If the node is still not running or that the UNREACHABLE after running the node itself is erroneous fix command the node most likely has problems beyond reach of scaconftool UNKNOWN The configuration server Make sure that the has not been able to deter mine the state of this node AUTO_RECONNECT and AUTO_REPAIR options are ON then you may try the restart com mand to resolve this state Table 7 3 ScaConf node states Scali System Guide Version 3 0 103 Chapter 7 Interconnect configuration 7 3 4 The fix
67. adline style wherethe arrow key gives access toa history of previous commands 7 3 2 1 User modes Depending on the privileges of the user scaconftool will start in either user mode or administrator admin mode The admin mode is available only to users with administrator rights i e user id 0 User identity is checked upon start and the mode for the configuration tool is set according to this information Admin mode gives full access to ScaConf while user mode only gives access to status information e User mode For ordinary users there s only a restricted set of commands available getopt list log reconnect select status unselect update version help and quit Admin mode Admin mode is only available to root and gives full access to all ScaConf operations like rerouting starting stopping daemons etc 7 3 2 2 Command truncation The commands and some of the keywords are also recognized if you truncate them as long as enough of the command is present to be unambiguous Node states node names and name of routing algorithms may not be truncated lis no st ais equal to typing list nodeid status all 100 Scali System Guide Version 3 0 7 3 Command line interface scaconftool 7 3 2 3 Node selection Some of the scaconftool commands allows you to operate on a set of nodes There are five different ways to select nodes The list command will be used as an example to illustrate these possibilities 1 Explicit noden
68. again after the first time a one shot You may receive a lot of alarms if this valueis set to low Window item Description Match all of the following Use logical AND between the monitoring variable comparisons below Excludes Match any Match any of the following Use logical OR between the monitoring variable comparisons below Excludes Match all Property Option menu to select a monitoring variable for comparison Operator Option menu to select operator for comparison lt 5 gt gt or C syntax Threshold Threshold value to compare against Uses default unit of selected monitoring variable e g percent for CPU load rpm for fan speed More Adds one more monitor variable comparison Fewer Removes one monitor variable comparison Description Name of alarm Email address Where to send the alarm notification Action What to do when the alarm is triggered Either None Reboot Machine or Shutdown Machine Seconds before re trigger Defined the minimum interval in seconds after an alarm has been triggered till it can be triggered again The default value is 1 which means no re trigger Table 6 4 Alarm Editor control overview Scali System Guide Version 3 0 6 4 Using Alarms 6 4 3 Alarm log viewer The View button in the alarm main window will open the log viewer window for the selected alarm This will show a dynamically updated list of events that h
69. al ports connected to a terminal server you must install the Scali console server to be able to use them as consoles Q Do you want to install the console server n y Selecting power switch es If you have power switches in the system you must install the Scali power switch server to be able to switch power on and off Q Do you want to install the power switch server n y If installation of OpenPBS Parallell Batch System is chosen a valid non root user has to be specified in order to enable some tests of this software after installation However the tests may be cancelled by not specifying a user Selecting OpenPBS oe S You may install the free queue system OpenPBS www openpbs org The ScaOPBS package is a binary distribution of OpenPBS with easy installation and configuration of OpenPBS and job submittion script for ScaMPI and MPICH applications Q Do you want to install OpenPBS queue system n y Testing of OpenPBS requires a non root username the username must be defined at all nodes Use empty username default to skip the OpenPBS test Q Please give username testuser 48 Scali System Guide Version 3 0 4 3 Scali software installation The following OpenPBS selection has been made OpenPBS will be installed on the system Test of OpenPBS will be performed as user testuser Q Do you accept the above configuration y The following section shows
70. ali user definable alarms are part of the monitoring system avail ablein professional versions of the software Y ou may define your own events to trigger an alarm based on any combination of comparative operations of any available monitoring variable When the alarm is triggered you can select between a set of predefined actions to be performed Combined with the possibility to define your own monitoring variables this makes for an extremely flexible and powerful solution 6 4 1 The Alarm window Scali Unive delfi Alarms Ethernet bug inactive SCI link1 monitor inactive Edit Delete Activate Deactivate View se Figure 6 10 The Alarm window You open the alarm main window by selecting Alarms from the Status menu The Alarm window contains a scrolled list over current alarms with status information and a row of buttons to perform operations on selected alarms along the right edge The function of the different buttons is pretty self explanatory but a summary can be found in table 6 3 Apart from Add all other buttons requires that an alarm is selected 84 Scali System Guide Version 3 0 6 4 Using Alarms first in order to work properly When an alarm is triggered it will change to red background colour in the alarm list This is the case for alarm CPU load too high in figure 6 10 Button Description Add Brings up the alarm editor window to create a new alarm Edit Brings up the alar
71. ames You may explicitly specify the node names on the command line separated by one or more blanks Example list nodel node3 node14 Will list information about nodel node3 and nodel4 2 All nodes The keyword aii is an alias for all nodes in the system The command list al1 will list to all nodes known to the configuration server 3 By nodestate It is possible to select a set of all nodes that is in one particular state Usethe list status all command to see which state the nodes are in list NO_SCI_DEV will list information on all nodes with status NO_SCI_DEV 4 By routing partition number You may select all nodes part of a particular routing partition list status 1 will show status for all nodes in partition 1 Partition number is prefixed with 5 Internal list scaconftool maintains an internal list of nodes which is used no other node selection mechanisms are provided The list is enpty when the tool is started Nodes may be added to the list with the select command and removed with the unselect command ref D 2 7 ConfT ool gt select As an example you may use this to get a list of all nodes except thosein state OK First type select all followed by unselect ox The list command without parameters will now list all nodes except those in state OK 7 3 2 4 Disabling server log messages The default setting for scaconftool is that log messages from server activity are shown directly in scaconftool with a ti
72. and Veridian Information Solutions Inc Visit www OpenPBS org for OpenPBS software support products and information 7 DISCLAIMER OF WARRANTY THIS SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND ANY EXPRESS OR IMPLIED WARRANTIES INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FORA PARTICULAR PURPOSE AND NON INFRINGEMENT ARE EXPRESSLY DISCLAIMED IN NO EVENT SHALL VERIDIAN CORPORATION ITS AFFILIATED COMPANIES OR THE U S GOVERNMENT OR ANY OF ITS AGENCIES BE LIABLE FOR ANY DIRECT OR INDIRECT INCIDENTAL SPECIAL EXEMPLARY OR CONSEQUENTIAL DAMAGES INCLUDING BUT NOT LIMITED TO PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES LOSS OF USE DATA OR PROFITS OR BUSINESS INTERRUPTION HOWEVER CAUSED AND IN ANY THEORY OF LIABILITY WHETHER IN CONTRACT STRICT LIABILITY OR TORT INCLUDING NEGLIGENCE OR OTHERWISE ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE This license will be governed by the laws of the Commonwealth of Virginia without reference to its choice of law rules 130 Scali System Guide Version 3 0 Chapter 9 ScaSH parallel shell tools The ScaSH parallel shell tool suite is a collection of scripts enabling the execution of commands in parallel on a large number of target nodes e scash Execute command on nodes in a Scali cluster e scahosts Checks nodes for availability e scacp Copies file s locally on node
73. arch x y z package from section A 2 Note that for Scali packages it is only the module part of the package file name that is used to specify the package It is important that no two package files with same module prefix is present in the repository when scapkg is used scapkg p ScaMPI r home software node 1 node 2 node 3 If no nodes are specified on the command line scapkg will use nodenames from the nodelist in the configuration file scaPkg conf as default For further information about the configuration file see section E 2 Usage scapkg dvVDsfeh p lt package gt r lt path to repository gt lt host gt Options D Specify packages to install The default is to use the packages found in the default repository c Specify which categories nodes belong to This switch can only be used in combination with explicit nodes given as arguments when p is not used and will only influence nodes not found in the configuration file r Specify the path to the repository See configuration file opt scali etc ScaPkg conf for default path to repository Scali System Guide Version 3 0 181 Appendix E ScaPkg Scali Software installation program s Enable sync mode scapkg will then try to install and remove packages in order to reflect the setup in the config file opt scali etc ScaPkg conf The need to remove packages can occur if one has manually installed packages f E nable force mode This wi
74. arguments to route on a single ring from scaconftool 7 4 2 2D Torus topology Two dimensional torus topology is detected by the linetopology TOP_TORUS_2D in configuration file opt scali etc ScaConf conf n a 2D torus each nodeis connected to two SCI ringlets one in each dimension of the torus Available routing types are Dimensional routing xy yx raw xy and raw yx e Maxcy Maximum connectivitY routing e C3 Close Commutative Connection routing Scali System Guide Version 3 0 107 Chapter 7 Interconnect configuration In short the differenceis that if one or morerings are faulty the maxcy and C3 routing algorithms will be able to connect all nodes with at least one working ring while dimensional routing algorithms will not be able to connect nodes on the faulty rings 7 4 2 1 Dimensional routing in 2D torus In dimensional routing all routing in one dimension are completed before packets are routed in the next dimension Dimensional routing for 2D torus topologies is selected with the reroute xy Of reroute yx Commands xy means that packets are routed in the x dimension first x dimension is equal to the horizontal dimension drawn for the list configuration command Thisis alsoequal tothe dimension connected through link 1 on the SCI adapter yx means that packets are travelling in the vertical dimension first Vertical refer tothe printout from the list configuration command in scaconftool The dimensional routing will not
75. arting scaconftool When installed scaconftool is located in the sbin directory of the Scali installation tree and can be started on the frontend by frontend opt scali sbin scaconftool To start the tool on any other node than the frontend the hostname of the frontend must be included with the H switch bash opt scali sbin scaconftool H lt frontend gt If scaconftool fails to start or exits abruptly during run chances are that it has lost the connection to scaconfsd In that case you need to check that the server is running and if not restart the server and start a new tool Scali System Guide Version 3 0 99 Chapter 7 Interconnect configuration 7 3 2 Using scaconftool The following command line options are available with scaconftool Usage opt scali sbin scaconftool Options Options 1 die enable e or disable d log from server e cmdl1 cemdn execute these commands in batch mode h help H lt host gt set the server host p lt port gt set the server port default is 32015 q make scaconftool as quiet as possible s lt number gt set thesleep constant usually anumber between 0 and 1 default is 0 7 v print version info When the tool has started it will prompt you with ConfToo1 gt to indicate that it is ready to accept commands A reference to commands and their syntax can be found in the file opt scali doc ScaConfC scaconftool txt Command line editing may be performed in re
76. as triggered the alarm listing exact time of day and actual parameter values You may clear the window using the Reset button no undo Fri Jun 14 00304334 CEST 2002 39 gt 50 Fri Jun 14 00304346 CEST 2002 100 gt 50 Fri Jun 14 00304346 CEST 2002 100 gt 50 Fri Jun 14 00304346 CEST 2002 Fri Jun 14 00304346 CEST 2002 Fri Jun 14 00304358 CEST 2002 Fri Jun 14 00304358 CEST 2002 Fri Jun 14 00 04 58 CEST 2002 Fri Jun 14 00 04 58 CEST 2002 Figure 6 12 The alarm log viewer 6 4 4 Example Defining a new alarm As a simple example we will define the alarm called CPUload too high which will send us an E mail if user CPU load passes 50 To define a new alarm press the Add button in the alarm main window The alarm editor appears and we can start defining the values fig 6 13 Editing CPUload to high Match all of the following Match any of the following Property Operator Threshold cpuuser gt Bi More Fewer Description Email address Action Seconds before re trigger CPuload to high jjoe company com None AT save Cancel Figure 6 13 Adding a new alarm Scali System Guide Version 3 0 87 Chapter 6 System Monitoring First select cpuuser from the Property menu then set the Operator to gt and the Threshold to 50 Typethename of the alarm in the Description field and your E mail address in the field next to it Set Action to none and Seconds to re trigg
77. atically by scaconfsd unless automatic rerouting is explicitly turned off The machine may also be routed manually using the Interconnect gt Set routing menu or via scaconftool with the reroute command As a result of the routing all unused or broken links will be disabled Remember that to check the status of the SCI links and nodes you can either use the Interconnect gt Link Status in Scali Universe or the list status link all command from scaconftool There may arise situations where one or more nodes or one or more links are erroneous Nodes are considered erroneous by ScaConf if they arein a different state than OK SCI links are considered erroneous if they are reported as DOWN Routing is computed according to the state of nodes and SCI ringlets An SCI ringlet is considered faulty if one of the links in the ring has failed This means that controllers reporting their links as DOWN will cause the entire ring to be considered broken You may check the status on links with the status link command in scaconftool A ring is also considered erroneous if one or more of the nodes part of this ring is erroneous 7 4 1 Ring topology If the machine is configured with all nodes on one SCI ringlet the routing algorithm is straightforward All packets to all destinations are sent and received on the ringlet Ring topology is detected by the line topology TOP_RING in configuration file opt scali etc ScaConf conf Use the reroute command with no
78. be able to connect any of the nodes on erroneous rings and will only be able to connect nodes where both horizontal and vertical ring are working A raw version of the dimensional routing is also available Use the reroute lt xy yx gt raw command from scaconftool With this algorithm one may force routing to be set as if all nodes and rings are ok This does not make all rings and nodes ok but it will set the routing tables on each node as if they where If raw xy routing is set on a system where nodes or rings has failed one will not get full connectivity The raw xy routing is mostly useful for debugging purposes 7 4 2 2 Maxcy routing algorithm for 2D torus The maxcy al gorithm MAXimum Connectivity is a fault tolerant routing algorithm capable of connecting the maximum number of nodes if one or more nodes or ringlet link has failed When all nodes are working the maxcy algorithm is equal to dimensional routing Maxcy routing is selected with the reroute maxcy Command Rings and nodes which are not OK as far as the configuration server is concerned will be considered failed when routing is computed and these nodes and rings will be routed around When a node fails this will make its two SCI ringlets useless for SCI communication This means that other nodes on these two failed rings will only have one operational SCI ringlet left When this happens the maxcy routing algorithm will still be able to connect all remaining n
79. be tested if some nodes are down You may state which nodes that are not to be tested with the x lt nodes gt option To ensure that communication between nodes on a particular ring stays on that ring on 2D systems dimensional routing will be used One run with xy and one testrun with yx routing This in turn means that all SCI controllers and links should be up and enabled Y ou need to run sci_hwdiag as root sci_hwdiag will detect how many processors the nodes have and start the MPI test application with the correct parameters You may manually overrun this with the 1 or 2 flag torun with either 1 or 2 process es per node With the c flag you may specify your favourite MPI application to use when running the tests The flag should be followed by a string enclosed in which should contain what you usually put between the command opt scali bin mpimon and before specification of the nodes No specification of the test application is equal to running with c opt scali contrib P MB 2 build Linux2 i86pc gnu PMB MPI1 multi 0 Scali System Guide Version 3 0 147 Chapter 10 Troubleshooting With the c bidirect parameter permutated bidirect from opt scali examples bin will be used as test application This was default behaviour for sci_hwdiag prior to version 1 2 0 Rings longer than 4 nodes will be divided into shorter parts to limit the number of permutation in the run_permutated_bidirect pr
80. bles Scali System Guide Version 3 0 89 Chapter 6 System Monitoring Menu Item size maxval Swap out kB s human 1024 Ethernet 0 input octets human 1024 Ethernet 0 output octets human 1024 Table 6 5 Default Performance monitoring variables Menu Item size maxval Disk Free percent 100 Disk Used percent 100 Memory Real Free percent 100 Memory Real Free kB human 134217728 Memory Real Used percent 100 Memory Real Used kB human 134217728 Memory Swap Free percent 100 Memory Swap Free kB human 134217728 Memory Swap Used percent 100 Memory Swap Used kB human 134217728 Table 6 6 Default System monitoring variables 6 6 The SNMP daemon scasnmpd Also part of the monitoring system theScali SNMP daemon scasnmpd is based upon the UCD SNMP package 18 The daemon has been extended by Scali with monitoring and configuration variables OIDs for Dolphin PCI SCI adapters and various other host specific variables The Scali SNMP daemon is configured to not interfere with existing SNMP daemons by using a non standard SNMP port The default port used by scasnmpd is 32016 90 Scali System Guide Version 3 0 6 7 Defining new monitoring variables 6 6 1 Manual start stop Although scasnmpd will be installed and started by the SSP install script it may be started stopped or restarted with the command below should it be necessary opt scali init d sca
81. braries under lib e doc all documentation e eta configuration files for Scali software e examples example MPI applications and source e contrib 3rd party software adapted to Scali software e init d boot startup scripts Soft links are made from system startup catalogues rc d and init d under etc by install script e kernel kernel related files e man manual pages e plugins plugins for the Scali Universe GUI e share share directory used by SNMP clients agents A 2 Scali package file naming convention Every software unit is distributed as a single package file module os arch x y z package where module is the software name e g ScaMPI ScaSCI ScaPkg x y z istherelease number e g 1 0 2 os is the operating system e g SunOS5 or Linux2 arch is the architecture e g sparc u i86pc or alpha package is e g pkg rpm or exe depending on the operating system 152 Scali System Guide Version 3 0 A 3 Scali software packages overview A 3 Scali software packages overview The pkg directory on the SSP distribution contains all necessary software packages for the entire Scali product range Universe Universe XE and ClusterE dge Hence depending on your license and product some of the packages in the table below may not have been installed on your system Package module name Description ScaComd Communication daemon provides a commu
82. cali System Guide Version 3 0 61 Chapter 5 Scali Universe Figure 5 1 illustrates how the different pieces fit together The plugins are all part of the binary distribution rpm of the Scali Universe GUI and will therefore run on the same system as the GUI no matter where the managed system is located Plugins are loaded dynamically on demand a demand determined by the licensing and capabilities of the managed system When new plugins are loaded they will show up as additional menu items in the Scali Universe M ainWindow The general idea is that the Scali Universe GUI is a somewhat fat dient for the Universe management system Available features and licensing are properties of the managed systems but will be reflected in the GUI on a per system basis This means that the GUI itself is free from licensing restrictions something which enables you to install copies wherever it seems feasible Your home office your portable the system console and so on All communication between the Universe GUI and the managed systems even the terminal emulators runs through a single secure encrypted socket connection This is the only connection needed for both local and remote cluster systems providing full remote management capabilities 5 2 Getting started with Scali Universe 5 2 1 Prerequisites The only prerequisites for running Scali Universe is that the machine you are using has network access to the frontends of the Scali systems you wa
83. class that must be defined in the class section as we talked about in 6 7 2 1 The format is usually sa for integer and 1 for strings Note some OI Ds returns the data type with the value like Gauge32 42 or Counter32 42 seethesnmpwalk examplein 6 7 1 In this case the data type must be a part of the format likethis Gauge32 d Then finally enter the oia either numerically or symbolically if thereis a MIB for you variables in opt scali share snmp mibs Remember that you can always mix the two formats It helps readability to use the prefixes host for 1 3 6 1 2 1 25 system for 1 3 6 1 2 1 1 and enterprises for 1 3 6 1 4 1 To add an oia entry for one of the OIDs listed in the snmpwalk example using symbolic OI Ds add the following lines oid process_count integer scasnmp Gauge32 d host hrSystem hrSystemProcesses 0 oid user_count integer scasnmp Gauge32 d host hrSystem hrSystemNumUsers 0 Or alternately with numeric OI Ds oid process_count integer scasnmp Gauge32 d 1 3 6 1 2 1 25 1 6 0 oid process_count integer scasnmp Gauge32 d host 1 6 0 All three process_count definitions are equal Scali System Guide Version 3 0 93 Chapter 6 System Monitoring 6 7 2 4 Adding the variable definition for Universe In the final section you need to define the variable as it appears in Scali Universe The definition takes the form variable descr size maxvalue expression variable Pe
84. connects installed Contains funtions from the Scali interconnect management system ScaConf Software Contains functions from the Software installation and management system Table 5 2 System MainWindow menu overview 5 3 3 Different system views the View menu The view menu gives you three different alternatives for node presentations in the MainWindow Thedefault Large Icons view is shown in figure 5 8 Other available views are Small Icons which gives room for more nodes on big systems and Detail which is a compound view combining some textual monitoring into with the graphical node status Both can be seen figure 5 9 Steeg Vert ila Eleven ez Ele Dr yow States Management Inercamect Software l esseseeeeeeseseseseseseeeei Figure 5 9 Main Window views Detail View left and Small Icons right Scali System Guide Version 3 0 69 Chapter 5 Scali Universe 5 3 4 Running programs the Run menu The Run menu is used for execution of programs on multiple nodes in a Scali System This may be MPI programs shell commands or terminal emulators Table 5 3 below gives an overview of the Run menu items Note here that the Submit Job item is only available to non root users and if a queue system is installed Run Menu Item Description MPI Program Opens theScaM PI monitor window to control execution of ScaMPI programs on the system MPICH Program Opens the
85. constructed from a base name followed by a numeric part Ideally the numeric part should reflect the physical location in the cabinet i e node XY where X and Y reflects the X and Y position respectively figure 3 1 A naming scheme like this has a lot of benefits especially on large systems when operating the system from command line tools and when mapping between names and physical node locations 14 24 34 44 13 23 33 43 12 22 32 42 11 21 31 41 Y Figure 3 1 A 4x4 duster with XY coordinates physical view Scali System Guide Version 3 0 19 Chapter 3 Hardware installation 3 1 2 Labelling When node names have been decided you should add labels with the node name on the back side of the cabinet of all nodes This is were you will be standing when inserting cables later on so being able to tell which nodes you are working on makes things a lot easier 3 1 3 Checklist Some issues to remember when installing a cluster e The duster must be placed in a location with sufficient cooling Also be sure not to obstruct air ventilation channels on nodes and infrastructure equipment e Beextremely aware of cooling requirements for rack mount systems since nodes are packed so close together Fully enclosed 19 racks must not be used remove front back doors if necessary e The power outlets must be capable of supporting the cluster s accumulated power consumption e Provide easy access to all equipment in the entire duster fo
86. ctions ICMS OUT OE MEMORY This error type indicates that it was not possible to allocate the requested amount of pinned memory Add more memory to the system or change configuration options to allow a more aggressive SCI memory allocation strategy Table 10 3 Some SCI error types 142 Scali System Guide Version 3 0 10 5 2 SCI Troubleshooting 10 5 SCI Interconnect problems Problem Description Solution Interrupt conflict During ScaSCI install the SCI hardware has an interrupt conflict with other hardware cards Please consult the operating system install documentation about how to solve the problem Wrong jumper set tings NB This only applies to D31X type of cards After installation of the ScaSCl software package run opt scali sbin scinfo m to check that jumper settings are correct Used available ATT entries should typically be RMAP_IO of 1 256 and RMAP_GATHERING of 1 256 when ATT size is 512k default i e the SCI driver is able to map a maximum of 256 512k 128M remote memory Check the ScaSCl release notes if your values are different and correct the SCI board jumper settings or the driver con figuration values No adapters found Use the opt scali sbin scicards script to find the number of installed SCI adapters on the node If the answer does not match your hardware installation of SCI adapt ers or you get the error message Scilnitialize failed because no
87. d Dryrun only print command do not execute b Run in background W Wait until command has finished on all hosts n n1 n2 Nodenames separated by space characters f lt file gt Use node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test rsh access One or more of the options following must be specified i startpid stopidK ill those processes that lie within the given process id range u lt user gt Kill those processes that match the given username s lt txt gt Kill those processes that match the given string devel name gt Kill level or name e g 9 HUP devel gt Kill level only numeric value allowed Scali System Guide Version 3 0 135 Chapter 9 ScaSH parallel shell tools 9 7 scarup node status The scarup command prints up time load information about nodes in a cluster Usage scarup options options Print help text h Print help text V Print machine name and command before each output block V Print version p Print machine name before each line in each output block d Dryrun only print command do not execute b Run in background w Wait until command has finished on all hosts n n1 n2 Nodenames separated by space characters f lt file gt Use node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excl
88. d by the SSP install script it may be started stopped or restarted with the command below should it be necessary 88 Scali System Guide Version 3 0 6 5 The monitoring daemon scamond opt scali init d scasnmpd start stop restart info 6 5 2 Configuration file Thescamond configuration file defines a subset of monitoring variables that will be available for monitoring by other applications It is located in opt scali etc ScaMond conf Duetoits wide format this configuration file is not very well suited for printing so if you re interested we suggest that you view the file on line using less or more 6 5 3 Default monitoring variables The predefined monitoring variables which occur in the default Status menu represents only a handful though probably the most interesting system parameters to monitor As we shall see in 6 7 Defining new monitoring variables this is only a small subset of the available parameters and can be easily extended and customised The default monitoring parameters are arranged in two major subgroups Performance and System as described in tables 6 5 and 6 6 Menu Item size maxval CPU Idle Time percent 100 CPU System time percent 100 CPU User time percent 100 Context switches s human 1024 1 0 in blocks s human 1024 1 0 out blocks s human 1024 Interrupts s integer 200 Swap in kB s human 1024 Table 6 5 Default Performance monitoring varia
89. de 2 Node 2 Node 1 Table 3 2 Connection list example for cabling an 8 node SCI ring Scali System Guide Version 3 0 27 Chapter 3 Hardware installation 8 7 ch 5 3 2 gt er Figure 3 9 Example of 1x8 single ring cabling in a 1x8 cabinet 3 4 3 3 Two dimensional SCI torus cabling The 2D torus topology consists of rings in two dimensions hence we use the same interleaving technique as for a single ring but this time we have multiple rings in two dimensions which makes things slightly more complex F or 2D torus topologies we use the following additional guidelines Connect the first dimension on SCI link 0 LO e Connect the second dimension on SCI link 1 L1 Following this connection scheme enables you to use the default suggestions for connections when installing the SSP software later on From LO OUT ToLO IN Node 1 Node x3 Node 3 Node 4 Node 4 Node x2 Node 2 Node 1 Table 3 3 Example cabling of one vertical ring in a 4x4 system x 11 4 28 Scali System Guide Version 3 0 From L1 OUT ToL1 IN Node 1y Node y Node 3y Node 4y Node y Node y Node 2y Node Ly Table 3 4 Example cabling of one horizontal ring in a 4x4 system y 1 4
90. dn t help all nodes You may check this with the 3 list status all command in scaconftool see section D 2 9 OS does not recog Some PCI slots do not fully support Try another PCI slot for the SCI card If nizes the SCI card the specification Interference with changing PCI slot for the SCI cards does AGP or other hardware devices not help please refer to Dolphin may happen http www dolphinics com sciping reports not TheSCl utility Reroute the cluster with scaconftool or responding nodes opt scali bin sciping or the scip ScaDesktop Make sure sciping is run ing command in the ScaConf utility only between nodes with same routing scaconftool reports not respond partition See section 7 4 4 ing nodes ScaSCl log message The ScaSCl log contains The problem is the MEM_BLOCK_SIZE RMAP_OVERFLOW RMAP_OVERFLOW message of setting in the ScaSCl response file that lost pages is reflected to the mem_block_size vari Unix WARNING kernel map able setting in the ssci conf file on all rmap overflow lost 39250 39254 nodes at ScaSCl package installation Unix WARNING kernel map time This variable regulates the size of rmap overflow lost 39238 39242 each page aligned block of memory allo error messages repeated multiple cated from the OS eg granularity of times in the log allocation Table 10 4 ScaSCI troubleshooting a Red applies to old type of SCI cards i e D31x Yellow applies to new
91. dt dE 176 D 2 12 Conf Tool gt OG DTN 177 D 2 13 ConfTool gt nodeid EE 177 D 2 14 Conf Tool gt CL Ce BEE 177 D 2 I5 COnf TO Ee e TEE 177 D 2 16 ee E EE 178 DPA EALE uge ET 179 D 2 18 ConfT Gol gt Hee 179 D 2 19 C onfl Gol gt estar EE 179 D 2 20 Conf TOA gt SC PING DEE 179 D 2 21 Cont ool gt Nai ER en 179 D 2 22 Conf LO gt UE 180 D 2 23 Conf Gol gt Versi ON BEE 180 Appendix E ScaPkg Scali Software installation program 181 E USINO SAPKO EE EE 181 AO e ege EE 182 E 2 1 ScaPkg conf path to package files repooeitorvl 182 E 2 2 ScaPkg conf package liSt AEN 183 E 2 3 ScaPkg conf dependency Iert 183 E 2 4 ScaPkg conf Guper Cateoorles EE 184 E 2 5 ScaPkg conf Node liSt Ne 184 E 2 6 Package response files ooncoonnonncnnnninnocnnncnnnnnorrnnnnnn crac rre 185 Appendix F Related Documentation uonnennnenonnunnnuunnununnunnnnnunnunnnnnnannnnannun nn 187 FL Retter EE 187 10 Scali System Guide Version 3 0 Chapter 1 Introduction 1 1 Purpose of the Scali System Guide The intention of the Scali System Guide is e to provide an overview of a Scali System e toprovide instructions for building a Scali System with respect to hardware and software installation and configuration e toprovideinstructions on how to use and manage a Scali System through the Scali Universe cluster management system Scali deliver its products e as turnkey supercomputer cluster systems TeraRacks e as interconnec
92. e AC 89 6 5 3 Default monitoring variables AAA 89 6 6 The SNMP daemon SCASNIMNPC ENN 90 6 6 1 Manual Start Stop near 91 6 7 Defining new monitoring varlables AAA 91 Scali System Guide Version 3 0 5 6 7 1 Obtaining the SNMP OlDe ENNEN 91 6 7 2 Editing the ScaMon configuration Te 92 6 7 2 1 Editing the class definition u usrnseereennnnn nenn nennen nennen 92 6 7 2 2 Editing hwgroup CLASSES Au 92 6 7 2 3 Adding the OlDs NN 93 6 7 2 4 Adding the variable definition for Universe 94 Chapter 7 Interconnect configuration usnuuannnnonnunnnnunnunnnnnnnnnnnnnunnnnnunnnnannnnnnnann 95 RR WEE 95 FMLA PEM IECCCUIG es ENN 95 7 2 SCI configuration from Scali Universe 96 7 2 1 The Interconnect menu enseenesnnnnnnnnnnnnnnnnennnnnnnennannnnsnnnnennnnnnannnnann 97 7 2 2 Link Status Window interconnect monitoring eeerneernnennnnerneenn nn 97 7 2 2 1 SCI link status gorapblce rara 98 7 2 2 2 SCI link status window options seersennnnennnnennnnnnnnennnnnn nenn 98 7 3 Command line interface Canto 99 7 3 1 Starting SACHNftool cn nm parken 99 73 2 USING SCACCO iio dadas 100 RE ME e MOS iaa aa add las 100 7 3 2 2 Command Cruncation arc rac rra rc 100 7 3 2 3 Node selection A 101 7 3 2 4 Disabling server log MESSAGES seen nennnnnennnnnnnen nennen 101 EPA A aiai eante ia a aa 102 73 20 0nAie HES Pie O ara 102 7 3 3 Status check of nodes and daemmons A 102 TIA THO TX COMMANG
93. e Alarms or the monitoring functionality of other subsystems like Queue Status Then there is a group for commands which all will present some possibly processed output of commonly used shell commands in a window like Uptime amp load The last view group is all related to graphical presentations of monitoring variables like the 2d bar view Common to all the graphical monitoring functions is that you will first have to select a view presentation and then the monitoring variable to present Monitoring variable selection is available from a submenu for each of the view selections Every view has the same variable sel ection submenu which corresponds to the list of variables defined for the scali monitoring daemon scamond This list is in turn customizable as a we will explain in 6 7 Defining new monitoring variables 78 Scali System Guide Version 3 0 6 3 Graphical monitoring Menu Item Description Alarms Brings up thealarm control window How tousealarms is described in section 6 4 Using Alarms Queue Status Brings up the queue status monitoring window this is described in detail in section 8 8 2 Monitoring jobs from Scali Universe Process list Shows processes on selected nodes User list Lists users on selected nodes Uptime amp load Shows uptime and load for selected nodes rup command Compact view Brings up a compact bar graph presentation ofthe selected mon
94. e can be downloaded from http www openpbs org To obtain the make system and patch files for ScaOPBS please e mail support scali com These will also be included in future releases of the ScaOPBS package 128 Scali System Guide Version 3 0 8 10 Software License 8 10 Software License OpenPBS Portable Batch System v2 3 Software License Copyright c 1999 2000 Veridian Information Solutions Inc All rights reserved For a license to use or redistribute the OpenPBS software under conditions other than those described below or to purchase support for this software please contact Veridian Systems PBS Products Department Licensor at www OpenPBS org 1 650 967 4675 sales OpenP BS org 877 902 4PBS US toll free This license covers use of the OpenPBS v2 3 software the Software at your site or location and for certain users redistribution of the Software to other sites and locations Use and redistribution of OpenPBS v2 3 in source and binary forms with or without modification are permitted provided that all of the following conditions are met After December 31 2001 only conditions 3 6 must be met 1 Commercial and or non commercial use of the Software is permitted provided a current software registration is on file at www OpenPBS org If use of this software contributes to a publication product or service proper attribution must be given see www OpenPBS org credit html 2 Redistribution in any form is only permit
95. e probe with link controller resets It makes most sense to use this in combination with the rings or positions modes Y ou may decide which node to be the origin in the cluster with the 1 lt node gt option This node will be listed first in the connection list and will have position 0 0 2D If you havea cluster where the nodes have different numbers of link controllers and or the rings are not connected through the same link controller you need to use the r flag This will create a connection list of all rings unbroken in the cluster Isolated parts which is not somehow connected to the starting node will not be included The topology will be TOP_FREE You may use this flag with the print write and rings modes It will not make sense to run any diagnostic mode connections and topology with the r flag Scali System Guide Version 3 0 145 Chapter 10 Troubleshooting 10 5 3 3 sci_hwdiag The sci_hwdiag program runs an MPI program systematically on parts of a Scali cluster to test the interconnect On older SCI cards pre PSB66 i e D31x SCI cards the b link retry must be set to zero The b link retry mecanism is contolled with the blink_retry_count parameter in the file opt scali kernel ssci conf To disable this SCI hardware feature please use opt scali libexec b_link sh off toturn it off To turn it back on use opt scali libexec b_link sh on This will turn b link back to the value beforelast time b_link sh off
96. e wait while the SCI network configuration is taking effect Unable to get system information from server Configuring Interconnect oe See E Configuring interconnect please wait some seconds Requesting server to reroute with routingtype SCA_ROUTE_MAXCY Configuring OpenPBS Leer GERS In the second example of the Post installation fix section a WulfKit installation is shown and the SCI network has to be specified manually 56 Post installation fix Sasa Sas aa Sa SS SS SS SS SS SS SS SS SS SS SS SS SS SS Specify SCI network o SEN Sieg The SCI nodes should be connected in one out of three possible topologies The simplest topology is a single ring where only SCI Link 0 or LO for short is used to connect all nodes The next topology is a 2D torus where LO is used to connect rings in one dimension and L1 is used to connect rings in the other dimension The third topology is a 3D torus where connections are the same as ina 2D torus and L2 is used to connect rings in the third dimension Scali System Guide Version 3 0 4 3 Scali software installation 1 Single ring using LO 2 2D torus using LO and L1 3 3D torus using LO L1 and L2 Q Please specify which SCI network topology is used 2 2 Q Please specify the number of nodes connected on a ring using SCI Link 0 or LO 1 16 2 Q Please specify the number of nodes connected on a ring using SCI Link 1 or L1 1 16 2 The configuration needs to know the position of
97. each node in the SCI network Please select which method of node position specification 1 Automatically from node list LO rings first 2 Manually give the position of each node Q Please enter your selection 1 The following SCI configuration has been defined name LO L1 12 scalil 11 111 scalil 12 211 scalil 21 121 scalil 22 221 Q Do you accept the above configuration y y Please wait while the SCI network configuration is taking effect Unable to get system information from server Configuring Interconnect SCH 2 Configuring interconnect please wait some seconds Requesting server to reroute with routingtype SCA_ROUTE_MAXCY 4 3 2 5 Functional testing Thelast parts of the installation deals mainly with tests of the system In the System Verification section some verification tests will be performed locally on each node System Verification EE EE EE EE EE EE EE Checking SCI driver 000m m OK e Checking SCI board jumpers __ OK Checking SCI links OK Testing SCI communication __ OK Scali System Guide Version 3 0 57 Chapter 4 Software installation In the System test section some system tests will be performed to verify that the system seems to work as supposed to System test Testing MPI communication
98. ecution pro grams which do not have the above mentioned problem set the value of RATESLEEP to 0 RSH_TESTACCESS true Define if the scahosts program should test the nodes for remote shell access to verify access to nodes The default is true FANOUT Define default FANOUT factor to be used when not explicitly stating the FANOUT factor on the commandline FANOUT is used for limiting the number of connections from one host when running ascash command When the number of nodes used as argument for the scash command exceedes the FANOUT factor the hosts will be divided into groups where FANOUT gives the number of groups and the scash command will be run as ascash command in each group in parallell Hence the number of con nections from one host will be limited to FANOUT number of connections Table 9 1 Options in the file ScaSH conf Scali System Guide Version 3 0 Chapter 10 Troubleshooting This chapter is the place to start if you are experiencing problems either using or installing a Scali system Some of the most common problems and their solutions are described here Remember that the FAQ Frequently Asked Questions lists at Scali s web site may contain information which is more up to date than the one contained in this document If your problems persists please refer to Technical Support on page 149 which explains how to get technical support 10 1 Hardware installation problems Before you inve
99. een introduced to make administration of the node configuration easier both for direct editing and for modifying the configuration file during installation of the SSP The format for a supercategory entry is supercategory lt supercategory gt lt categories gt where lt supercategory gt s the name of supercategory and lt categories gt iS a list of all categories mapping to this supercategory Here is an example of super category mappings supercategory workstation user mon_n supercategory sci_node base_n mon_n comm_sci_n queue_n supercategory smp_node base_n mon_n comm_smp_n supercategory eth_node base_n mon_n comm_eth_n queue_n supercategory sci_frontend base _f mon E comm_sci_f devel_sci supercategory smp_frontend base _f mon E comm emp E queue E supercategory eth_frontend base Ef mon E Gomm eth_f queue_f Herethe supercategory workstation will contain the categories user and mon_n When an entry in the nodelist section referrs to the supercategory workstation it will include packages belonging to the categories user and mon_n E 2 5 ScaPkg conf Node list The nodelist section defines which nodes to install packages on and which categories or supercategories they belong to Each host may belong to any number of categories The format for a node list entry is node lt name gt lt categories gt where lt name gt is the name of the node and lt categories gt is alist of all categories or supercategories
100. ense dat B 6 6 Missing license software ScaLM package You may also experience problems if the Scali licensing software is not properly installed TheScali licensing softwareis distributed in the package ScaL M which must be installed on every node in the system including the front end To check that this is really the case use scash from the front end and doa opt scali bin scash pbw rpm q ScaLM If you find ScaLM to be missing on some nodes the most efficient way to fix this is to use the scapkg software package installation utility The command opt scali sbin scapkg p ScaLM will make a system wide installation of ScaLM Please refer to Appendix E ScaPkg Scali Software installation program for more details about scapkg B 7 Older versions with FLEXIm Note Prior to release 2 1 of the SSP Scali were using the FLEXIm license manager system from Globetrotter Software Inc If you are experiencing problems regarding FLEXIm please refer to this section of a version 2 0 or older version of the Scali System Guide Older versions of the System Guide are available from the download section on Scali s web site The FLE XIm End User s Manual is also available online from Globetrotter Software Inc at http www globetrotter com TOC htm Scali System Guide Version 3 0 163 Appendix B Scali Software Licensing 164 Scali System Guide Version 3 0 Appendix C SCI Utility Programs C 1 SCI hardware status programs Th
101. ential connection scheme is described in 3 5 Console switching J ust like with the console switches another connection scheme will not present any problem for the system but you will spend more time with power server configuration without the benefit of intelligent default values Note f you usea private network with your system we strongly recommend that the power switches are connected to the public network If not you will have severe problems using the power switches if the frontend gateway goes down With Scali turn key systems TeraRack remote power switching is preconfigured A description can be found in the Scali Configuration Description delivery document Scali System Guide Version 3 0 33 Chapter 3 Hardware installation 34 Scali System Guide Version 3 0 Chapter 4 Software installation Aimed at OEMs VARs and kit builders this chapter explains how to set up the OS and install Scali software on the cluster If you own a turn key Scali system TeraRack this chapter can be viewed as supplementary information 4 1 Software installation overview Before software installation is attempted the cluster and interconnect hardware must be installed according to the guidelines in Chapter 3 The software installation can then be divided into two distinct steps OS installation and configuration Install and configure the operating system accor ding to the guidelines outlined later in this chapter e Sca
102. ep the format exactly as received from Scali including whitespaces and 1 as it otherwise might not work 160 Scali System Guide Version 3 0 B 4 License installation B 4 License installation When a license file is in place it must be installed or distributed to all nodes in the system During new SSP installations or SSP upgrades theinstall program will query you for FEATURE lines or a license file and automatically generate and distribute the license fileto the right locations The problem is that you seldom have your permanent license at this point in time so you will probably have to install a new license for your existing installation later This can be done in one of two ways B 4 1 License update installation with SSPinstall Starting with SSP 3 0 0 the SSPinstall program has a u option for upgrading a license file Using SSPinstall handles all aspects of license installation and is now the preferred way to install new licenses J ust doa opt scali sbin SSPinstall u and follow the instructions This will run the entire license entry part of the SSP installation all over so you can again choseto enter the license as a file or as separate FEATURE lines Please refer to 4 3 2 2 Installation Configuration for an example of license entry B 4 2 Manual license update installation If you fail to use the SSPinstall you may still install the license manually using the scarcp utility from ScaSH Assuming you areon
103. er to 10 Save your settings by pressing the Save button The alarm has now been defined and will show up in the Alarm main window Note that a newly defined alarm will always be set to inactive In order to activate it for use you must select it and press the Activate button Now to test the new alarm running any reasonably demanding program should push CPU load beyond 50 If you keep the alarm main window open you can see how the alarm background changes to red as the alarm is triggered Looking in your mailbox you could expect to find a mail similar to this Subject Scali alarm reporter Date Wed 12 Jun 2002 16 55 38 0200 From root lt root cmpq4 11 scali com gt To joe company com DO NOT REPLY TO THIS MESSAGE This is an automated message to inform you that the following alarm was triggered on Linux cluster cmpq4 11 scali com The expression for this alarm is cpu_user gt 50 The CPUload too high alarm was triggered on Wed Jun 12 16 55 38 CEST 2002 on the following nodes cmpq4 11 62 0 gt 50 cmpq4 12 62 0 gt 50 This alarm has no valid action 6 5 The monitoring daemon scamond The Scali monitoring daemon scamond provides filtered monitoring values tothe clients This means that two clients requires the same parameter only one SNMP request is sent Hence both the network load and CPU load from the scasnmpd is reduced 6 5 1 Manual start stop Although scamond will be installed and starte
104. erate a new license request at a later time simply run the now installed SSPinstall program with the 1 option and follow the instructions opt scali sbin SSPinstall 1 B 2 3 4 Last resort If you fail to run the SSPinstall program you can collect the necessary host Ds for the license request manually by running a small ScaLM utility program called Imhostid on every node The Imhostid is part of the ScaLM package and can be used in combination with scash to collect the hostid s for the entire system like this opt scali sbin scash pbw opt scali bin lmhostid scalil 11 The host ID of this machine is 0080c9876bla scalil 12 The host ID of this machine is 00a0c9876d57 scalil 21 The host ID of this machine is 00a0c9876bc1 scalil 22 The host ID of this machine is 00a0c9876acb Note that if you havea separate frontend you must remember to run Imhostid on this separately because it will not normally be included in the scash host list B 3 The license file B 3 1 Default location The default license file for Scali software is opt scali etc license dat This file holds the licenses for all Scali software and a copy must be distributed to the same location on all nodes in the system Scali System Guide Version 3 0 159 Appendix B Scali Software Licensing B 3 2 SCALM_LICENSE_FILE environment variable The SCALM_LICENSE_FILE environment variable may be used to alter the name and location of the license fi
105. erconnect menu added to the Main Window 96 Scali System Guide Version 3 0 7 2 SCI configuration from Scali Universe 7 2 1 The Interconnect menu The Interconnect menu has the following options Menu Item Description Set routing Opens the routingtype submenu which enables you to change the routing type for the system The available routing types will depend on SCI topology and license Please refer to section 7 4 Routing Auto routing Brings up a submenu which allows you to enable disable automatic rerouting The default value is ON Link Status Opens the link status window Table 7 1 The nterconnect menu items 7 2 2 Link Status Window interconnect monitoring The Link status window shows the status of all the SCI links in the network This is really an SCI specific monitoring view placed in the Interconnect menu Links are gr ouped per node and arranged in a 2D row column fashion with a node placement similar to that of the Large cons view in the MainWindow figure 7 3 Scali Universe delfi SCI link status LC a 0 el 0 LC1 EGG LCO LCH 0 delfi2 1 delfi2 3 delfi2 4 delfi2 2 LC 0 LC 0 LC1 0 LC1 0 LECO 0 LCO 0 Lco 0 LCH D delfi4 1 delfi4 3 delfi4 4 delfid 2 LC 0 0 LC1 0 LCO 0 LCO 0 delfi3 1 delfi3 3 delfi3 4 Be 0 E LC1 0 ECO 0 LCO 0 LCH 0 LCH delfil 1 d 3 delfil 4 delfil 2 Figure 7 3 High speed SCI network link status window Scali System Guide Version 3 0 97 C
106. erwise PBS daemons will not be started at the nodes and OpenPBS will not work properly 8 3 8 PBS enforcement ScaOPBS offers an option to enforce use of PBS to access the nodes in the system In order to ensure that jobs use only nodes that PBS has assigned any kind of remote login execution on nodes not assigned must be blocked The solution used is based on work from Willy Weisz at VCPC and makes use of the following facilities e prologue epilogue feature of OpenPBS version 2 3 15 e PAM Pluggable Autentication Modules lt http ww us kernel org pub linux libs pam gt Prerequisites The following files in the directory etc pam d of any available node have to be amended e login e rsh e sshd if ssh is used to login to the target node e rexec any other name of a remote login execution service on target nodes The following line must be present in all of these files Default for Redhat 7 x Scali System Guide Version 3 0 119 Chapter 8 Batch system ScaOPBS session required lib security pam_limits so Secondly the root user must have access without password between all nodes in the system Running opt scali libexec scaopbs config e will enforce pbs usage Similarly opt scali libexec scaopbs config d will disable this option The user will be rejected login to nodes in the cluster when PBS enforcement is enabled rsh scalil 12 Too many logins for lt user gt Permission denied rlogin
107. f Scali System Guide Version 3 0 137 Chapter 9 ScaSH parallel shell tools Option Default value Description RCP_CMD rep p Which program should be used to copy files from to a remote node The program specified must support the following calling convention RCP_CMD lt host gt lt fromfile gt lt tofile gt or RCP_CMD lt fromfile gt lt host gt lt tofile gt LCP_CMD cp P Which program should be used to copy files on the local node The program spedfied must support the following calling con vention LCP_CMD lt fromfile gt lt tofile gt PARLIMIT expr ulimit n 3 When using the background option with one of the ScaSH tools this parameter will control the number of commands that are actually run in parallel ScaSH tools will wait until the below number of processes are finished before issuing the next batch RATESLEEP Heavy use of rsh rcp can lead the inet daemon to enter a non responsive state This is related to the fact that rsh uses socket ports below the 1024 limit and exhausts the number of free ports OR that the number of successive connections to the shell service are too many to be handled Both problems arise because the sockets enter a TIME_WAIT state which uses 120 seconds to ter minate To limit the number of successive rsh accesses to each host use RATESLEEP to adjust interval between each access to that node If you are using ssh or other remote ex
108. fault category selection of sci_ and eth_ node frontend and shows how to add smp_node and smp_frontend categories to the frontend machine in the system Other predefined categories may also be chosen These are defined in opt scali etc ScaPkg conf in the lines prefixed with the keyword supercategory The choice of smp_node and smp_frontend category will 52 Scali System Guide Version 3 0 4 3 Scali software installation imply installation of software needed for running ScaMPI applications on an SMP machine without involving the SCI network Hencean SCI adapter is not needed This will typically be a choice for development and porting purposes Determine node categories Please wait while retrieving information from nodes The following node categories have been defined scalil 11 sci_frontend eth_frontend sci_node eth_node scalil 12 sci_node eth_node scalil 21 sci_node eth_node scalil 22 sci_node eth_node Q Do you accept the above configuration y n Please specify nodename s to modify brace expansion allowed Press lt return gt on an empty line after adding your node s O Please enter node name s scalil 11 O Please enter node name s The following nodes have been selected for modification scalil 11 Q Do you accept the above selection y Please select one or more of the following categories workstation sci_node smp_node eth_node sci_frontend smp_frontend eth_fr
109. files repository The path to repository section defines a path to the directory holding the software packages repository Note an architecture identification string will be appended at the end of the path corresponding to the output from the systype c script from the ScaEnv package when searching for packages made by Scali For example if usr local src pkg isthe repository a Linux i86pc system would expect to find package files in usr local src pkg Linux2 i86pc and an UltraSPARC system 182 Scali System Guide Version 3 0 E 2 Configuration would expect to find package files in usr local src pkg SunOS5 sparc u When searching for general packages not Scali packages the search order would be usr local src pkg Linux2 i86pc and then usr local src pkg if the package was not found at the first search location It is possible to specify more than one path to repository locations The search order for each package will be as listed in the path to repository section of the configuration file The path to repository specified in the configuration file may be overridden with the r option E 2 2 ScaPkg conf package list The package list section defines which packages are associated with which categories Each line in the package list is on the format package lt name gt lt categories gt where lt name gt is the package module part and lt categories gt is one or more categories separated by space The categories are freely de
110. fined and are referred to in the Super categories section or directly in the Node list section of the configuration file Each package can belong to any number of categories Packages with no category are not to be installed only kept in repository Packages will only be installed on nodes with corresponding categories or supsercategories No user changes should normally be necessary Hereis an example package list Format package lt name gt lt categories gt package ScaBoot base_n package ScaConfC comm_sci_f package ScaEnv base_f base_n devel user E 2 3 ScaPkg conf dependency list The dependency list section defines dependencies between packages Each package may have several dependencies A line in the dependency list is on format dependency lt pkgname gt lt deppkg gt where lt pkgname gt is the package module and lt deppkg gt is a list of all packages that this package depend upon Note that no user changes should be necessary in this section Hereis an example package dependency list entry defining dependencies for the package ScaConfC Format dependency lt pkgname gt lt deppkg gt dependency ScaConfC ScaConfSd ScaConfNd ScaSH Scali System Guide Version 3 0 183 Appendix E ScaPkg Scali Software installation program E 2 4 ScaPkg conf Super categories The supercategories section contains mappings where several categories are mapped into single supercategories This extra level of abstraction has b
111. from simple xload style 2D history graphs to accelerated 3D OpenGL graphics 6 1 1 Architecture LAN scasnmpd viv Remote Universe GUI scamond u node process Local Universe GUI frontend cluster nodes Figure 6 1 ScaMon components in context TheScaM on system consists of several software components The Scali SNMP daemon scasnmpd must run on every node to be monitored The Scali monitoring daemon scamond must run on thefrontend of the cluster Scamond allows clients to subscribe to monitoring variables which will be extracted over SNMP from the nodes at regular intervals Clients local or remote uses SSSC Scali Secure Socket Communication to communicate with scamond and can therefore safely monitor the system over the Internet Scali System Guide Version 3 0 77 Chapter 6 System Monitoring 6 2 Using monitoring from Scali Universe 6 2 1 The Status menu The ScaM on plugin for Scali Universe adds the Status menu to the M ainWindow This is the main entry point to monitoring functionality An overview of the Status menu is given in table 6 1 Process list User list Uptime amp load Co Suh ar Compact view scalil 11 sq a 2d bar view 3d bar view 2d history view 3d history view z 3d system view scalil 21 sq Figure 6 2 The Status Menu Note that the Status menu arranged as three logical groups F rom the top you have access to monitoring subsystems lik
112. g Parallel Shell commands The parallel shell window allows you to run shell commands on selected nodes in parallel This is a graphical frontend to a subset of the ScaSh parallel shell utilities See ScaSH parallel shell tools on page 131 To open the parallel shell window select Parallel shell from the Run menu You may then type in the command in the Command to run field and press the Run command button to start it J ust likein the MPI launchers the Run command button doubles as a Stop command button while the command is running Output from the commands will be presented in the window as shown in figure 5 12 Heiki commana to nn par SNA l Run command root 21662 3028 1652 s Jand9 3 root 3802 3028 1652 s Jand9 root 31503 2948 1608 SN Jan 9 root 28118 2948 1608 SN Jand9 Figure 5 12 Example of a Parallel shell command 5 3 4 5 Node terminal sessions By using the Node Terminal option from the Run menu terminal emulators will be started with connections to all selected nodes As you will soon find out this is a very practial feature for a number of reasons Firstly the terminal emulator is built intothe Scali Universe so you do not need any other package on the system in order to run Xterm like terminal sessions This is true even for the MS Windows version of the product Secondly the communication runs through the single encrypted socket connection SSC from Scali Universe to
113. g Scali Universe The difference is that some tools notably configuration and maintenance modules are not available to regular users Maintenance and configuration modules gives access to remote power control interconnect configuration and software installation 5 3 Using Scali Universe When logged in toa Scali System the M ainWindow of Scali Desktop appears The main window of the Scali Desktop shows all nodes in the network The node symbols explained in table 5 1 are used to indicate the operating system and status of each node File Run View Status 7 delfi2 1 3 delfi2 4 delfi2 2 Ls B delfi4 1 3 delfi4 4 delfi4 2 Ls La deina 1 delfi3 4 delfi3 2 La Lal dem delfil 2 Figure 5 8 Scali Universe system main window with four selected nodes 5 3 1 Node selection One important use of the main window is node selection This because most of the application launchers available from the Run menu requires that you have selected a set of nodes as targets for the application Some launchers also has a livelink to the main window which means that while having a launcher window open you may change the node selection in the MainWindow and re run the application on the new set of nodes Scali System Guide Version 3 0 67 Chapter 5 Scali Universe You may select set of nodes by left dragging the mouse in the window Alternatively nodes may be toggeled between selected and unselected state by lt
114. g diagnostic tools and how to get assistance from Scali Appendix A collects useful meta information about the Scali software platform like contents of the distribution media daemon overview and configuration file overview Appendix B is a detailed description of the Scali licensing system ScaLM Appendix C is a detailed reference to the SCI interconnect utility programs Appendix D is a reference to the ScaConf interconnect configuration system Appendix E is a detailed reference to scapkg the Scali software installation program Appendix F provides a list of related documentation you may consult for additional information 1 3 How to read this guide This guide is written for skilled computer users and professionals It is assumed that the reader is familiar with the basic concepts and terminology of computer hardware and software since none of these will be explained in any detail Depending on your user profile some chapters are more relevant than others We recommend that e System installators should read Chapters 2 3 4 8 10 and B e System administrators should read everything e Ordinary users should read Chapters 2 5 6 7 and 9 1 4 Acronyms and abbreviations Abbreviation Meaning HPC High Performance Computer NIC Network Interface Card MPI Message Passing Interface OEM Original Equipment Manufacturer SCI Scalable Coherent Interface SSP Scali Software Platform is the name of all Scali software pac
115. g together the term Scali Universe is commonly used to refer to the GUI 5 1 Overview TheScali Universesystem management GUI enables singlesystem view monitoring configuration and use of Scali cluster systems The Scali Universe is currently available in two versions Universe and Universe XE Universe is the entry level cluster management solution which provides basic installation monitoring and management features Universe XE is the extended professional cluster management solution which incorporates more features for professional industrial cluster computing In this manual features available only in Universe XE will be marked with EEE In the GUI features not available will simply not show up or be greyed out in the menus LAN Universe GUI framework Internet Queue Monitoring 2 i i gt Plugin Plugin Plugin ugi teen local workstation remote cluster Figure 5 1 Scali Universe overview The Scali Universe GUI is built around a plugin architecture which consists of a framework and a number of client side plugins The framework provides backbone functionality while the plugins provides the GUIs for different components of Scali Universe In this chapter we will explain the framework and some basic plugins The more specialized plugins will be explained in chapters that also address the related functionality like monitoring in chapter 6 or the queue system interface in chapter 8 S
116. guration ScaSH Installing package ScaDiag Preparing package configuration ScaDiag Installing package ScaLM Preparing package configuration ScaLM Installing package ScaPkg Preparing package configuration ScaPkg Installing package Checking access to nodes OK Preparing nodes for install _ OK pe Scali System Guide Version 3 0 4 3 Scali software installation Entering the license by specifying an existing license file will reduce the risk for errors compared with copy and paste of each FEATURE line Setup license s 7772227277222 Please select which type of license s you have 1 FEATURE line s typically demo license key s 2 License file typically permanent license key s 3 Skip license information I will do it later NOTE Later tests may fail due to missing license key s Q Please enter your selection 1 2 Q Please enter the path to the license file root license dat Node categories is a way of specifying what type of software to install on each node eth_node and sci_node depending on the presence of a SCI adapter or not will bethe default selection for a node The choice of eth_node category will imply installation of software needed for running MPICH applications over ethernet The choice of sci_node category will imply installation of software needed for running ScaM PI applications over SCI network Determine
117. guration files in opt scali etC nunneensnensnennnnnennnnnnnnn nennen 156 ScaC OT packages iia tala Mites 171 Available server options Default values are marked with 173 Scali System Guide Version 3 0 189 190 Scali System Guide Version 3 0 List of figures 2 1 2 2 2 4 2 5 3 2 33 3 4 3 5 3 7 3 8 3 9 3 10 3 11 3 12 3 13 System architecture of a 4x4 Scali system nensennnnensnennnnnnnnennnnnnnnn nennen 16 Scali system cabinet examples Standard half size Terarack left anda ruggedized version for portable supercomputing rtoabt 17 OEM partner packaging F ujitsu Siemens hpcLine left and Dell right 17 A 4x8 Scali system boxed in Scali cabinets AEN 18 Open front and rear side of a Scali system in a Scali Cabinet eee 18 A 4x4 cluster with XY coordinates physical view 19 Using private network for the Scali System A 21 Scali system with channel bonding AE 22 SCI hardware from Dolphin PCI adapter daughter card standard cable left and flexi cable riobt EE 23 Identifying LO and L1 connector pairs on SCI cards D39bl 24 D31x SCI adapter card jumpers A 24 Node with SCI adapter card and daughter card inserted AAA 25 Avoid long cables by interleaving interconnect in each ing 27 Example of 1x8 single ring cabling in a 1x8 cabinet A 28 Example of 2x4 two dimensional system cabling in a 2x4 cabinet sss 29 Example of 2x4 two dimensional system cabling in a 1x8 Cabinet 29 A 3X3X2 3D S E T
118. h go to each host based on configuration data Find out which packages are valid for each category Find out which packages are valid for each host 54 Scali System Guide Version 3 0 Match valid package names with available packages in repository Removing hosts not allowed for updates or no packages available Starting remote polling script on hosts 4 3 Scali software installation Copy host dependent package list to respective hosts Waiting on feedback from hosts which packages are needed Poll for list of needed packages on remote hosts Remove hosts not in need of any updates Copy packages to hosts Copy files to hosts Checking platform Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Package Linux2 ScaComd is ScaCons is ScaDesk is ScaEnv is ScaExecd is ScaFPMB2 is ScaLM is ScaOSusbrm is ScaPowd is ScaSCI is ScaSH is ScaSISCI is ScaSNMPd is ScaSNMPt is ScaSSP is ScaConfNd is ScaConfSd is ScaDiagSC is ScaFmpich is ScaMond is ScaMPI is ScaMPICHt is ScaMPItst is ScaOPBS is ScaSCIddk is ScaConfC is ScaMPE is ScaSCIadap is Tell all hosts that all i86pc now now now now now now now now now now now now now now now now now now now now now now now now now now now now copied copied copied c
119. hapter 4 Software installation Introduction SE SE EE EE E EE EE E EE EE E EE EE EE EE EE EE EE EE E EE SE Welcome to the Scali Software Platform SSP installation program This program will guide you through the installation procedure It is divided into several sections Introduction some general information some sanity checks Configuration specify software to install specify nodes to install on Installation check that the nodes are correctly prepared the actual software installation on all nodes post installation setup configuration Verification verify that the installation was successful If you abort in the middle of the installation process you have the option of saving the current state and restore it at the next invocation You may also confirm each single step of the restore to do modifications anywhere in the restore process Whenever your input is required you will see a line like this Q lt default option gt The default option is given in square braces and is chosen if you just press return If you experience problems during installation please contact support scali com Wulfkit customers please use the e mail address wulfkit support dolphinics no version SSP_3_0_2 SSPinstall 1 96 Checking permissions OK Restoring info from savefile Ed We have detected a state file from a previous installat
120. hapter 7 Interconnect configuration 7 2 2 1 SCI link status graphics Within the Links Status window the SCI links for every node are displayed as a stack of virtual LED indicators as shown in figure 7 4 The most important information liesin thecolour ofthe LED indicator Green means up and enabled and represents the OK statefor a link Y ou really want tosee all green LEDs If an SCI link goes down ScaConf will disable it and possibly some others as a result of the new routing which is applied to avoid the problem Disabled links show up as yellow LED indicators Y our system may be fully operational even with some disabled links but you should try to resolvethe cause ofthe problem as the redundancy of the system decreases with every failed component Link LED indicator GREEN enabled and UP YELLOW disabled RED enabled and DOWN Link number Node Name Link error counter Figure 7 4 Link status graphics explained Finally a red LED indicates that the link is down and enabled As a permanent state this represents a really bad error situation which has not yet been handled by ScaConf Normally you ll only seethered LE Ds as a transitional state The Link error counter positioned tothe right of each LED indicator also gives important information about the state of individual SCI links High numbers of SCI link error may be a sign of a bad or loose SCI cable For counts higher than 0 the number
121. he system and the underlaying SCI interconnect and must run on every node to be managed The Scali System Guide Version 3 0 95 Chapter 7 Interconnect configuration Scali interconnect configuration server daemon scaconfsd is the component that posses the intelligence in the system It must run on the frontend of the cluster F or communication between the node daemons and the server daemon ScaConf uses the ScaCom communication backbone this also implies that the Scali communication daemon scacomd must also run on every cluster node Scali Universe Clients local or remote uses SSSC Scali Secure Socket Communication to communicate with scaconfsd and can therefore safely configure the cluster system over the Internet In addition there is a text based interface to the server called scaconftool which provides the most complete management interface to the configuration system 7 2 SCI configuration from Scali Universe When managing systems with SCI the ScaConf plugin for Scali Universe will add an Interconnect menu to the main window The Interconnect menu enables you some control over the routing of the high speed SCI network of the system Scali Uni Set routing F dimensional routing y first Auto routing dimensional routing x first Link status c3 routing improved maximum connectivity yx routing no errorchecking xy routing no errorchecking scali maximum connectivity Figure 7 2 Int
122. his since it will allow for centralized and efficient BIOS setup and inspection 3 3 Ethernet setup Scali systems requires Ethernet access between the frontend and all nodes in the system The Ethernet is used for control and monitoring even for systems with a high performance interconnect installed Generally you have two choices for setting up the Ethernet either to include all nodes in the local LAN or to provide a private network for your Scali system 3 3 1 Using the site LAN This is the simplest and least secure option There s not really a lot to consider here J ust connect all nodes toa HUB or preferably a switch in your local network g ethO T Node 1 4 ethO 7 r gt Node 2 Firewall Je Need NIS DNS ethO Frontend Node File Server eth etho Node n v vv Ethernet switch HUB Ethernet switch HUB Site LAN Scali system private network Figure 3 2 Using private network for the Scali System Scali System Guide Version 3 0 21 Chapter 3 Hardware installation 3 3 2 Using a private network Using a private LAN for your system provides better security and performance With this solution the frontend is set up as the gateway between the local LAN and the processing nodes The frontend therefore needs one more Ethernet port adapter than the compute nodes Scali recommends that the primary Ethernet port ada
123. in a larger scrolled window The2D history view scales dynamically on the value axis 82 Scali System Guide Version 3 0 6 3 Graphical monitoring 6 3 6 3D History View RZ Figure 6 8 3D History view The3D history diagram resembles a moving mountain range and enables you to easily spot deviating values anywhere in the system within the history time span that is visualized Increasing values ofthemonitoring variableis shown asincreasing height Nodes arelaid out along one axis while time runs along the other The 3D history view supports all 3D camera controls 6 3 7 3D System View compound view Figure 6 9 3D System view on a 2D SCI system Scali System Guide Version 3 0 83 Chapter 6 System Monitoring Especially on SCI systems the 3D system view will provide you with a more useful information in a single compound view The nodes can selectively be shown as plain colour coded boxes or as a platform including a 3D bar graph for monitoring selected values The layout of the nodes in 3D space tells you the topology of the system If an SCI network is present this will be shown as thick pipes between the nodes These pipes will change colour according tothestate of theSCl link following the same col our code as described in Link status graphics explained on page 98 The layout of the nodes in 3D space tells you the topology of the system The 3D System View supports all 3D camera controls 6 4 Using Alarms Sc
124. ion attempt You have the option of restoring the information recorded up to the point where you last exited If you choose to restore you also have the option of qualifying each step if you want to change something in the recorded information Please choose one of the following y restore default s restore with single step qualifying 42 Scali System Guide Version 3 0 4 3 Scali software installation n ignore restore possibility Q Do you want to restore the recorded state y n Accept license agreements This is licensed software Before installation can proceed you must accept the following license terms for Scali and integrated third party software Scali AS Software License terms GNU General Public License terms Console Server License terms MPICH License terms OpenPBS Software License terms DG WN EE Q Select the license terms to be read or type yes to confirm that you have read and do accept the license terms above 1 ATTENTION USE OF THE SOFTWARE IS SUBJECT TO THE SCALI AS SOFTWARE LICENSE TERMS SET FORTH BELOW USING THE SOFTWARE INDICATES YOUR ACCEPTANCE OF THESE LICENSE TERMS IF YOU DO NOT ACCEPT THESE LICENSE TERMS YOU MAY RETURN THE SOFTWARE FOR A FULL REFUND The License Terms below govern your use of the accompanying Software unless one of the following exceptions apply You have a separate signed agreement with Scali AS The Software is licensed
125. is chapter will describe some SCI utility programs These programs are located inn the bin and sbin directories of the Scali installation They are useful for testing and monitoring the SCI hardware and of limited interest to the ordinary user and should not be used under normal circumstances C 1 1 sciping The sciping program checks the reachability of SCI nodes through an adapter Default adapter number is 0 The most important options are listed below Thesciping command will send an SCI packet via the SCI interconnect from the node it is run on to the nodes with the nodeid specified on the command line This utility is useful for testing connectivity between nodes Usage opt scali bin sciping L x t s lt tag gt lt nodeid0 gt lt nodeidl gt Options a lt adapter gt Use adapter number adapter instead of adapter 0 m lt count gt Send count requests for each node on command line c lt sec gt Continuous mode Repeat requested pings every sec seconds p Print performance statistics t ping u Allow loop back pings potentially unsafe h Print this text V Only print nodes that fails to respond links that are down L Write a log message to the system log at both nodes requires all involved nodes to run a similar sciping Scali System Guide Version 3 0 165 Appendix C SCI Utility Programs C 1 2 scimonitor Thescimonitor program watches link status changes for an SCI adapter The default is adapter number 0
126. is goes for 2D torus topology First run is for all horizontal rings second run is for all vertical rings third run the machine is divided into 2x2 squares of neighbour nodes In 3D torus topology the cluster will be divided into IcO_rings Icl_rings and Ic2_rings No rectangles or cubes are tested 146 Scali System Guide Version 3 0 10 5 SCI Interconnect problems Mjob1 Job 3 Idle nodes Job2 Ejo 4 Figure 10 1 Vertical horisontal and rectangle groupes of nodes running the same test application job in a 4x4 2D torus The wraparound connections of the tours are not showen Thetest application will be run on all IcO_rings vertical in parallel as showen in part 1 of Figure 10 1 then on all Ic1_rings horizontal in parallel as showen in part 2 of Figure 10 1 When thering by ring tests are completed the cluster will be devided into 2x2 rectangles of nodes as showen in part 3 and 4 of the fiugre The rectangles will be split in several groups such that no two nodes not in the same rectangle are on the samelc0 or Icl ring There will be a staircase like pattern of the rectangles which are run in parallel Larger clusters will have more groups a 6x6 cluster will have 3 groups with 3 rectangles in each Nodes which are down will beignored and rings in which these are members will not be tested No rectangles will
127. itoring parameter for all nodes 2d bar view Brings up a standard 2D bar graph presentation ofthe selected monitoring parameter 3d bar view Brings up a 3D bar graph presentation ofthe selected monitoring parameter 2d history view Brings up a two dimensional xload type presentation of the selected monitoring parameter 3d history view Brings up a 3D history graph carpet graph ofthe selected monitoring parameter 3D system view Brings up a 3D compound view of the system Table 6 1 The Status Menu 6 3 Graphical monitoring As explained earlier each of the monitoring variables made available from the monitoring daemon may be combined with any suitable graphical presentation The different graphical presentations available in Scali Universe each have different strengths which will be highlighted in the following sections Scali System Guide Version 3 0 79 Chapter 6 System Monitoring 6 3 1 Common functionality 6 3 1 1 Pop up menu Common to all presentation views is a pop up menu with options for the presentation The pop up menu is activated with a right click in the window Menu items are explained in table 6 2 Note that not all options are available with every view Menu Item Description Large Icons Selects the default Large Icons view of the presentation This often contains more textual information than the Small Icons view Small Icons Selects the space saving
128. job from Scali Universe to 4 nodes with 2 processes at each node The Number of nodes field specifies the total number of nodes to allocate for the job The Processes per node field specifies the number of processes at each node ie total number of processes Number of nodes Processes per node The Command Line field specifies the actual command to be started by PBS This command will be started at the first node allocated for the job The ScaMPI mpirun and similarly MPICH mpirun script will pick up the node specification from the PBS NODESFILE and PBS_NPN environment variables and distribute the processes to the allocated nodes Regular commands will only be started at the first allocated node The Job Name field describes the name associated to the job this name will show up in the queue status window figure 8 5 If no job name is given the basename of the Command Line will be used eg mpimon for the command line specified in figure 8 1 124 Scali System Guide Version 3 0 8 8 Using ScaOPBS from Scali Universe The Queue field is used for selecting the queue where the job will be submitted Reserve nodes only will allocate the specified number of nodes for the user but will not start any application When selecting Reserve nodes only the Minutes to reserve field appears where the duration of the reservation must be specified The name of the reserved nodes may be obtained from the Job details window see figure 8 6 or from the
129. kages VAR Value Added Reseller Table 1 1 Acronyms and abbreviations 12 Scali System Guide Version 3 0 1 5 Terms and conventions 1 5 Terms and conventions Unless explicitly specified otherwise gcc gnu c compiler and bash gnu Bourne Again SHell are used in all examples Term Description Node A single computer in an interconnected system consisting of more than one computer Cluster A cluster is a set of interconnected nodes with the aim to act as one single unit Scali system A cluster consisting of Scali components Frontend A computer outside the cluster nodes dedicated to run configuration moni toring and licensing software MPI process Instance of application program with unique rank within MPI_COMM_WORLD UNIX Refers to all UNIX and lookalike OSes supported by the SSP i e Solaris and Linux Windows Refers to Microsoft Windows 98 M e NT 2000 Table 1 2 Basic terms 1 6 Typographic conventions Term Description Bold Program names options and default values Italics User input mono spaced Computer related Shell commands examples environment variables file locations directories and contents GUI style font Refers to Menu Button checkbox or other items of a GUI Command prompt in shell with super user privileges Command promt in shell with normal user privileges Table 1 3 Typographic conventions Scali Syste
130. ktop bin ScaDesktop exe and the program will start After an introductory glimpse of the Universe startup screen the System Management window will appear Scali Universe can control multiple Scali Systems which again may consist of hundreds of nodes The Scali System Management window is where you select which Scali System s to manage 5 2 4 Configuration of the Scali Universe GUI Before you can start using the Scali UniverseGUI you must providesome information about the systems you want to manage This is known as configuration of the Scali Desktop As aresult you will havea personalized desktop configuration user profile The user profiles are stored in the file HoME scali ScaDesk conf When you start Scali Universe for the first time you will see a dialogue box with the following warning Could not read confi gfile using system defaults This is because no configuration file was found in HOME scali J ust press OK to get on with the configuration The Scali System Management window will then appear Figure 5 2 Initial unconfigured Scali System Management window Scali System Guide Version 3 0 63 Chapter 5 Scali Universe Eile Edit Delaware Idefix Local Scali 1 Test system Figure 5 3 An example Scali Universe session with multiple monitoring views 64 Scali System Guide Version 3 0 5 2 Getting started with Scali Universe Initially only a default Scali System with system
131. l configuration files node property file and server configuration may be backed up by running the command opt scali libexec scaopbs config b lt backup dir gt where lt backup dir gt is the directory where to create the backup files 116 Scali System Guide Version 3 0 8 3 OpenPBS configuration 8 3 3 The node property file The default node property file is named var spool PBS server_priv nodes It iS only available at the frontend and has the following format lt node gt lt nodename gt lt arch gt lt mpi impl gt where e lt node gt and lt nodename gt are the name of the node e lt arch gt is the architecture for the system i e Linux2 i86pc Linux2 ia64 Linux2 alpha or SunOS5 sparc u e lt mpi imp gt IS ScaMPI Or MPICH Example node configuration file for a 4 node system with ScaMPI and MPICH installed cat nodes ScaOPBS node file node name followed by properties node 1 node 1 Linux2 i86pc ScaMPI MPICH node 2 node 2 Linux2 i86pc ScaMPI MPICH node 3 node 3 Linux2 i86pc ScaMPI MPICH node 4 node 4 Linux2 i86pc ScaMPI MPICH Please see section 3 2 2 at page 21 in the Portable Batch System Administrator Guide available at the frontend as opt scali contrib pbs doc pbs_admin_guide pdf for further description of the node configuration file 8 3 4 Server configuration qmgr The server is configured at runtime by the application opt scali contrib pbs bin qmgr Server configurati
132. l restorethe contents of the configuration window from the last saved profile Finally to finish the configuration press Close in the System Configuration window and the System Management window will be updated to show a cluster icon and name for each of the configured systems Seal Universe Die Edit Figure 5 6 Scali System Management window with five Scali Systems 5 2 5 Logging in to a system Like any other single networked computer you must alsolog into a Scali System before you can start using any of the tools in Scali Universe Tologin to a Scali system you simply double click on the corresponding system icon in the System Management window First you will see small countdown graphic while the system is contacted When connection is established you will be presented with a login dialogue box for this system Enter your user name and password and press OK Login Kn Password a OK Cancel Figure 5 7 Login dialogue box Theusername and password will then be encrypted and sent for verification locally on the frontend for this Scali system Please note that it is therefore your access rights on the Scali system that counts not your access rights on the workstation running the Scali Universe GUI Depending on whether you login as a regular user or as root the Scali Desktop will operate in either regular user mode or system administrator mode 66 Scali System Guide Version 3 0 5 3 Usin
133. lation SCI is an international standard IEEE 1596 of a high speed interconnect Dolphin Interconnect Solutions makes a number of SCI products among which their PCI SCI adapter cards are used by Scali The performance of these cards is amazing Several hundreds of megabytes per second and sub microsecond latency on the SCI link layer with PCI performance approaching the limits of the bus 3 4 1 SCI hardware overview Figure 3 4 SCI hardware from Dolphin PCI adapter daughter card standard cable left and flexi cable right There are currently two generations of PCI SCI adapter in use The old D31x series and the newer D33x series The main difference is that the new cards supports 64bit 66M hz PCI and 3D SCI topologies Needless to say all new Scali systems with SCI are equipped with D33Xxcards but the SSP still supports D31x series Scali System Guide Version 3 0 23 Chapter 3 Hardware installation Depending on model an SCI card supports up to three SCI links which we refer to as LO L1 and L2 On newer boards the connectors are clearly marked if not you can safely assume that connector numbering starts with the connector pair closest closest to the mainboard PCB being LO ref 3 5 PCB side IN connectors yellow OUT connectors blue Figure 3 5 dentifying LO and L1 connector pairs on SCI cards D335 3 4 2 SCI adapter installation For older D31x cards please make sure that all 4 jumpers areinstal
134. le Simply set it to the full path of the license file If SCALM_LICENSE_FILE contains a valid file name ScaLM will look for licenses in this file before the file in the default location B 3 3 License file example After generation by the SSP install program or after manual editing the license file could look something like this for a 4 node system with ClusterE dge product additional support for large SMPs with ScaM PI and support Scali AS license file ClusterEdge license FEATURE clusteredge scald 3 000 permanent 4 12ED7FE94B27402DFBAB30A645F794075F5BA2C580514DDAE421B000C8 HOSTID 0034567966b 00345677f4c 003456764c 0034457eac0 Large SMP support for ScaMPI FEATURE mpismp scald 1 910 permanent 4 2123EF271B000FF12587EE12345DFBAB30A645F794543F5BA2C5805143 HOSTID 0034567966b 00345677f4c 003456764c 0034457eac0 Support FEATURE support scald 1 000 1 apr 2003 1 97152A67768551B00052F7230301B1DBCFE8B58D31C988AB51E3C6F3DB HOSTID 0034567966b Lines starting with are comments You may have several FEATURE lines one for each licensed software product or add on B 3 4 Adding or updating features licenses When you receive a new software license from Scali be it a demo or a permanent license this will be in the form of FEATURE lines The FEATURE lines should be inserted in the license file license dat replacing older ones for the same feature When receiving a multi line FEATURE it is important to ke
135. le may also be used to determine the number of processes allocated The environment variable PBS_NPN may be used to get the specified number of processes pr node Example start up script cat mpiexample bin sh Get number of nodes allocated NO_OF_NODES cat PBS_NODEFILE egrep v S wc 1 awk print Multiply with number of processes pr node to get total number of processes NO_OF_NODES expr NO_OF_NODES SPBS_NPN opt scali bin mpirun machinefile PBS_NODEFILE np NO_OF_NODES npn PBS_NPN opt scali examples bin producer Example submission opt scali bin scasub np 4 npn 2 mpiexample This example will start a total of 4 processes two processes pr node Scali System Guide Version 3 0 123 Chapter 8 Batch system ScaOPBS 8 8 Using ScaOPBS from Scali Universe Scali Universe provides a graphical user interface towards OpenPBS 8 8 1 Submitting jobs using Scali Universe In user mode Scali Universe provides options to submit jobs to OpenPBS This option is only available when Scali Universe is started in user mode as OpenPBS does not permit root to submit jobs To submit a job select Run gt Submit Job Scali Universe scalil Job submission File Options Number of nodes 4 Processes per node 2 Reserve nodes only Command Line optscali bin7mpirun opt scali examples bin allZall Browse Jon namo scali_exec Figure 8 1 Example of submitting a ScaM PI
136. led as indicated by figure 3 6 before inserting the adapter card D33x series cards are jumperless Figure 3 6 D31x SCI adapter card jumpers If you have purchased a multi dimensional SCI network using an SCI daughter card for the second D311 D312 or third D337 set of SCI links please make sure the daughter card is securely mounted tothe main SCI card beforethe combo is inserted into the node 24 Scali System Guide Version 3 0 3 4 Dolphin SCI interconnect installation Then insert the SCI adapter card with daughter card if applicable intoa freePCI slot in the node If there are ISA slots present you should if possible avoid the PCI slots sharing interrupt lines with ISA slots Be sure to fix the board properly in the slot for mechanical stability Repeat the procedure on all nodes in the cluster Figure 3 7 Node with SCI adapter card and daughter card inserted Note On older D31x cards the LEDs on the back of the SCI board will be red yellow until the proper software has been loaded this is quite normal When the software is loaded and the SCI link is OK the LEDs will turn to green On newer D33x cars the LEDs will turn to green as soon as the SCI link is OK even if no software has been loaded For more details on how to interprete the SCI card LEDs please refer to SCI Troubleshooting on page 143 3 4 2 1 64bit 66 MHz PCI bus operation With the new D33x cards you
137. li software installation nstall the Scali software with the SSP Scali Software Platform installation program install The installation program installs selected Scali software products on the entire cluster Installation includes configuration and testing of the installed hardware and software on different levels Scali software and instructions can be found on the SSP CD ROM or downloaded from http www scali com download 4 2 Operating system preparation Before the SSP can be installed each node in the cluster needs a properly configured installation of a supported operating system Supported operating systems and versions are clearly listed in the SSP release notes which can be found in the news file in the SSP distribution or on the Scali web site Please also read the doc os file included in your SSP distribution CD ROM download This file may contain last minute information that did not makeit intothe manual The following sections contains both general and platform specific operating system configuration settings which again may be either mandatory M or optional O Scali recommends that you apply all the suggested configuration settings for your platform You must however apply all the mandatory M ones Scali System Guide Version 3 0 35 Chapter 4 Software installation 4 2 1 M Enable rsh root access from to all hosts From all hosts nodes including frontend have root rsh access to all other hosts n
138. ll skip the version test and install it anyway e Enable erase mode This will remove packages This switch may not be used in combination with sync mode D Enable dryrun mode No action taken but everything is checked d Enable debugging mode This will generate logfiles under tmp S Enablesequential mode The default is to run as much as possible in parallel F Specify fanout factor other than default for scash and scarcp v Enable verbose mode V Show version h Usage information It is possibleto use scapkg for handling general packages i e not Scali specific packages A general package may not have the string Sca as prefix in its filename Another restriction for general packages is that the first part of the filenameshould be identical with the name of the package E 2 Configuration The ScaPkg configuration file is located in opt scali etc ScaPkg conf The configuration file will be created as part ofthe SSP installation process and contains information for scapkg which packages belongs to which categories Each node will be assigned packages according to the category to node mappings specified in theconfi guration file Changes tothe configuration file can be made either manually with a text editor or as a result of running the SSP installation program The ScaPkg configuration file is divided into five logical sections which are described in sections E 2 1 through E 2 5 E 2 1 ScaPkg conf path to package
139. load oooccconnccincoconccinancncnonaninarnnorarar an nnnnnnrn ns 81 3D bar view on a large system 132 nodes cooooconnccccnnoccncononcnnonccnnnannc nana nononnno 82 2D history view showing two of four NOES ANEN 82 ERR GR GAN ER 3D System view on a 2D SCI evstem EE 83 TheAlarm WIM GOW dauere ee vk indiana el 84 The alarm editor window EN 85 The alarm SA 87 Adding a MEW aarm eege ea ea esate EE EES 87 ScaConf components in Context 95 Interconnect menu added to the Main Window A 96 High speed SCI network link status window ueenneersnennnennnnersnnnnn ernennen 97 Link status graphics explained ANEN 98 Example of submitting a ScaMPI job from Scali Universe to 4 nodes with 2 processes at each NOE A 124 Submit window with Reserve nodes only selected nennen nenn 125 Example of submitting a MPICH job from Scali Universe to 4 nodes OMG process Dt MOMS ccoo radica aaa da did 125 Options Advanced window for J ob Sub selon 126 Example Queue Status display in Scali Universe 127 Example Queue detail window in Scali Universe 128 Vertical horisontal and rectangle groupes of nodes running the same test application job in a 4x4 2D torus The wraparound connections of the tours are not showen ENNEN 147 Scali System Guide Version 3 0 Symbols 2D Bak ViCW O ern be Ee alten 81 KRAV NA 82 3D Bar EE 82 ID ELISEO Y WA EEN 83 EE RE ER A Alarm Edito WNA E 85 Alarmi log vl Wer eisian ar a SEA ge iu 87 Alarm WANGOW
140. lt nodel gt lt node2 gt tVh f lt frontend gt Use lt frontend as frontend gt default host specified in opt scali etc frontend name n lt nodel gt lt node2 gt Use specified nodes as node list default hosts specified in opt scali etc ScaSH scahosts p lt property file gt Specify node property file default file with an entry for every node lt node gt lt nodename gt lt arch gt lt mpi impl gt s lt server config file gt Specify pbs server configuration file default opt scali libexec pbs_server conf This script will configure OpenPBS for this node only The frontend name is an important property at all nodes hence the script must be run at all nodes with identical settings if the frontend name is changed The remaining parameters are defined at the frontend only hence the script is only required run at the frontend Running the script with default options will reconfigure ScaOPBS as it is configured after installation The node property and server configuration files may be modified the recommended method of modifying these files are to create a backup of the current configuration modify these backup files and reconfigure the current setting using the scaopbs config script with the modified files as input How to create backup files and the node property and server configuration files are described in the following sections 8 3 2 Backup of local configuration files Loca
141. m Guide Version 3 0 13 Chapter 1 Introduction 14 Scali System Guide Version 3 0 Chapter 2 Scali system overview This chapter gives an overview of the different components in Scali systems and software 2 1 Description of a Scali system A Scali system can be described with a simple equation standard qualified nodes high speed interconnect Scali s Software Platform Affordable Supercomputer Sincea Scali system is built using high volume low cost off the shelf nodes and a very powerful interconnect the price vs performance is extremely good Scali systems are available in the range from small two node systems to large HPC systems with hundreds of nodes The Scali software integrates the cluster nodes interconnect to a single system parallel computer All of the different Scali software components together constitutes the Scali Software Platform SSP The Scali Software Platform SSP may be divided in 3 domains of usage e Installation Scali software installation configuration and testing SSPinstall e Administration Maintenance Parallel software installation tool ScaPkg Configuration management of the SCI interconnect ScaConf Remote power switching ScaP owd Console management and broadcasting ScaCons Running OS commands in parallel on the cluster ScaSH System diagnostics ScaDiag Operational use Running OS commands in parallel on the cluster ScaSH Running MPI applications ScaMPI MP
142. m editor window to edit the selected alarm Requires an existing alarm Delete Deletes the selected alarm Activate Activates enables the selected alarm Deactivate Deactivates disables the selected alarm View Brings up the alarm view window which shows a dynamically updated log of the selected alarm Table 6 3 Alarm Main window control summary 6 4 2 The Alarm Editor Editing New alarm Match all of the following x Match any of the following Property Operator Threshold as More Fewer Description Email address Action Seconds before re trigger Wegen TI Wel Save Cancel Figure 6 11 The alarm editor window Thealarm editor is used both for defining new alarms and editing existing ones Hence itis activated by pressing either Add or Edit in thealarm main window The model we ve used to describe a condition that should set off an alarm is to combine a series of Scali System Guide Version 3 0 85 Chapter 6 System Monitoring boolean expressions using either AND or OR logical functions The boolean expressions are constructed by comparing a monitoring variable to a reference value by means a logic operator lt lt gt gt As a response to the alarm being triggered you can select between different actions to perform in addition to sending an E mail Beware the field called Seconds before re trigger This is default set to 1 which means that the alarm should not be triggered
143. mand will be used Example list nodeid status LO L1 scali2 11 scali2 13 Lists nodeid status and links states for scali2 11 and scali2 13 Example list status all List status for all nodes D 2 10 ConfTool gt fail The fail command turns off nodes in the sense that status become UNREACHABLE Usage fail lt list gt Example fail Fail nodes in the active list Example fail all Fail all nodes D 2 11 ConfTool gt reconnect The reconnect command does a reconnect to specified nodes Usage reconnect lt list gt Example select UNREACHABLE reconnect Attempts to reconnect all nodes with state UNREACHABLE 176 Scali System Guide Version 3 0 D 2 scaconftool command reference D 2 12 ConfTool gt log The log command is used to enable or disable server log messages in the tool Usage log enable disable Example log disable Disable log messages from the server Example log enable Enable log messages from the server D 2 13 ConfTool gt nodeid The nodeid command is used to set SCI nodel Ds Usage nodeid lt node name gt lt id gt lt id gt Theid can be either decimal octal leading 0 or hex leading Ox Without arguments nodeid will be set to default for all nodes Example nodeid Set nodeid to default for all nodes Example nodeid scali2 24 0x2400 Set nodeid for node scali2 24 to 0x2400 D 2 14 ConfTool gt reload Therdoad command does a reload of the SCI driver on
144. mestamp like this ConfTool gt 11 51 57 WARNING Lost connection to node daemon on scalil 21 11 52 05 NOTICE Automatic reroute routing type is SCA_ROUTE_MAXCY 11 52 05 NOTICE System rerouted with maxcy routing algorithm These messages generally helpful but if you find them annoying they can be turned off by the 10g command To disable the server log messages type log disable TO enable server log messages again type log enable You may also enableor disable log messages from start with the I command line option to scaconftool Scali System Guide Version 3 0 101 Chapter 7 Interconnect configuration 7 3 2 5 Batch mode Torun scaconftool in batch mode use the e option when starting the tool The commands you want scaconftool to run must be enclosed by double quotes and separated by The following batch job will restart the configuration daemon on nodes in state NO_DAEMON frontend scaconftool e daemon scaconfnd restart NO_DAEMON scaconftool will first executedaemon scaconfnd restart NO_DAEMON command This will launch a scash job to restart scaconfnd on all nodes in state NO DAEMON 7 3 2 6 On line help The help command provides an on line reference to scaconftool commands To get help on a specific command just type help lt command gt i e to get help on the list command type help list The help command gets its input from a plain text file located in opt scali doc ScaConf scaconftool txt
145. n example of a correctly formatted license request for a 4 node system SSP_3_0_1 version 1 94 license request Date Mon Feb 18 14 42 27 CET 2002 Resellers company N A Lic owner company Scali Scalable Linux Systems AS Recipient email user scali com Order reference no SC1234 Frontend machine scalil 11 Domain name scali com Current license Feature Version Expire Type clusteredge 3 000 1 mar 2002 Demo Host Hostid Cpu Kernel Distro scalil 11 0080c9876bla 2 2 2 19 7 0 8smp Red_Hat_7 0_Guinness scalil 11 0080c9876bla 2 2 2 19 7 0 8smp Red_Hat_7 0_Guinness scalil 12 00a0c9976d57 2 2 2 19 7 0 8smp Red_Hat_7 0_Guinness scalil 21 00a0c9876bc1 2 2 2 19 7 0 8smp Red_Hat_7 0_Guinness scalil 22 00a0c9876acb 2 2 2 19 7 0 8smp Red_Hat_7 0_Guinness 158 Scali System Guide Version 3 0 B 3 Thelicense file B 2 3 2 Generating the request during installation Fortunately the SSP install program will generate and format a license request as part of the installation You will be asked to provide the customer details while the necessary technical information is extracted automatically from the system Finally you will be given an option to send the license request right away The file containing the license request will be placed in tmp SSP_3_0_1 licrequest This is for SSP 3 0 1 the SSP version number will of course change to equal the current distribution B 2 3 3 Generating a new license request If you wish to gen
146. ncan 38 4 2 7 2 SUSE Linux NIS setup cnn ncnnn cnn nn ennnnnnn 38 4 2 7 3 Solaris NIS setup AEN 39 4 2 8 M Provide OpenGL for Scali Universe GUI nn 39 4 3 Scali software installation EN 39 4 31 SSP IECH Le EE A0 4 3 1 1 SSP install program options ocoonicnnnnncnccononinnoncnnncnnonanan cnn 40 4 3 2 Installation explained step by sten 41 43 251 Introducir padres 41 4 3 2 2 Installation Configuration uuursersnsrsnnennennnennnennnnnnnnennn nn 45 4 3 2 3 Software installation ersnnersnnnnnnnnnnnnnnnnennnen nennen nennen 54 4 3 2 4 Post installation Dn 56 4 3 2 5 Functional testing uurnnerssennnennnnnnnnnnnnnnnnnnnnnnn nennen nn 57 2 3 35SP Uninstallzu ass se Hansen RE 59 4 3 4 Uninstallation example A 59 Chapter 5 Scali Universe uuunnnnonnenonnunnnnnannunnnnunnnnunnunnnnnunnnnannunnnnunnnnnnnnnn nennen 61 LEMMER Ee Eech AA 61 5 2 Getting started with Scali Universe 62 5 2 PREP TEE 62 5 2 2 LASCAL ACI VE 62 52 3 RUNNING a iris ica CO aida A 63 5 2 4 Configuration of the Scali Universe GU 63 5 2 5 Logging IN toa SS aia Sn 66 4 Scali System Guide Version 3 0 5 3 Using Scal RE EE 67 5 321 Node SOCIO icon tp 67 5 3 2 MainWindow menu Oervlew crono cnn rr nn ran 68 5 3 3 Different system views the View mem 69 5 3 4 Running programs the RUN menu 70 5 3 4 1 Working with TUNSE S An 70 5 3 4 2 Running ScaMPI MPI programme 71 5 3 4 3 Running MPICH Drot ams 72 5
147. nds PBS commands are available in the directory opt scali contrib pbs bin man pages are available for most commands The most frequent used commands are qsub submit pbs job qstat show status of pbs batch jobs pbsnodes pbs node manipulation qdel delete pbs batch job qmgr pbs batch system manager Please see man pages for each command for detailed description and options 8 6 scasub The scasub script is a wrapper for the OpenPBS qsub program for easy submission of ScaMPI and MPICH parallel jobs Unlike other OpenPBS applications and scripts scasub is located in opt scali bin scasub The scasub script actually builds a job execution script and submits this script to OpenPBS using qsub scasub has the following options scasub lt params gt lt progname gt program options np lt np gt Total number of processes default 1 for non mpi programs 2 for mpi programs npn lt npn gt Number of processes pr node default 1 nodes lt nodes gt submit job at given nodes comma separated mpimon Submit parallel job using ScaMPI mpimon mpich Submit parallel job using MPICH mpirun v Verbose qsparams lt parameters gt Specify native queue system parameters mpiparams lt parameters gt Specify parameters to mpimon mpirun i lt file gt Use lt file gt as input for stdin a Start time for job CC YY MM DD hhmm 1 lt resource_list gt Specify res for job man pbs_resources
148. nen 152 A 2 Scali package file NAMING CONVENTION urnnneernnennnnnnnennnnnnnnnnnnnnn nn nnnn nn 152 A 3 Scali software packages OVervew cnn rn rn 153 A 4 Scali software daemon OVE VIEW cnn rro rr rr 155 A 5 Scali configuration Tleowervlew nr 156 Appendix B Scali Software Licensing eee 157 A o EE 157 B 2 Requesting licenses nunsenssnnnennnannnnnnnannnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnennnsnnnnennannnnsnnnn 157 B 2 1 Automated demo licenses generation from customer ID tag 157 B 2 2 Demo licenses general requests AEN 157 B 2 3 Permanent licenses nernnannsnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnennnnnnnsnnnn 158 B 2 3 1 Permanent license request format example ooocccnnccinicconccnonos 158 B 2 3 2 Generating the request during installation eee 159 B 2 3 3 Generating a new license request 159 B 2 374 aSt 1 GSOMM eege ine ann ek 159 8 Scali System Guide Version 3 0 B 3Thelicense EEN 159 B 3 1 Default OCIO cocinan it EEN 159 B 3 2 SCALM_LICENSE_FILE environment variable 160 B 3 3 License file lu EEN 160 B 3 4 Adding or updating features Uicemsesl AA 160 B 4 License installation uenesnnnnnnnennnannnnnnnnnnnnnnnnnnnnnnnnnnnnssnnnnennnnnnnnnennnannn nenn 161 B 4 1 License update installation with SSPinstall AAA 161 B 4 2 Manual license update installation nn 161 GN Een Le EE 162 B 6 Troubleshocting ersssrsensnannnnnnnnnnnnnnnnnnnnnnnnnnannnnsnnnnnnnnnnnsnnnnnnn
149. nica tion backbone for clusters ScaConfC Interconnect configuration clients scaconftool ScaConfN d Interconnect configuration node daemon ScaConfSd Interconnect configuration server daemon ScaCons Console server package ScaDesk Scali Universe graphical interface SaDiag Diagnostic tools ScaDiagSC Diagnostic tools for SCI systems ScaEnv Environment settings ScaE xecd Remote execution daemon ScaF mpich Free MPICH binary packaging ScaF PDisp Front panel display driver LEDs ScaF PMB2 Free PMB2 binary packaging SalP Internet Protocol driver for SCI Scal Padap ScalP adaption kit ScaLM License manager ScaMAC Media driver for SCI ScaMACadap MAC driver adaption kit Table 10 5 Overview of Scali software packages distributed with the SSP Scali System Guide Version 3 0 153 Appendix A Scali software details Package module name Description ScaMACddk MAC driver development kit ScaM ond Monitoring server daemon ScaMPE MPE packaging ScaMPI High performance MPI implementation ScaMPICHt MPICH test packages ScaM PI tst ScaM PI test packages ScaOPBS Open PBS binary packaging ScaPkg Cluster software package installation tool ScaPowd Remote power switch control daemon ScaSCl SCI driver ScaSCl adap ScaSCI adaption kit ScaSClddk ScaSCl driver development kit ScaSensor Sensor configuration package ScaSH Parallel shell tools ScaSH Meme Compatibility library f
150. nnection list example for cabling an 8 node SCI ring nennen 27 Example cabling of one vertical ring in a 4x4 system x4 1 4 eccess 28 Example cabling of one horizontal ring in a 4x4 system y J1 4 29 Example cabling of LO on a 3x3x2 torus 231 21 x41 2 3 wees 30 Example cabling of L1 on a 3x3x2 torus 231 21 y 1 2 3 31 Example cabling of L2 on a 3x3x2 torus x4 1 2 3 yJ1 23 31 System M ainWindow node symbols AAA 68 System MainWindow menu overview enneennessennnnnnnnnennnnnnnnnnnnnnnn ernennen nn 68 Main window Run menu items evplalne AEN 70 Management menu Opti ONS nennen nn 75 Software menu OPtions uueensennnnennnnnnnnnnnnnnsnnnnnnnnnnnnnsnnnnnnnnnnnnnsnnnnnnnnnannnn na 75 The Status ME nee ee E ee 79 Monitoring view window options menu 80 Alarm Main window control summer 85 Alarm Editor control Oovervlew ernennen nn 86 Default Performance monitoring variables ANN 89 Default System monitoring variable 90 The Interconnect menu items A 97 SCI link status window POP UP MENU OPTIONS nunnneernnnnsennnnnennnnnnnnnnnnnennnn nn nn 99 SECO le EE 103 Options in the file SCASH ont 137 Software installation roublechoctimg AAA 140 LED behaviour on D33x Garde AAA 141 SOMES Cl ge ee 142 le Kiebeee De Eu e alle TE 143 Overview of Scali software packages distributed with the SSP 153 Scali daer OVELVI iii 155 The confi
151. nnnnnnnnnennnannn nenn 162 B 6 1 Error Invalid FEATURE line nn 162 B 6 2 Error Invalid version X X NN 162 B 6 3 Error FEATURE has evpirel EEN 162 B 6 4 Error Host ID is not found nersessennsnnersnnnnnnnnnnnnnnnnnn nn nnnnnnennn nn 162 B 6 5 Missing license file uunnnnnnnnssnnnnnnnnnnnnnnnnnnannnnsnnnnonnnnnnnnnnnnnnnnnnnn nn 163 B 6 6 Missing license software ScaLM package AA 163 B 7 Older versions with ELE lm 163 Appendix C SCI Utility Programs nnuaunnuunnnnonnunonnunnnnunnunnnnunnnnnunnunnnnunnnnnnnnnnnnnannnn 165 C 1 SCI hardware status Programs EN 165 e We Tei A eaea E EEA ee Eaa ehad ai aaan aaan aE onin TEE oh Pinin PiN en 165 CLZ SEGMONITOF EE 166 CFS SA ENSO RETTEN 166 CEFA Lee EE 166 EE ee EE 166 E Ee Ile EE 167 Gale Ee Le e DEE 168 CE8 SCUIGCONG soniri einga eege a a a aaao Ed Ee 168 E Sarl le DT 168 C2110 EE 169 GaP TU tee EE 169 Appendix D ScaConf Reference unannenonnunnnnunnnnnnnunnnunnnnunnnnunnnnnannunnnnunnnnannnn nenn 171 D 1 Installation and package dependencies uk 171 D 2 scaconftool command reerence EN 171 D 2 1 ConfT OO ACO aE o 172 D 2 2 ConfTool gt daemon EE 172 D 23 C0NT Re e EK ET 172 D 2 4 Conf Tool gt e EEN 172 D 2 5 ConfT ool Seto EE 173 D 2 6 Contl 0ol 1NTO s EE 174 D 2 7 Cont 0ol gt Sel EE 174 Scali System Guide Version 3 0 9 D 29Conftodzunsedet AANEREN 174 D 2 9 Conf Tool 3lst ae er na 174 D 2 10 C0NfT 001 gt talud nl 176 D 2 11 ConfTool gt reconne
152. node categories _ Please wait while retrieving information from nodes The following node categories have been defined scalil 11 sci_frontend eth_frontend sci_node eth_node scalil 12 sci_node eth_node scalil 21 sci_node eth_node scalil 22 sci_node eth_node Q Do you accept the above configuration y y If you install a product which includes extended hardware monitoring features this will be automatically set up at this stage given that qualified hardware for such purpose is used Monitoring of CPU temperatures and cooling fan speeds are typical examples of such features Specifying node hardware You may benefit from extended system monitoring when using qualified node hardware from certain vendors If your hardware is not listed just use the default value Hardware not listed It is assumed that all hosts are using the same hardware Scali System Guide Version 3 0 47 Chapter 4 Software installation Hardware not listed Dell PowerEdge 1550 Dell PowerEdge 1650 Dell PowerEdge 2650 SuperMicro SuperServer 6010H SuperMicro SuperServer 6022C SuperMicro SuperServer 6012P 6 ANU PB WNHEe Q Please select what hardware is used 1 1 Hardware not listed Q Do you accept the above configuration y Specifying software to install a Selecting console switch _ 2 If you have the seri
153. node selection read section 7 3 2 3 lt list gt is to be replaced by keyword all meaning all nodes node state meaning all nodes in this state e g OK NO _ DAEMON etc list of node names separated by blanks meaning all nodes explicitly listed left blank meaning all nodes in internal list routing partition number eg 1 meaning all nodes with this routing partition number Scali System Guide Version 3 0 171 Appendix D ScaConf Reference D 2 1 ConfTool gt console The console command is used to connect to the console on a specified node This command only works if a console server is available and the ScaCons package is installed correctly The lt console options gt are passed as is to the program opt scali bin console Se man console for an explanation of the console program Usage console lt console options gt Example console nodel scaconftool will try to connect to console on nodel D 2 2 ConfTool gt daemon The daemon command is used to start or stop daemons on nodes Usage daemon lt daemon name gt lt action gt lt list gt Example daemon scacomd restart UNREACHABLE Restart the communication daemon on all nodes in state UNREACHABLE Example daemon scaconfnd restart NO_DAEMON Restart the configuration daemon on all nodes with state NO_ DAEMON D 2 3 ConfTool gt fix The fix command attempts to bring the nodes in thelist intostateOK This is achieved by restarting daemons that does not r
154. nodes but it takes the same arguments as the command snmpget see man snmpcmd part of the ucb snmp tools package for a full description To add a class for a default SNMP daemon you need to add a line like this class stdsnmp internal s s snmpaget v 1 p 161 s public s The two ss are replaced with host name and OID respectively 6 7 2 2 Editing hwgroup classes The second section of interest is the hwgroup class with is actually optional If the SNMP OIDs you want to monitor only appear on a selection of you nodes ie you ve a heterogeneous cluster you can define them into a group like this hwgroup name members hwgroup myset node23 node44 Where we put the two nodes into a group 92 Scali System Guide Version 3 0 6 7 Defining new monitoring variables 6 7 2 3 Adding the OIDs The third section in where you define the OI Ds you need to retrieve oid hwgroup name size class format oid oid cpuidle integer scasnmp d enterprises ucdavis systemStats ssCpuIdle 0 oid cpusystem integer scasnmp d enterprises ucdavis systemStats ssCpuSystem 0 oid cpuuser integer scasnmp d enterprises ucdavis systemStats ssCpuUser 0 If the hwgroup is empty like here it just means that you will monitor on all nodes in etc scali etc ScaSH scahosts The name is mapped to a variable in the next section The size is almost always integer but occasionally strings are used to measure up down values The class refers toa
155. nodes to check if they are ok and reconnect them if they are REPAIR_INTERVAL 3 300 Server will wait for this many seconds before it will try to reconnect nodes that is reported as UNREACHABLE Recommended value 10 AUTO_REROUTE ON OF F Server will automatically reroute cluster if change of state in one or more nodes is detected AUTO_RECONNECT ON OF F Nodes will be reconnected to the cluster auto matically when they come up if this option is ON AUTO_NODE_ID ON OF F Automatic checking and setting of SCI nodeid on or off ROUTING TYPE SCA_ROUTE_DEFAULT Default routing type for the topology Use info routalg to get a list of all routing type availa ble for your topology Table 10 9 Available server options Default values are marked with Scali System Guide Version 3 0 173 Appendix D ScaConf Reference OPTION VALUE DESCRIPTION REROUTE_DELAY 0 3 10 Seconds to wait after a reroute is issued before new status is sampled SAVE lt none gt Save the options to file This will cause server to start with the saved settings if restarted RESTORE lt none gt Restore factory defaults for all options Table 10 9 Available server options Default values are marked with D 2 6 ConfTool gt info The info command prints information on specified topics Usage info lt topic gt lt topic gt lt topic gt topology prints the topology of the machine rou
156. note that only the Console option is available to ordinary users all other items requires root privileges Furthermore Console and Power functionality requires additional hardware and will be greyed out if no proper hardware is found The Queue Enforcement option will only be visible if OpenPBS integration has been installed Menu Item Description Console Opens a console window on the selected nodes Reboot Initiates software reboot at all selected node confirmation is needed Shutdown nitiate software shutdown for all selected nodes confirmation in needed Power Opens the Power submenu This contains options to perform a hardware power Off power On or power Cycle on all selected nodes Queue enforcement Opens the Queue enforcement submenu to set the queue enforcement status Available options are Enabled or Disabled Table 5 4 Management menu options 5 3 6 Software installation Software menu When logged into the Scali system as root you will get access to the Software menu This is really the Scali Universe graphical interface for the underlying ScaPkg software installation system described in detail in appendix E From this software menu you may install and maintain software packages across all nodes in a Scali System Table 5 5 gives a summary of the Software menu options Menu Item Description Install Opens the software installation window Table 5 5 Software menu options
157. nt to install these tools please follow the instructions in the file opt scali contrib pbs doc README at the frontend node 8 4 Example of ScaOPBS usage The following section gives an example of submitting a small ScaMPI and MPICH program and checking the status of the execution Login as a standard user root is not allowed to submit jobs to PBS Tosubmit a ScaMPI job type opt scali bin scasub mpimon np 2 opt scali examples bin producer Two nodes will be allocated by OpenPBS for the job The start up script created by scasub will beinitiated by OpenPBS at the first listed node in the list of allocated nodes for the job The start up script will determine the list of nodes allocated for the job from a set of environment variables and will start the ScaM PI application at these nodes using mpimon To check status use opt scali contrib pbs bin qstat Two output files should be generated in current directory producer o1 stdout from application should list TEST COMPLETE producer el stderr from application should be empty Similarly to submit a MPICH job use opt scali bin scasub mpich np 2 opt scali contrib mpich examples cpi To show nodes available for pbs use the command opt scali contrib pbs bin pbsnodes a ScaOPBS will configure pbs with a single queue To show queue status issue opt scali contrib pbs bin qstat Q Scali System Guide Version 3 0 121 Chapter 8 Batch system ScaOPBS 8 5 Open PBS comma
158. nt to use If you want tousethe 3D monitoring options you must alsohavean OpenGL 1 2 compatiblelibrary installed On Linux systems this is provided by the Mesa version 3 2 or newer You should also consider a graphics card which provides hardware acceleration of OpenGL under your operating environment 5 2 2 Installation The Scali Universe GUI is installed on the frontend node as part of the SSP installation but in most cases the frontend is not your local workstation so you would want to install it locally too After SSP installation you will find the scaDesx software package in the repository opt scali pkg lt platform gt on the frontend node The ScaDesk name is a relic from the time the Scali Universe GUI was called the Scali Desktop To install the package on Linux systems use the rpm package manager program like this rpm ivh ScaDesk Linux2 i86pc lt version gt rpm 62 Scali System Guide Version 3 0 5 2 Getting started with Scali Universe For instructions on how to install the Scali Universe GUI on other platforms please consult the text file instructions in opt scali doc ScaDesk INSTALL 5 2 3 Running Due to the heavy use of graphic presentations we recommend that you run Scali Universe on your local workstation To start Scali Universe on Unix systems type opt scali bin scadesktop On Windows systems use the Run option from the start menu and select or double click the file C Program Files ScaDes
159. ntrib bin pmandel i opt scali contrp bin cool points Browse MPI monitor options display pe 25 0 0 Processes per node 1 region region region region region region region region Starting opt scali contrib bin pmandel i opt scali contrib bin cool points 1 rmin 0 80000000000000004 imin 0 05000000000000000 rmax 0 699 rmin 2 00000000000000000 imin 2 00000000000000000 rmax 2 000 rmin 0 78270040000000007 imin 0 14599959999999998 rmax 0 780 rmin 0 74153488005713408 imin 0 10702688557231872 rmax 0 741 rmin 0 36705586003000001 imin 0 63196462272000009 rmax 0 367 rmin 0 27062999999999998 imin 0 62687999999999999 rmax 0 263 rmin 0 25135043699999998 imin 0 00008203000000000 rmax 0 25135 rmin 0 5000000000000000 imin 0 09500000000000000 rmax 0 739 Figure 5 11 Scali MPI monitor window with output from an MPI program From the File menu in Scali MPI monitor you may open and save runsets 5 3 4 3 Running MPICH programs IftheScali MPICH distribution is installed ScaF mpich you will also get a graphical launcher window for MPICH applications Select MPICH program from the Run menu to open the window Except for the missing field for MPI monitor options the MPICH launcher behaves exactly like the ScaMPI MPI monitor window described in the previous section The MPICH launcher supports runsets 72 Scali System Guide Version 3 0 5 3 Using Scali Universe 5 3 4 4 Runnin
160. odes including frontend Please do the following steps on every node Insert frontend name and all nodenames into the HoME rhosts file as root Enable rsh commands as root from the frontend to all nodes RH 7 x Add rsh to the file etc securetty Enable the rsh service by doing chkconfig rsh on 4 2 2 O Enable rsh access from to all hosts for common users Enable rsh access to all hosts nodes for all non root users Please do on all nodes including frontend Add all node names including frontend name in the file etc hosts equiv 4 2 3 O Enable rlogin and login as root from front end This step is optional but will ease administration of the nodes Notethat this will have implications on the access security to the nodes in the cluster If access security is an important issue we will suggest to separate the cluster from the rest of the network by a firewall or by using the frontend as a gateway to the duster two NICs On Linux enable rlogin as root from the frontend to all nodes RH 6 x Edit etc pam d rlogin remove the pam_securetty check or add hosts to pam authentication RH 7 x Add rlogin to the file etc securetty Enable the rlogin service by doing chkconfig rlogin on SuSE Run yast choose System administration Change configuration file and set the entry ROOT_LOGIN_REMOTE to yes On Linux enable login as root using telnet and enable login as root over serial console RH 6 x Edit etc pam d login
161. odes as long as the node is part of at least one working SCI ringlet Therouting might result in some disabled links Operational nodes where one or more rings are broken will have their link disabled in the direction of the broken ring 108 Scali System Guide Version 3 0 ClusterE dge 7 4 Routing 7 4 3 3D torus topology Three dimensional torus topology is detected by the topology rop_trorus_3p line in the configuration file opt scali etc ScaConf conf na 3D torus each nodeis connected to three SCI ringlets onein each dimension of the torus The following routing types are available in a 3D torus e Dimensional routing e C3 Close Commutative Connection routing 7 4 3 1 Dimensional routing in a 3D torus Dimensional routing is available for a 3D torus but it is less fault tolerant than C3 routing When using dimensional routing packets are routed along link O first then along link 1 and then along link 2 You may specify another order of dimensions as a parameter to the reroute command The command ConfTool gt reroute dimensional order 210 will route link 2 first then link 1 and link O last If a node or a ring fails the number of nodes in the main routing partition partition number 1 see 7 4 4 depends on the order of the dimensions in the routing algorithm Other nodes not part of the main partition will not have SCI connectivity to nodes in the main partition These node may route other SCI traffic though 7 4
162. ogram The following flags are parameters to vary the behaviour of the run_permutated_bidirect program u Unidirectional mode only b Bidirectional mode only q Permutated bidirect quick mode work reduced with a factor of 10 Q Permutated bidirect very quick mode work reduced with a factor of 100 d Dryrun nothing really done These flags have no effect without the c bidirect flag With the c PMB flag the Pallas benchmark is used as test application This is default if no other test application is specified with the c option By default all output from the test application will be written to dev null If you wish to save the output from the test runs you may use the o flag With no parameter the output from the different runs will be written to files in the current directory The file name will contain the name of the nodes it was run on With a parameter to the o option you may give the path to a directory with write access where you want the files to be put Between each test run sci_hwdiag will print the value of non zero counters in the SCI driver Link errors and Times handler will be listed in a summary at the end Other non zero counters will only be listed during the run Please be aware that the ScaSCl software detects and handles all these incidents hence data transfer is always safe For more information on how to interpret the results please take a look at opt scali doc ScaDiagSC GUIDE Various
163. on includes setting the server attributes and establishing queues and their attributes 8 3 4 1 Listing the current server configuration Tolist the current server configuration type opt scali contrib pbs bin qmgr Max open servers 4 Omgr print server The example output from qmgr below shows a the default server configuration for ScaOPBS with a single queue called scali_exec enabled and running Scali System Guide Version 3 0 117 Chapter 8 Batch system ScaOPBS Create and define queue scali_exec create queue scali_exec set queue scali_exec queue_type Execution set queue scali_exec enabled True set queue scali_exec started True Set server attributes set server scheduling True set server default_queue scali_exec set server log_events 511 set server mail_from adm set server query other_jobs True set server scheduler_iteration 600 The example above shows the default server configuration for ScaOPBS with a single queue called scali_exec enabled and running To modify the server configuration change the server configuration file created by scaopbs config b lt backup dir gt and then reconfigure ScaOPBS using scaopbs config s lt server config file gt Please see section 3 5 at page 26 in the Portable Batch System Administrator Guide available at thefrontend as opt scali contrib pbs doc pbs_admin_guide pdf for further description of the server configuration 8 3 4 2
164. onitoring data from the SNMP daemons to its clients scasnmpd node Scali SNMP daemon this daemon provides the monitoring system with information Scali System Guide Version 3 0 Table 10 6 Scali daemon overview 155 Appendix A Scali software details A 5 Scali configuration file overview Configuration files for Scali tools and programs will always be stored in opt scali etc Files with the extension example are example configuration files installed by some packages The table below gives a summary of configuration files file ScaConf conf description Configuration file for ScaConf the cluster configuration system ScaConf nodeidmap If present this file ensures node names are mapped to specific node d s ScaDesk conf Default configuration file for Scali Desktop will be cop ed as basis for the personalised user profile in SHOME scali ScaMond conf Scali monitoring daemon configuration file ScaPkg conf Scali software installation system configuration file Lists packages node categories and the relation between the two ScaPkg adminfile Solairs specific configuration file for pkgadd ScaSH conf ScaSH parallel shell tools configuration file ScaSH scahosts Host list for ScaSH parallel shell tools conserver conf ScaCons console server configuration file conserver passwd Password file for the ScaCons console server Table 10 7
165. onnect cable guides on the rear figure 2 5 A LED panel on the front door of the enclosure may be configured to show a monitoring parameter like the CPUs activity Figure 2 4 A 4x8 Scali system boxed in Scali cabinets Figure 2 5 Open front and rear side of a Scali system in a Scali cabinet 18 Scali System Guide Version 3 0 Chapter 3 Hardware installation Aimed at OEMs VARs and kit builders this chapter explains how to organize and configure your node hardware and how to install and connect both Ethernet and SCI interconnects to prepare a cluster system for SSP installation The main focus is on the high speed interconnect installation and cabling since this has proved to be the most challenging part If you own a turn key Scali system TeraRack this chapter can be viewed as supplementary information 3 1 Hardware organization The most practical way to organize a cluster of interconnected computers is to stack them in a suitable system cabinet or rack With large numbers of nodes a 2 dimensional 2D arrangement is usually required For rackmount nodes any 19 rack of reasonable quality could be used just make sure airflow cooling is sufficient F or nodes with deskside minitower cabinets Scali recommends using a Scali system cabinet or another boxed solution known to serve the requirements of cluster hardware management 3 1 1 Node naming scheme We recommend that you select a naming scheme where the node names are
166. ontend Categories sci_frontend eth_frontend sci_node eth_node smp_frontend smp_node The following categories have been selected sci_frontend eth_frontend sci_node Scali System Guide Version 3 0 53 Chapter 4 Software installation eth_node smp_frontend smp_node Q Do you accept the above selection y y The following node categories have been defined scalil 11 sci_frontend eth_frontend sci_node eth_node smp_frontend smp_node scalil 12 sci_node eth_node scalil 21 sci_node eth_node scalil 22 sci_node eth_node Q Do you accept the above configuration yl y 4 3 2 3 Software installation During the Software installation part most of the Scali software will be distributed to the nodes and installed This is also true for the frontend Normally there will be no need for user interaction during this part of the installation Software installation EE EE Installing packages e Wait Checking accessibility of package repositories Checking availability on hosts chosen to be updated please wait This program installs software on specified or default hosts version 1 52 repository opt scali repository Checking configuration Cleaning up from previous runs on all hosts Find platforms on hosts Find out on each host which packages are needed Find out which packages we have for the platforms detected Removing invalid platforms Removing invalid hosts due to invalidated platforms Find out which packages whic
167. ool Starting and stopping daemons from the tool is only available in admin mode 7 3 11 Setting server options From scaconftool you may control configuration server options with the setopt and getopt commands Please refer to the complete list of available options and their default values in Table 10 9 on page 173 To change an option use the setopt command as shown below disables automatic rerouting ConfTool gt setopt AUTO_REROUTE OFF Some server options are of type ACTION This means that rather than setting a parameter they will cause the configuration server to perform an action The action setopt SAVE will save your current server options settings to the file opt scali etc ScaConf opt from which they will be restored when the server is restarted The setopt RESTORE action will set the value of all server options back to their factory default 106 Scali System Guide Version 3 0 ClusterE dge 7 4 Routing 7 4 Routing In order for the nodes to communicate over the SCI network there must be a routing path between the nodes The route a packet will travel from a source node toa destination node is decided by a routing algorithm ScaConf provides several different routing algorithms depending on topology and license type You may check out the available routing types for your system in the Interconnect gt Set routing menu see figure 7 2 in Scali Universe or the info routealg command from scaconftool Routing is set autom
168. opied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied copied to to to to to to to to to to to to to to to to to to to to to to to to to to to to HG H HG GG GHG Ge OH GG Lu GHG W H GHG packages have been copied Scali System Guide Version 3 0 host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s host s 55 Chapter 4 Software installation Get results from hosts Poll for log files on remote hosts Checking log files scalil 11 No errors scalil 12 No errors scalil 21 No errors scalil 22 No errors Cleaning up after us and exiting 4 3 2 4 Post installation fix ThePost installation fix section will take care of configureing parts of the software which has to be configured after the software has been installed The first example shows a Clusteredge installation Note that the configuration of the SCI network is automatically taken care of When installing Open PBS some of the configuration has to be performed after installation of the software Post installation fix EE Specify SCI network 2 8 Wait Pleas
169. or Cray SGI ShMem ScaSI SCI SISCI interface to SCI driver ScaSNMPd SNMP daemon ScaSNMPt SNMP text tools ScaSSP the Scali Software Platform main package con tains install programs and documentation Table 10 5 Overview of Scali software packages distributed with the SSP 154 Scali System Guide Version 3 0 A 4 Scali software daemon overview A 4 Scali software daemon overview After a successful installation of the SSP there will be quite a few new daemons running on your system Common to all is that they can be started stopped restared using the standardised scripts located in opt scali init d using the following command opt scali init d lt daemon gt start stop restart info Note that if one of your nodes doubles as front end it will run all daemons daemon location description mpid node mpi application launcher please refer ScaMsPI Users Guide scacomd node Scali communication daemon provides commu nication services for other Scali software scaconfsd front end Scali configuration server daemon handles interconnect configuration and remote power switching scaconfnd node Scali configuration node daemon this is the interface between the configuration system and the node interconnect hardware scaexecd front end Scali remote execution daemon provides secure remote execution scamond front end Scali monitoring server daemon provides com pression and filtering of m
170. ossibility of running the uninstall program from the directory where the Scali software once were untarred and unzipped tmp SSP uninstall or run the uninstall program fromtheCD mnt cdrom uninstall 4 3 4 Uninstallation example When starting the uninstall program you will be asked to confirm that you really want to start uninstalling the SSP There will also be a second question where you can choose to keep the repository or not The repository is where all the software packages in the SSP are kept During installation the installation program will distribute and install single software packages from the repository If you are going to uninstall the software on the system and do a reinstallation using the same software versions you will typically choose to keep the repository root scalil 11 root opt scali sbin SSPuninstall Checking installation EE Welcome to the Scali Software Platform SSP uninstallation program This program will guide you through the uninstallation procedure Whenever your input is required you will see a line starting with O D The default option is given in square braces and is chosen if you press return If you experience problems during uninstallation please contact support scali com Wulfkit customers please use the e mail address wulfkit support dolphinics com Scali System Guide Version 3 0 59 Chapter 4 Software installation 60 Checking permissions 7 7 7 OK Find
171. our cluster Select opt scali sbin SSPinstall After a while the install script will ask gt Checking upgradeability I u It seems like we are able to upgrade your existing installation Q Please confirm if you would like to upgrade y y Select yes to upgrade after a while you will be presented with the questions from Installing OpenPBS during SSP installation on page 114 8 2 3 Running the ScaOPBS installation tests manually You may run the ScaOPBS installation tests manually by using the command opt scali libexec scaopbs config t lt testuser gt Testing OpenPBS installation gt 7 7 Testing OpenPBS with mpimon OK Testing OpenPBS with mpich eebe OK Please provide a testuser name that is properly defined at all nodes 8 3 OpenPBS configuration 8 3 1 Manual reconfiguration of ScaOPBS The ScaOPBS configuration is based on information from the SSP packages ScaSH opt scali etc ScaSH scahosts and SSP install opt scali etc frontend name Scali System Guide Version 3 0 115 Chapter 8 Batch system ScaOPBS The default configuration will do for most systems but if the configuration for these packages is changed or if you would like to manually override this configuration you may use the scaopbs config script This script has the following syntax opt scali libexec scaopbs config h Usage scaopbs config f lt frontend gt n
172. ous from the invoice that came with the product If in doubt just contact Scalit Extended support contracts or pay per request are also available When requesting support please state your name and company 10 6 2 How to report the problem It will save both your time and ours if you do the following before requesting support e Read the documentation again Quite often the problems can be resolved from the relevant chapters in the manuals or FAQs e Collect version information This includes version information for all involved software Scali SW compilers operating system Versions of HW SCI cards computer architecture and chipset versions BX LX NX e Isolate your problem to a small test case if applicable The shorter the better e Record the sequence of events causing the problem 1 Wulfkit customers should contact their reseller or Dolphin Interconnect Solutions Scali System Guide Version 3 0 149 Chapter 10 Troubleshooting You can reach Scali s technical support using either E mail support scali com Preferred Phone 47 2262 8950 Fax 47 2262 8951 Comments regarding beta software may be sent to E mail beta support scali com Support for Wulfkits are handled by Dolphin Interconnect Solutions Please contact wulfkit support dolphinics com 10 6 3 Other Feedback Scali welcomes any suggestions improvements feedback or bug reports concerning this Scali System Guide or the hardware and softwa
173. pings nodes over SCI interconnect It uses scash Usage sciping lt list gt Example sciping nodel node2 node3 send sciping from the listed nodes to the listed nodes D 2 21 ConfTool gt help The help and command print a help message Usage help lt command gt Example help restart Scali System Guide Version 3 0 179 Appendix D ScaConf Reference Print information about the restart command Example help Print a list of all available commands D 2 22 ConfTool gt quit The quit command exits the command tool Usage quit D 2 23 ConfTool gt version The version command prints the version of scaconftool Usage version 180 Scali System Guide Version 3 0 Appendix E ScaPkg Scali Software installation program TheScali software installation program scapkg enables the maintenance of software packages on several nodes across different platforms in parallel It will automatically compare versions of installed packages with versions on packages in a repository and install remove or upgrade the software packages on a selected set of nodes accordingly E 1 Using scapkg Before installing packages you may use the dryrun option D to check what the command actually does Use h to list all scapkg install options As an example we will show how to install aScaMPI package from the file scamPI Linux2 i86pc 1 8 5 rpm located in the repository home software Here we recognize the format module os
174. pplies only to workstations actually running thegraphical application remote or local e Linux Mesa has been tested e Microsoft MS OpenGL has been tested e SunOS Solaris Mesa and SunOpenGL has been tested on Sparc 4 3 Scali software installation When the operating system has been properly configured you may install your Scali software products Note that you will need to obtain a valid license for your product in order to complete the installation The recommended procedure of doing this is to first use your customer ID tag to obtain a demo license from demolicense scali com see Requesting licenses on page 157 for details and use this demo license for the initial installation Then you should request the permanent license Scali System Guide Version 3 0 39 Chapter 4 Software installation 4 3 1 SSP installation If the Scali software comes on a CD start the installation as root by typing the following mnt cdrom install If the Scali software has been downloaded from Scali s web site unzip and untar the tarball in a temporary directory on thefrontend machine Start the installation as root by typing the following in this example we assume the software has been unzipped and untarred in the tmp SSP directory tmp SSP install If for any reason the installation has to be terminated before it is finished use Ctrl c toterminate When the installation is to be completed run as root on the frontend the
175. previous mentioned commands or run the main installation program directly by typing opt scali sbin SSPinstall 4 3 1 1 SSP install program options When running the installation program by one of the methodes mentioned above the actual program running iS opt scali sbin SSPinstall Usage SSPinstall ciftvlmVuh c start directly in configuration stage i start directly in installation stage t run system functional tests only v run system verification tests only 1 create permanent license request V print SSP version m lt media path gt path to media packages f skip frontend check u upgrade license file h show this help message If the configuration went OK but there were problems with the package installation you may restart from the package installation again using the i option 40 Scali System Guide Version 3 0 4 3 Scali software installation If you arerunning with a demolicense and want to create a permanent license request use the 1 option And if you have obtained a new license demo or permanent you may use the u option to distribute and install the license on the entire cluster system 4 3 2 Installation explained step by step Theinstallation program will guide you through theinstallation process which installs the selected Scali software on the whole cluster The installation program will detect if you are performing a clean installation or an upgrade The installation process is
176. pserv start e cdto var ypandrun make e run sh etc rc d init d ypbind start On the NIS clients all other nodes e run authconfig and specify NIS domainname and request server via broadcast 4 2 7 2 SuSE Linux NIS setup On the NIS master usually the frontend e Run yast choose System administration Change configuration file and set YP_DOMAINNAME to NIS domainname eset YP_SERVER to correct IP address e Set START_YPSERV to yes eset START_YPBIND to yes 38 Scali System Guide Version 3 0 4 3 Scali software installation On NIS dients all other nodes e Run yast choose System administration Change configuration file and eset YP_DOMAINNAME to NIS domainname eset yP_SERVER to correct IP address 4 2 7 3 Solaris NIS setup On the NIS master usually the frontend e set NIS domainname run domainname lt yourdomain gt e EITHER have these files under etc timezone netgroup bootparams OR edit var yp Makefile to not use these maps e run ypinit m e edit etc nsswitch conf e edit var yp securenets e run usr lib netsvc yp ypstop e run usr lib netsvc yp ypstart On the NIS clients all other nodes e set NIS domainname run domainname lt yourdomain gt e run ypinit c e run usr lib netsvc yp ypstop e run usr lib netsvc yp ypstart 4 2 8 M Provide OpenGL for Scali Universe GUI The Scali Universe GUI requires OpenGL library for some of its monitoring views Notethat this a
177. pter giving ethO on the frontend is used for the connection to the local LAN while the second Ethernet port adapter giving eth1 is connected to the Ethernet switch H UB of the private network For all processing nodes the primary Ethernet port adapter eth0 should be used Please refer to figure 3 2 for a diagram 3 3 3 Using channel bonding Channel bonding with fast Ethernet adapters is a low cost solution to increase the throughput of systems which lacks a high performance interconnect like SCI For the moderate cost of one additional Ethernet adapter per node and one additional Ethernet switch you may double the bandwidth of the Ethernet latency however will remain unchanged Some cluster enabled nodes even comes with dual Ethernet controllers on board which makes this option very attractive Note that the use of channel bonding implies the use of a private network Figure 3 3 shows one cabling diagram based on the recommendations from 3 3 2 hi ode et ethO Node 2 Fe etho eth1 gt Node 3 n 4 EE ethO Frontend Ces Node eth2 eth ethO eth Node n vv y vv Ethernet switch HUB Ethernet switch 1 Ethernet switch 2 Site LAN Scali system private network Figure 3 3 Scali system with channel bonding 22 Scali System Guide Version 3 0 3 4 Dolphin SCI interconnect installation 3 4 Dolphin SCI interconnect instal
178. r maintenance A cluster has a lot of interconnect cabling try to keep the cables as short as possible 3 2 Node BIOS preparation In order to prepare the nodes for cluster use you will have to check a number of BIOS settings as described in the following sections 3 2 1 Set MP level Ina multiprocessor nodetheBIOS sometimes has an option of configuring theso called MP level This specifies the protocol version used by the CPUs to interoperate Possible values are usually 1 1 and 1 4 We recommend that you set this option to 1 4 3 2 2 Interrupt level mapping Some BI OSes have the option of enabling interrupt level mapping If possibleturn this feature on to reduce the possibility of conflicts with other devices 3 2 3 Power on startup If your system includes external power switches it is important that you configure the power section in the BIOS so that the node is automatically started when power is applied to the node 20 Scali System Guide Version 3 0 3 3 Ethernet setup 3 2 4 Enable ECC Make sureE CC is enabled in BIOS this is not always the default setting The use of memory with ECC capabilities is essential when working with systems with tens to hundreds of gigabytes of memory 3 2 5 BIOS redirection to serial port Some BI OSes now provides the possibility to redirect BIOS output to the serial port If your system has a serial console switch and your BIOS supports redirection we recommend that you enable t
179. r or a symbolic name sciemsg see C 1 3 gives a description of the cause of the error Some common SCI error messages are explained here Scali System Guide Version 3 0 141 Chapter 10 Troubleshooting SCI error type Description ICMS_NO_RESPONSE Reporting of this message type when attempting communica tion with an existing node is an indication that something is wrong because the remote node did not respond Either the physical connection is broken there is no route back from the destination or the node is down or unable to respond ICMS_CONNECTION_REFUSED This message type is returned when the connection request was refused by the memory owner either becausetheresource remote memory does not exist or it is not currently offered to new connections ICMS_OUTBOUND_MAP_FULL This error type indicates that not enough SCI map resources are availableto satisfy the application need That isthe local SCI adapter address translation table ATT is full and there is no room for more remote memory mappings This is unre lated to the state of the interconnect except of course that processes may hang and block resources because of some interconnect problem See wrong jumper settings in Table Table 10 4 ICMS_FAILURE This error type indicates that an operation failed due to an implementation detail which does not map to the abstract level This may indicate a server memory alloc situation or to many open SCI driver conne
180. r power switch for host scalil 21 powsw01 Q Please enter power switch port outlet number for host scalil 21 3 50 Scali System Guide Version 3 0 4 3 Scali software installation Please specify which power switch node scalil 22 is connected to Configuration Power switch powsw01 Q Please enter power switch for host scalil 22 Type baytech scali pow3 powsw01 Q Please enter power switch port outlet number for host scalil 22 4 The following power switch mappings have been defined Power switch powsw01 powsw01 powsw01 powsw01 Host scalil 11 scalil 12 scalil 21 scalil 22 Q Do you accept the above configuration Port 1 2 3 4 y Please wait while retrieving information from the system Preparing repository Copying packages to repository ScaComd ScaConfC ScaConfNd ScaConfSd ScaCons ScaDesk ScaDiag ScaDiagSC ScaEnv ScaExecd ScaFPDisp ScaFPMB2 ScaFmpich ScaIP ScalPadap ScaLM ScaMAC ScaMACadap ScaMACddk ScaMPE ScaMPI ScaMPICHt ScaMPItst ScaMond ScaOPBS Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing Removing old old old old old old old old old old old old old old old old old old old old old old old old old Scali System Guide Ver
181. re described herein Such comments should also be sent to support scali com 10 6 4 Keeping informed Apart from regular visits to the Scali web site Scali also recommends subscription to the two mailing lists maintained by Scali These are scali announce which is where Scali posts the latest news about Scali products releases patches and so on and scali user which is a discussion group for Scali users Instructions on how to join the appropriate mailing list can be found on http www scali com support 150 Scali System Guide Version 3 0 Appendix A Scali software details A 1 General information A 1 1 Scali software platform CD ROM The Scali Software Platform SSP CD ROM contains software packages for all Scali products needed to get a Scali system installed and operative Below is a description of the top level file system layout on the SSP CD ROM README Start by reading this NEWS The SSP release notes e doc documentation e install TheScali software installation program uninstall The Scali software removal program e pkg All the software distribution files e bin binaries used during installation A 1 1 1 Installation for the impatient Before starting the installation program you should check out the online documentation for hardware configuration in doc Hw and operating system prerequisites in doc Hw for the latest information which may not have made it into the manual Hardware installa
182. rformance CPU Idle time percent 100 cpuidle After the keyword variable you must enter a descriptive string which will show up in the monitoring variable menu in Scali Universe If you want your newly defined variable to show up in one of the submenus you must define the entire menu path separated by white space within the descr field In the example above the CPU Idle time label is added tothe Performance submenu the very one you see by default in the Scali Universe GUI If you want to use space within the menu label you must double quote the string like you ve seen here The size field is used to tell about the scaling of the graph human percent and integer all expect integer value from the expression but percent will always scale the Y axis from O to 100 The maxvalue is really just advisory In the expression we can give an expression involving many oids variable Performance Processes per user integer 10 process_count user_count Here we just defined a Processes per user monitoring variable based on the two oid s we defined above Note all mathematical operators and the parenthesis must be surrounded by space H ence an expression like the following will silently be ignored process_count user_count You may notice that in the alarm section you may now define alarms based on your new oids Note After you change the ScaM ond conf file you must restar
183. rs are those that are held by only one node Nodes with exclusive partition number can operate separately You may list the partitions with the list partition all command in scaconftool 7 4 5 Testing the routing Once routing is set it may be tested with the sciping command in scaconftool This command will send an sciping between all nodes listed The sciping command is a combination Of scash and the command opt scali bin sciping The response is a list of possibly not responding nodes for each node in the list Remember that in a system where one or more nodes or rings are down there may be nodes that will not respond This applies to faulty nodes or nodes where both SCI rings are unusable All nodes with the same partition number should be able to reach each other with sciping i sciping 1 7 5 The configuration server daemon The configuration server is available in the ScaConfSd package The server must run on the frontend and will start automatically when installed and when the frontend is re booted The server can be stopped started or restarted manually with the opt scali init d scaconfsd Script on the frontend opt scali init d scaconfsd start stop restart When the server starts it will read a configuration file to get the name of all nodes and the configuration of the machine Default configuration file is opt scali ScaConf conf Another file may be specified with the f option The f option is only available if
184. rse installation e multiple patches to improve fault handling reliability and scalability Please check the file opt scali contrib pbs doc RELEASE_NOTES at the frontend node for a complete list of patches The Scali Universe graphical cluster management interface includes functionallity to submit and monitor OpenPBS jobs PBS consists of four major components commands the job Server the job executor and the job Scheduler Scali System Guide Version 3 0 113 Chapter 8 Batch system ScaOPBS Commands are used to submit monitor modify and delete jobs Some commands are available for all users others are available only for the administrator Commands are conformant with POSIX 1003 2d Batch Environment Standard The job server pbs_server provides the basic batch services such as receiving creating a batch job modifying the job protecting the job against system crashes and placing the job into execution The pbs_server daemon is running at the frontend node The job executor pbs_mom is placing the job into execution when it receives a copy of the job from the server It creates a new session as identical tothe user login session as possible It will return thejob s output to the user The pbs_mom daemon is running at all the execution nodes in the cluster Thejob scheduler pbs_ sched iS controlling the policy for which job is run whereand when The scheduler communicates with the various job executors to obtain information about
185. s in a Scali cluster e scarcp _ Copies file s from to local machine to from nodes in a Scali cluster e scaps Prints processes on nodes in a Scali cluster e scakill Kill processes on nodes in a Scali cluster e scarup Prints status on nodes in a Scali cluster e scawho Prints user names and number of processes on nodes in a Scali cluster Thefile opt scali etc ScaSH conf contains configuration parameters to the ScaSH suite of tools It will be sourced by the tools and must adhere to bourne shell script syntax Table 9 1 on page 137 describes the options in the configuration file and lists their default values All programs in the ScaSH suite will use the nodes reported by the scahosts program unless nodenames are specified on the command line 9 1 scash the parallel shell command The scash utility located in opt scali bin is a UNIX script that can run the same shell command on a set of Scali system nodes The target nodes may be specified in a confi guration file or on the command line see description of scahosts program in 9 2 The command options are listed below Usage scash options n nodenames lt command gt c command lt nodenames gt options Print help text h Print help text V Print machine name and command before each output block V Print version p Print machine name before each line in each output block d Dryrun only print command do not execute b Run in background W Wait
186. see is layers where the links connecting these layers are not shown The lt stacked link gt parameter decide which link that is not shown Default is link 2 Display formats Name Node name Nodeid H ex number Status OK indicates that everything is ok UNREACHABLE indicates the nodeis down or unreachable from ScaConf s point of view The reason might be no power or that scacomd does not run on this node NO DAEMON indicates the ScaConf configuration daemon is not running on the node NO_SCI_DEVICE indicates the SCI driver is not loaded UNKNOWN isan initial state indicating that no information is received from the node yet Linko EN indicates that SCI link controller is enabled DIS indicates that SCI link controller is disabled UP indicates that the SCI link is up DOWN indicates that the SCI link is down Link1 same as LinkO Link2 same as LinkO Pos Tuple triple coordinates for 0 1 and 2 link to indicate wherein Scali System Guide Version 3 0 175 Appendix D ScaConf Reference the topology the node is located Part Integer routing partition number Example output ConfTool gt list nodeid status LO L1 all Name NodeId Status LinkO Linkl scali2 11 0x1100 OK EN UP EN UP scali2 12 0x1200 OK EN UP EN UP scali2 13 0x1300 OK EN UP EN UP scali2 14 0x1400 OK EN UP EN UP Example list Lists information on nodes in built in list Use select to add nodes to the internal list The format from previous list com
187. sion 3 0 versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions versions copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying copying repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository repository EIA AAA 51 Chapter 4 Software installation ScaOPBSdk Removing old versions and copying files to repository ScaOSusbrm Removing old versions and copying files to repository ScaPkg Removing old versions and copying files to repository ScaPowd Removing old versions and copying files to repository ScaSCI Removing old versions and copying files to repository ScaSCIadap Removing old versions and copying files to repository ScaSCIddk Removing old versions and copying files to repository ScaSH Removing old versions and copying files to repository ScaSISCI Removing old versions and copying files to repository Sc
188. snmpd start stop restart info 6 7 Defining new monitoring variables The monitoring capabilities in the Scali Universe is easily extended to monitor other SNMP variables than the default set It s necessary with some knowledge of SNMP to create the necessary configuration statements Y ou can here create a monitoring value based on a function off many SNMP OIDs like sums ratios etc 6 7 1 Obtaining the SNMP OIDs First of all you need to know the SNMP Object ID you want to monitor We recommend that you have the snmpget and snmpwalk command available on Red Hat systems they re located in the ucb snmp utils Of net snmp utils package You should test that your SNMP variables are available in the SNMP daemon with the snmpget or snmpwalk command like this snmpwalk p 32016 localhost birka host 1 host hrSystem hrSystemUptime 0 Timeticks 310235448 35 days 21 45 54 48 host hrSystem hrSystemDate 0 2002 6 13 12 0 5 0 2 0 host hrSystem hrSystemInitialLoadDevice 0 1536 host hrSystem hrSystemInitialLoadParameters 0 auto BOOT_IMAGE 1inux2417 ro root 4806 BOOT_FILE boot vmlinuz 2 4 17 2smp console ttyS0 host hrSystem hrSystemNumUsers 0 Gauge32 4 host hrSystem hrSystemProcesses 0 Gauge32 131 host hrSystem hrSystemMaxProcesses 0 0 Or if you prefer the numeric OI D s snmpwalk On p 32016 localhost birka host 1 1 3 6 1 2 1 25 1 1 0 Timeticks 310235107 35 days 21 45 51 07 1 3 6 1 2 1 25
189. start due to a license problem this will be clearly indicated by an error message logged to the system logfile Thelog files are var log messages for Linux and var adm messages for Solaris B 6 1 Error Invalid FEATURE line This means that the license file found by the application does not contain any valid FEATURE lines entries or that the encryption key of the FEATURE is invalid B 6 2 Error Invalid version x x gt y y This means that the version of the FEATURE you retrying to check out is newer then the one you have in your license file The most common reason for this is that you are trying to use and outdated copy of the license file This may happen if you upgraded your software and forgot to update the license file B 6 3 Error FEATURE has expired This error occurs only with time limited licenses and tells you that the license for this FEATURE has expired The only time limited license used by Scali software are DEMO licenses B 6 4 Error Host ID is not found This error occurs with host bound licenses only and means that the requested FEATURE is not valid for this host Most Scali software licenses are host bound 162 Scali System Guide Version 3 0 B 7 Older versions with FLEXIm B 6 5 Missing license file Naturally a missinglicensefile will create problems To ensure you that thelicensefile has been properly distributed to all nodes you can use scash like this opt scali bin scash pwb ls 1 opt scali etc lic
190. stigate your problems further make sure the hardware installation is in accordance with the guidelines in Chapter 3 10 2 Software installation problems Before you investigate your problems check that your software installation is according to the guidelines in Chapter 4 If you experience problems during installation always attach a copy of the output from the install script if you contact support scali com 1 Wulfkit customers should contact wulfkit support dol phinics com Scali System Guide Version 3 0 139 Chapter 10 Troubleshooting Problem SCI test fails during SSP install Solution 1 Check that the cabling is according to the file opt scali etc ScaConf conf f you find that thereis a mismatch correct the cabling or the file whichever isin error and restart scaconfsd opt scali init d scaconfsd restart Then you can rerun the SCI test from the install script 2 Check that the kernel is supported If kernel is not supported run ScaSCladap 3 Check that all links and nodes are OK Start opt scali sbin scaconftool ssue command list no st links all and check that all nodes has status OK and that all links are UP and ENABLED If there are any nodes that are UNREACHA BLE or links that are DOWN fix this problem before retrying the SCI test from the install script 4 Look for lines containing ssci in var log messages on all nodes This is log error messages created by the ScaSCI dri
191. sts on the command line when running the program either with the f option which gives the path to a file containing host names or with a list of hosts at the end of the command If neither the f option nor the hostlist is present scahosts will look for hosts in the file HomE scahosts If this file is not available it will look for hosts in the file opt scali etc ScaSH scahosts Thefile containing hostnames should have one name per line The option rsH_tEstaccess in the configuration file scasH conf see table 9 1 may be set to disable checking for remote shell access to validate a node Usage scahosts options host 132 Scali System Guide Version 3 0 9 3 scacp local file copy options f lt file gt Use node names from file containing nodenames separated by newlines V Print verbose information Z Do NOT check access to nodes x n1 p2 List of nodenames to be excluded V Print version h Print help message 9 3 scacp local file copy The scacp program copies file s locally on nodes in a Scali cluster Usage scacp options lt from gt lt from gt lt to gt options Print help text h Print help text R Copy recursively V Print machine name and command before each output block V Print version p Print machine name before each line in each output block d Dryrun only print command do not execute b Run in background W Wait until command has finished on all hosts n n1 n2
192. t and management systems to OEMs and VARs e as build your own supercomputer kits WulfKits Consequently the wrapping and contents of your Scali products may vary but it is still called a Scali system throughout this document 1 2 Overview of the Scali System Guide The Scali System Guide is organised as follows Chapter 2 presents an overview of Scali systems and software Chapter 3 describes the hardware installation procedure including the high speed interconnect adapters Chapter 4 describes the software installation procedure including the operating system configuration and the Scali Software Platform Chapter 5 describes the Scali Universe graphical system management framework and how to install configure and start using it Chapter 6 describes the use and configuration of the Scali monitoring system in detail with graphical monitoring alarms and user defined variables Chapter 7 gives a detailed description of Scali interconnect configuration system with examples of usefrom Scali U niverse and the scaconftool ASCII based configuration client Chapter 8 describes theintegration configuration and use of the OpenPBS queue system with Scali systems Chapter 9 describes the Scali parallel shell tool suite starting with the fundamental scash and then moving on to the more specialised tools like scaps Scali System Guide Version 3 0 11 Chapter 1 Introduction Chapter 10 describes how to troubleshoot common problems usin
193. t the scamond service for the changes to take effect This must be done as root with the command service scamond restart 94 Scali System Guide Version 3 0 Chapter 7 Interconnect configuration Aimed primarily at system administrators this chapter describes organisation of and how to use the Scali configuration and management system for high speed interconnects ScaConf 7 1 Overview Currently supporting SCI networks of different topologies ScaConf provides automatic configuration of SCI networks for maximum connectivity and utilization With distribution check of SCI node l Ds and routing as primary functions ScaConf is strictly necessary in order to make any but the smallest SCI network operational Advanced routing algorithms developed at Scali see Routing on page 107 allows ScaC onf to bypass failed nodes with the least possible negative impact on total system performance The ScaConf system has been designed for automatic operation but users with root privileges still have access to management interfaces both through the Scali Universe GUI and the ASCII based scaconftool 7 1 1 Architecture LAN scaconfnd scaconfsd y V Remote Universe GUI a node process Local Universe GUI frontend cluster nodes Figure 7 1 ScaConf components in context The ScaConf system consists of several software components The Scali interconnect configuration node daemon scaconfnd provides an interface between t
194. talg prints the available routing algorithms Example info topology routalg D 2 7 ConfTool gt select The sdect command adds nodes to the list of active nodes Usage select lt list gt Example select scali2 11 scali2 12 select all select NO_DAEMON D 2 8 ConfTool gt unselect The unsdect command removes nodes from the list of active nodes Usage unselect lt list gt Example unselect scali2 13 scali2 12 unselect all unselect UNREACHABLE D 2 9 ConfTool gt list The list command displays various information about nodes Usage 174 Scali System Guide Version 3 0 D 2 scaconftool command reference list lt format gt lt list gt or list configuration lt stacked link gt lt format gt status nodeid LO L1 L2 links partition position configuration status Status for the node nodeid N ode identification for SCI devices LO Status for link O L1 Status for link 1 if available L2 Status for link 2 if available partition Routing partition number position X Y Z coordinates Node name is always printed Default is that only nodeid and node name is printed The format is stored and will be used if the next list command does not specify any format configuration List the configuration of the cluster in a matrix representation reflecting the topology of the cluster On 3D systems to display the system in a flat view one link is stacked and what you
195. ted for non commercial non profit purposes There can be no charge for the Software or any software incorporating the Software Further there can be no expectation of revenue generated as a consequence of redistributing the Software 3 Any Redistribution of source code must retain the above copyright notice and the acknowledgment contained in paragraph 6 this list of conditions and the disclaimer contained in paragraph 7 4 Any Redistribution in binary form must reproduce the above copyright notice and the acknowledgment contained in paragraph 6 this list of conditions and the disdaimer contained in paragraph 7 in the documentation and or other materials provided with the distribution Scali System Guide Version 3 0 129 Chapter 8 Batch system ScaOPBS 5 Redistributions in any form must be accompanied by information on how to obtain complete source code for the OpenPBS software and any modifications and or additions to the OpenPBS software The source code must either be included in the distribution or be available for no more than the cost of distribution plus a nominal fee and all modifications and additions to the Software must be freely redistributable by any party including Licensor without restriction 6 All advertising materials mentioning features or use of the Software must display the following acknowledgment This product includes software developed by NASA Ames Research Center Lawrence Livermore National Laboratory
196. ter no gt Options h Print this help string V Print version information an gt Use adapter number lt n gt s lt id gt Sets new nodeid for the adapter Scali System Guide Version 3 0 167 Appendix C SCI Utility Programs C 1 7 scinfo Thescinfo utility gives global information and statistics for all SCI instances and are particularly useful as an aid in error diagnostics Usage opt scali sbin scinfo v a lt adapter number gt Options m show memory and att usage statistics Ku show processes with active SCI connections l how SCI link statistics p show pkt statistics driver to driver comm C show driver configuration and version i how general interrupt statistics V all of the above equal to mlpci h print help text C 1 8 scireconf Thescireconf program may be used to reconfiguratethe Scali SCI driver This utility is only to be used upon instructions from Scali Usage opt scali sbin scireconf d parameter value Options d parameter Delete comment out the parameter setting parameter Display current setting if any parameter value Change and enable parameter with this value C 1 9 scireload Thescireload command will reload the Scali SCI driver on the machinethe command is run on This command is only available to root and it should not be necessary to use this command under normal circumstances Usage opt scali sbin scireload 168 Scali System Guide Version 3 0 C 1
197. the system frontend This gives you secure remote terminals on any nodein the system at the dick of a button without struggling with ssh Xterm and DISPLAY variables Scali System Guide Version 3 0 73 Chapter 5 Scali Universe If more than one node was selected you will get a small window named Master in addition to one terminal window for each sel ected node This Master terminal window provides you with terminal broadcasting and command history MASTER delfi nodef Figure 5 13 The Master window for terminal broadcasting When you move the mouse cursor into the mouse window and start typing everything you type is replicated to all terminal windows Using the up and down arrow keys allows you toscroll up and down in previous commands Notethat for security reasons the echo in the Master terminal window has been turned off Typing exit into the Master window will close all sessions 5 3 4 6 Frontend terminal session Similar to the node terminals the Frontend Terminal option enables a quick and easy way to get a terminal connection to the system frontend 5 3 5 Management menu The Management menu provides the tools for system management related tasks of the Scali system including the use of remote power and console switches el BUS Reboot Shutdown Figure 5 14 The Management menu 74 Scali System Guide Version 3 0 5 3 Using Scali Universe The items in the Management menu are described in table 5 4
198. tion is described in detail in chapter 3 Operating system configuration and Scali software installation in described in chapter 4 When you are ready to start the installation type the following as root install Theinstallation program will guide you through the installation process which will install SSP components according to your licenses on the whole cluster The installation program will detect if this an upgrade and re use configuration settings as appropriate Note that the SSP installation program provides a number of useful command line switches see SSP install program options on page 40 One example is the ability to rerun parts of the installation like the test section with t install t Scali System Guide Version 3 0 151 Appendix A Scali software details A 1 1 2 Uninstall If you wish to remove the SSP at a later stage you can run the uninstall program as root uninstall This will remove all traces of the SSP on all nodes A 1 2 Scali software directory structure All Scali software is installed under the directory opt scali Thetop level directory structure is as follows e bin all normal executables to be run by users e sbin all daemons and executables to be run by administrators e libexec executables which are hidden from normal invocation e g used by applications under bin e lib libraries used by our applications and by end users e include indude files for li
199. torus From this figure we can produce some general connection tables 3 5 through 3 7 for each of the SCI links The resulting cabling on a 2x9 physical node arrangement can be inspected in figure 3 13 The cabling scheme works equally well for a 1x18 node rack but will probably require somewhat longer cables on theL 2 links This illustrates how node placement becomes more important with 3D systems From LO OUT To LO IN node zx1 node zx3 node zx3 node zx2 node zx2 node zx1 Table 3 5 Example cabling of LO on a 3x3x2 torus z4 1 2 x gt 1 2 3 30 Scali System Guide Version 3 0 3 4 Dolphin SCI interconnect installation From L1 OUT ToL1 IN node zly node z3y node z3y node z2y node z2y node zly Table 3 6 Example cabling of L1 on a 3x3x2 torus z4 1 2 y 1 2 3 From L2 OUT ToL2 IN node 1xy node 2xy node 2xy node 1xy Table 3 7 Example cabling of L2 on a 3x3x2 torus x4 1 2 3 y 1 2 3 Figure 3 13 Example cabling of
200. uded Z Do NOT check access to nodes the default is to ping and test rsh access 136 Scali System Guide Version 3 0 9 8 scawho user information 9 8 scawho user information The scawho program prints usernames and number of processes on nodes in a Scali cluster Usage scawho options options Print help text h Print help text V Print machine name and command before each output block V Print version p Print machine name before each line in each output block d Dryrun only print command do not execute b Run in background W Wait until command has finished on all hosts n n1 n2 Nodenames separated by space characters f lt file gt Use node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test remote shell access 9 9 ScaSH configuration file Thefile opt scali etc ScaSH conf contains configuration parameters for the ScaSH parallel shell tools suite It will be sourced by the tools and must adhere to bourne shell script syntax Table 9 1 describes the options in the configuration file and lists their default values Default Option value Description RSH_CMD rsh n Which program should be used to execute a command remotely The program specified must support the following calling con vention RSH_CMD lt host gt lt command gt Table 9 1 Options in the file ScaSH con
201. ue bars you will get exact readout in the form of a pop up label with the node name and value of the monitoring variable 6 3 3 Two dimensional 2D bar view Scali Universe delt delfi2 1 elfi2 4 delfi2 2 delfi4 1 delfid 3 delfid 4 delfid 2 delfi3 1 delfid 3 delfi3 4 delfi 2 delfil 1 delfil 3 delfil 4 delfil 2 Figure 6 5 2D Bar small icons view of system load The 2D bar view presents monitoring variables as classic bar graphs with a node layout similar to the node placement in the MainWindow Numeric values are presented above each bar for exact readout This view also supports a small icons view which is great on larger systems Scali System Guide Version 3 0 81 Chapter 6 System Monitoring 6 3 4 3D bar View Figure 6 6 3D bar view on a large system 132 nodes The 3D bar view is a translation of the 2D bar view into three dimensions Node layout is similar to that of the MainWindow with a 3D bar in each node position Thenumeric value of the measured variable is floating above the bar representation The 3D bar view supports all 3D view camera controls 6 3 5 2D history view Scali Universe Scali 1 CPU User time 100 0 50 0 0 0 Figure 6 7 2D history view showing two of four nodes The 2D history view is very similar to the classic Xload program and enables you to easily spot deviating values within the history time frame with precise readout Here one graph per nodeis arranged
202. un reloading essential drivers and reconnecting the nodes to configuration server Usage fix lt list gt Example fix UNREACHABLE scaconftool will attempt fix all nodes with status UNREACHABLE D 2 4 ConfTool gt getopt Thegetopt command lists server options and their values If thelist of options is empty all available options will be listed Some of the options are defined as ACTION These options does not havea value but represents actions that theserver will perform when activated with the setopt command Usage getopt lt option gt lt option gt 172 Scali System Guide Version 3 0 D 2 scaconftool command reference lt option gt Can be replaced by any legal options Example getopt getopt AUTO_REPAIR D 2 5 ConfTool gt setopt The setopt command sets the value of the specified server options Some options are defined as ACTION Applying setopt on this type of option will causethe configuration server to perform the action Saving options to file and restoring factory defaults are available actions Usage setopt lt option value gt lt option value gt lt option value gt Can be replaced by any legal options Use getopt with no parameters to get a list of all legal options Options are described in Table 10 9 Example setopt AUTO_REPAIR OFF AUTO_REROUTE OFF OPTION VALUE DESCRIPTION AUTO_REPAIR ON OF F Turn auto repair on or off When on server will automatically poll erroneous
203. under GNU Public License see GPL txt SCALI AS SOFTWARE LICENSE TERMS 1 You acknowledge that the software and the documentation for the Scali software products are copyrighted by Scali AS and that while you may acquire a license to use them on a non exclusive basis you will acquire no title or intellectual property rights 2 You may not modify the content of any of the files of the software or the online documentation 3 The software contains proprietary algorithms and methods You may not attempt to reverse engineer or disassemble the software 4 The software is operated under the control of a license manager that regulates licensed usage of the software You may not attempt to modify or tamper with any function of this license manager Scali System Guide Version 3 0 43 Chapter 4 Software installation 5 Transfer Your license will automatically terminate upon any transfer of the Software Upon transfer you must deliver the Software including any copies and related documentation to the transferee The transferee must accept these License Terms as a condition to the transfer 6 Copies and Adaptations You may only make copies or adaptations of the Software for archival purposes or when copying or adaptation is an essential step in the authorized Use of the Software You must reproduce all copyright notices in the original Software on all copies or adaptations You may not copy the Software onto any public network 7
204. until command has finished on all hosts Scali System Guide Version 3 0 131 Chapter 9 ScaSH parallel shell tools a Use at now to execute command S Which remote execution copy command to use r key Replace key with machinename if key is found in command n gt Limit on number of parallel processes to run in background mode F lt n gt Fanout value Default is 8 Use O or 1 to force no fanout n n1 n2 Nodenames separated by space characters f lt file gt U se node names from file containing nodenames separated by newlines x n1 p2 List of nodenames to be excluded Z Do NOT check access to nodes the default is to ping and test remote shell access c lt cmd gt Theshell command to be run lt Node names gt The default if no nodes are specified is all nodes in the user s local scahosts file home user scahosts If that file does not exist the global scahosts file opt scali etc ScaSH scahosts is used Whenever you explicitly specify nodes the contents in the config files is ignored lt command gt Any command you want 9 2 scahosts nodes availability check The scahosts program located in opt scali bin will check a number of hosts for availability An available host is one that answers to a ping request and is open for remote shell access The program will print the name of the available hosts Which hosts the program will check may be specified in several ways One may specify ho
205. uration in an upgrade Choose a different package repository please answer yes to the question below Under normal circumstances this should not be neccesary Available for a limited number of platforms only Q Do you want to run it in expert mode n Checking source media 5 5 OK 4 3 2 2 Installation Configuration Thesecond part of the installation deals mainly with configuration of the cluster Next is an example from a ClusterE dge installation Note the use of brace expansion when specifying the node names in the cluster For a big system this way of specifying the node names will greatly reduce the workload when installing Installation configuration a 222222222222 222222222222 22 Checking upgradeability 2 3 INSTALL NOTE Information from previous installation is not found Upgrade is not a choice Specifying frontend name 2 a This program should only be started on the machine you want to b the frontend The frontend is among other responsible of installing Scali software and upgrades on the cluster running the configuration and monitoring servers In small systems lt 8 nodes this is usually one of the compute nodes In larger systems a separate node is preferred The name of this machine scalil 11 Q Are you sure you want to use this machine as a frontend y y Read HW documentation 7 77 2 gt
206. ver When installed the SCI driver is always loaded upon reboot of a node If the SCI driver somehow needs to be reloaded this can be done with the reload command from within scaconftool The reload command is only available to root Reloading the SCI driver on a node will clear the routing table on that node leaving the cluster in need of rerouting This is done automatically unless the AUTO_REROUTE server option has been set explicitly to OFF 7 3 7 Access to console on one node The console command may be used to connect to the console on one node For this command to work a console switch must be available and the ScaCons package must be installed correctly See man console for more information The second argument tothe console command is passed directly tothe opt scali bin console program 7 3 8 Hot restart of the configuration server Y ou may force the configuration server to perform a hot restart with the restart command A hot restart means that the server breaks connections to all nodes clears its configuration database and restarts its configuration thread all without losing the connections to the clients As the configuration fileis read on startup only this means that you can make changes to the configuration filetake effect on thefly without the need for a full restart A full restart takes significantly longer and of course breaks all connections Scali System Guide Version 3 0 105 Chapter 7 Interconnect
207. ver and they might give a clue on why the SCI interconnect does not have full connectivity When installing SSP or run ning scapkg get the error message all ports in use Thesolution is to reduce the number of nodes toinstall on and run several rounds of scapackge untill all nodes are covered This may be a problem on large systems New versions of scash is supposed to handle this but if you run into this situation during SSPin stall you can stop the SSPinstall script and try to install pack ages with scapkg Run opt scali sbin scapkg on half of the nodes first then on the rest If you still get all ports in use try running on a smaller set of nodes When scapkg has been run with all nodes you may resume with the installation process Access test to nodes failed for one or more nodes Read the file opt scali doc OS and make sure rsh ssh configura tion on the nodes in question are according to the prerequisites stated there Installation of ScaSCI driver rpm complains about unre solved symbols SSP only support certified RedH at base or update linux kernels If you on your own risk try toinstall SSP with a different kernel you will need to run ScaSCladap to adapt the ScaSCI driver to the kernel Table 10 1 Software installation troubleshooting 10 3 License prob lems Troubleshooting licensing problems is described in the appendix Scali Software Licensing on page 157
208. you how configuration of serial console switch and power switches is performed during installation This requires the use of qualified switch hardware and a product that supports console and power switching Specifying console switch map Sam Please specify the console mappings O Please enter console switch hostname for host scalil 11 scali cs1l Q Please enter console switch TCP port number for host scalil 11 10001 10001 O Please enter console switch hostname for host scalil 12 scali cs1 Q Please enter console switch TCP port number for host scalil 12 10002 O Please enter console switch hostname for host scalil 21 scali cs1 Q Please enter console switch TCP port number for host scalil 21 10003 O Please enter console switch hostname for host scalil 22 scali cs1 Q Please enter console switch TCP port number for host scalil 22 10004 The following console switch mappings have been defined Host Console switch Port scalil 11 scali csl 10001 scalil 12 scali csl 10002 scalil 21 scali csl 10003 scalil 22 scali csl 10004 Q Do you accept the above configuration y y Specifying power switches Please classify your power switch es Please select what type of switch the power switch number 01 is Select switch type by entering a number from the list below 1 Baytech RPC 3 with ethernet connection 2 Pulizzi with

Download Pdf Manuals

Related Search

Related Contents

RedMax BT280 User's Manual  取扱説明書 オムロン ねむり時間計  STERLING 72170110-0 Installation Guide    and site-rite ii® hand benders  Manuel de l`opérateur  SBS Aero  Manual - Animo  Targus Zierra  Bedienungsanleitung ()  

Copyright © All rights reserved. Failed to retrieve file