Pipeline Pilot Interface User Guide
Contents
1. Pipeline Pilot Interface Version 1 0 0 4 User Guide for Package version 1 0 0 and above and Pipeline Pilot version 8 0 and above Edgar Derksen Sally Hindle Institut Algorithmen und Wissen schaftliches Rechnen Fraunhofer 2015 BioSolveIT GmbH An der Ziegelei 79 53757 St Augustin Germany Phone 49 2241 2525 0 support biosolveit de Contents Contents 1 Quick Start Steps 1 1 Download and Import current PipelinePilot Package 12 TEEN Tools ade nee need ee 1 3 Create or Update Protocols x Eure an eek a a a ed Bu a 2 Installation 2 1 BioSolvelT Web Installation 2 Hm m mn nn 2 2 Using Offline Installation Components 00000000005 2 3 Using an External Tool Installation 0 0 0 00 000 2 3 1 To Connect to an Tools Installation on the Pipeline Pilot Server 2 3 2 To Connect to Tools Installation on a Remote Linux Server 2 4 Running Tools in Parallel in Pipeline Pilot 2 4 1 Running in Parallel on the Pipeline Pilot Server 2 4 2 Running in Parallel on a Remote Linux Cluster 2 4 3 Example Scenarios and Required Settings 2 5 _Uninstallationl 2 2 2 mo mn nn 3 Trouble Shooting 3 1 Problems using the ssh MEEO gt oee eee ew eda Besa ES vaks 3 2 Further help and BioSolvelT PDF Reporter 0 4 Tips and Tricks 4 1 Other Significant Parameters in at C2. via SSH gt Host gt using scp This is time consuming It is possible that the Pipeline Pilot Server and lt Run FTrees gt via SSH gt Host gt actually share the same file system rendering the scp process unnecessary Select True if the Pipeline Pilot Server and lt Run FTrees gt via SSH gt Host gt share the same File System no copying of data is necessary Selecting False means Pipeline Pilot copies all the data to and back from the lt Run FTrees gt via SSH gt Host gt this is just a little slower but will always still work Leave the parameter set to False if you are uncertain 4 2 Accessing Other Domains within Pipeline Pilot Often in house data or even your own working data are accessible from a windows computer via a domain a path starting for example z or which you cannot find from within Pipeline Pilot That means you must first literally transfer the data to the Pipeline Pilot Server itself If you are using a Linux Pipeline Pilot server this hint does not apply To get around this problem and make the Pipeline Pilot working environment much more flexible you can allow users access to domains you need Administrator rights to be able to do this Also check first that you should change these settings as they may have already been set to fit the current environment e Go to the Scitegic Server Home Page for example via the Help menu in your Pipeline Pilot client e Click on Pipe
3. H gt Executable gt enter the path to the Tools installation on the Linux machine For example software BioSolveIT tools bin tools 3 For the parameter lt sshHost gt enter the Linux machine host name e User specific steps 1 For the parameter lt Run Tool gt via SSH gt User gt enter the user login name for ssh on the Linux machine 2 For the parameter lt Run Tool gt via SSH gt Password gt enter the user password for ssh on the Linux machine 10 CHAPTER 2 INSTALLATION 3 There are more advanced options to be found under lt Run Tool gt via SSH gt Options gt for more specific ssh parameters Note the option lt Run Tool gt via SSH gt Options gt Delete Results gt this may be useful for trouble shooting later You can save these settings in the component be sure not to save your own user specific login details in components available to others Implementation User Mode Default v a E Run LeadIT via SSH gt E via SSH 5 Executable leadit exe software BioSolvelT LeadIT leadit Host hostl User userl Password Implementation Parameters Figure 2 3 ssh Connect to Tools Installation on a Remote Linux Server via ssh All files necessary for the calculation will be transferred via scp between the Pipeline Pilot server and ssh Linux machine Files copied and files created the remote server are automatically deleted at the end of the job leaving no trace However in ca
4. Pipeline Pilot server will run with the number of processes given in lt Processes gt How ever in this case the processes the server receives are not the tool calculations themselves but instructions for starting the ssh jobs The server iterates through its 10 jobs each time spawn ing an ssh job on a Linux host Beware that the Pipeline Pilot server does not know how many processors the Linux hosts have so make sure you choose the number of lt Processes gt to fit the number of Linux hosts and their number of processors respectively be careful not to overload the Linux hosts Figure 2 6 shows the detailed settings required to set up the calculation as in figure 2 5 Implementation User Mode Expert gt a E Run LeadIT via SSH BE on PP Serve E via SSH Executable leadit exe software BioSolvelT LeadIT leadit Host hostl host2 host3 host host5 l User userl Password id Options E Parallel Processing Options True vl Batch Size 1 Ue Server Processes 10 Preserve Order True vir Implementation Figure 2 6 The options for parallel processing for the example shown in figure 2 5 would look similar to these As you may already have realised you could enter more than one Pipeline Pilot server at the lt Server gt parameter along with another entry for number of lt Processes gt as a comma 2 4 RUNNING TOOLS IN PARALLEL IN PIPELINE PILOT 13 separated list to execute a doubly p
5. TALLATION 9 license is available from a license server simply type in the name of the server in this format servername or if you have a license file then you may browse for it using the facility You can save these settings in the component Implementation User Mode Default ME El Run LeadIT on PP Server gt E on PP Server r Use Preinstalled LeadIT 1 License Server or License File server Executable leadit exe C Program Files BioSolvelT LeadIT leadit exe A v Implementation Parameters Figure 2 2 PPServer Connect to Tools Installation on the Pipeline Pilot Server 2 3 2 To Connect to Tools Installation on a Remote Linux Server Your existing Tools installation could be on a Linux computer remote from the Pipeline Pilot server in this case we offer an alternative so you can use the remote installation in stead Here the calculations carried out by Tools will be done on the remote Linux machine Pipeline Pilot logs into the Linux machine using ssh having copied all relevant files to the machine and will run the calculation there finally copying back all data it needs to the Pipeline Pilot server to continue with the pipeline This method is given the label ssh You can see an example in figure 2 3 e Requirement Tools are installed on a Linux machine available to the Pipeline Pilot server via ssh e General Steps 1 Set lt Run Tool gt to via SSH 2 For the parameter lt Run Tool gt via SS
6. aded settings v 1 1 0 from C userdir leadit settings pxx gt gt LeadIT_base license check BioSolveIT keys succeeded gt gt Licensed modules LeadIT CDOCK FlexE PHARM PERMUTE SCREEN DEVELOP 17 18 CHAPTER 3 TROUBLE SHOOTING 3 1 Problems using the ssh Method You may also experience problems using the ssh login for example the user name is unknown or the host is not found 3 2 Further help and BioSolvelT PDF Reporter More complicated errors may arise during the running of tool Again though the errors will be collected and as much information shown as possible If you are familiar with the tool you may want to take a look at all the output of the job yourself to see if you can recognize the problem In this case you can look in the temporary folders Pipeline Pilot sets up internally to find the output or if you are working with the ssh method set the parameter lt Run Flrees gt via SSH gt Options gt Delete Results gt to False so that you may then find the files retained on the ssh host these will be in the directory set under the ssh parameter lt Run FTrees gt via SSH gt Options gt Temp Path gt see the help text associated with this parameter to find its default value essentially a cryptically named folder whose name begins with the date and time of the job If you still do not know what is causing the errors write down as much information as possib
7. arallel calculation Note to get this method to work you will need to change the maximum number of processes per Pipeline Pilot server The Pipeline Pilot Client will let you enter any number for the lt Processes gt parameter and does not warn you if this number is above the maximum Changing the maximum number requires Administrator rights The number of processes per Pipeline Pilot server is usually restricted to the number of processors of the server You must override this maximum to be able to set the number of processes you want for your parallel calculation In the above example the maximum must be set to 10 or more Take the following steps e Go to the Scitegic Server Home Page for example via the Help menu in your Pipeline Pilot client e Click on Pipeline Pilot Administration Portal and log in with the Administrator user name and password e In the last field of the table Maximum number of simultaneous parallel processing subprotocols allowed change the value click Save and log out again 14 CHAPTER 2 INSTALLATION 2 4 3 Example Scenarios and Required Settings ssh PPServer Parallel Processing Options Number of PP servers in list Number of ssh Hosts in list Number of Processes PPServer False 1 Behavior The calculation will run as one complete job on the Pipeline Pilot server ssh PPServer ssh Parallel Processing Options False Number of PP servers in list Number of ssh Hosts in li
8. do this you must already have Tools installed somewhere on your system outside of Pipeline Pilot To install Tools yourself visit the download page at BioSolvelT http www biosolveit de download and fetch the download package for your system for the latest Tools package Follow the instructions in the package to install Tools and receive your licenses Enter the license infor mation for Tools as described in the package and not using the parameter lt Run Tool gt on PP Server gt License Server or License File gt as for the internal installation To use an external installation of Tools you must change the value of the parameter lt Run Tool gt on PP Server gt Use gt in the Implementation tab to preinstalled Tool 7 8 CHAPTER 2 INSTALLATION There are actually two ways to use Tools with an external installation These are by using Tools installed directly on the Pipeline Pilot server or by accessing a remote machine where Tools are installed using ssh method The method is selected using the parameter lt Run Tool gt Both methods are covered in more detail below 2 3 1 To Connect to an Tools Installation on the Pipeline Pilot Server The most common scenario is that you will have an installation of Tools on the Pipeline Pilot server If you choose this option you just have to enter the path to the executable and configuration file as parameters in the Implementation tab Pipeline Pilot will then just start Tools whenever th
9. e chapter 2 Create or Update Protocols Import components now existing in the defined Client Destination Dir via drag and drop to your components collection Drag and drop new components into a new protocol 5 CHAPTER 1 QUICK START STEPS 3 Update via a right click drag and drop of new component onto component reference in a protocol 4 Now you are ready to setup the new components and run your protocol Installation 2 1 BioSolvelT Web Installation The easiest way to download and install BioSolvelT packages and tools is to download and run BioSolvelT Web Installer component Download BioSolvelT in PipelinePilot package from http www biosolveit de download Read UserGuide within that package for fur ther details Note that both PP Client and PP Server need web access to run BioSolveIT Web Installer Without web access you need to follow one of the following sections in this chapter 2 2 Using Offline Installation Components For an offline installation working similar to BioSolveIT Web Installation please download for example LeadIT in PipelinePilot package and use the contained LeadIT Installer com ponent For further details read also the Userguide of LeadIT in PipelinePilot package 2 3 Using an External Tool Installation You can also use an existing external installation of the Tool software at each component This means you also have the opportunity to use settings different to those set by default To
10. er should receive The lists are both comma separated and must be in corresponding order Remember to adjust also the lt Batch Size gt accordingly Note the path to the external installation of Tools must be same on all servers 2 4 2 Running in Parallel on a Remote Linux Cluster We have developed an implementation in the component whereby a large cluster can be incorporated to run tool jobs without them having to be Pipeline Pilot servers However it must be a Linux cluster and the component must use the ssh method You must also have enough tool licenses available to the cluster The settings in the Implementation tab must be made as for running the ssh method with two important changes Instead of one lt Run FTrees gt via SSH gt Host gt enter a comma separated list of the host names in the Linux cluster Then you must choose how many batchs in total you want the job to be split into and enter this total in the lt Processes gt parameter Remember to change the lt Batch Size gt to fit with the number of processes 12 CHAPTER 2 INSTALLATION 10 Processes 1 Server 10 ssh logins 5 ssh hosts Host 1 Host 2 Host 3 Host 4 Host 5 Figure 2 5 The method of parallel processing on Linux clusters in the tool component The Pipeline Pilot server is given 10 processes The 10 processes start an ssh job respectively distributed amongst the ssh hosts Figure 2 5 may help clarify how the method works 8 y help y The
11. ey are required by making a call to the executable that you entered This method is given the name on PP Server On Pipelinepilot Server You can see an exam ple in figure 2 2 e Requirement Tools are installed on the Pipeline Pilot server You can see which machine is the Pipeline Pilot server by starting your copy of Pipeline Pilot Client on your own workstation and find the name or IP of the server shown at the bottom right of the status bar see figure P 1 You must find where the tool installations are on that machine Pipeline Pilot Professional Client Eile Edit View Tools Window Help DEIS x EBS Er PS Alm A EEE Favorites pr E 4 Generic Chemistry eder Prot Comp Hele B Help os Parameters Implementation Information Error Ready Server localhost 6 0 2 Figure 2 1 See where your Pipeline Pilot Server is installed e General Steps 1 Set lt Run Tool gt to on PP Server 2 Set lt Run Tool gt Use gt to Preinstalled Tool 3 For the parameter lt Run Tool gt on PP Server gt Executable gt enter the path to the tool installation on the Pipeline Pilot server For example C Program Files BioSolveIT tools3 tools exe 4 Add your license for the executable of Tools in the field for the parameter lt Run Tool gt on PP Server gt License Server or File gt in the Implementation tab if your 2 3 USING AN EXTERNAL TOOL INS
12. le relating to your installation scheme You can also create a PDF Report unsing BioSolveIT PDF Reporter component which summarizes installation data into one pdf file Send all the information to support biosolveit de 3 2 FURTHER HELP AND BIOSOLVEIT PDF REPORTER 19 Last error eadlT Docking Error LeadlT executable run into Error LeadiT Exit Code not 0 See Jobs Window gt LeadlT_Docking_Debug for more Information OK Help Locate Error Details gt gt Figure 3 1 An error box reporting that the Tools exited with error Protocol Name Status Start 7 4 New Protocoll Error Wed Oct Errors txt a LeadIT Docking De Jobs Help Error Figure 3 2 The full error report can be found in the Jobs tab 20 CHAPTER 3 TROUBLE SHOOTING Tips and Tricks 4 1 Other Significant Parameters in at Components For detailed documentation of all parameters refer to the documentation you find in the Help area of the Pipeline Pilot window We list here particularly interesting parameters those that greatly influence the workflow or change the outcome of calculations or those that may help you understand what is happening in the component 4 1 1 lt Has Same File System gt Implementation lt Run Tool gt via SSH gt Options gt Has Same File System gt Normally for an ssh job Pipeline Pilot must first copy all the data required by the tool to the lt Run FTrees gt
13. line Pilot Administration Portal and log in with the Administrator user name and pass word e Inthe Security tab go to Authentification e For the Authentification Method choose DOMAIN and a set of parameters will appear e Enter the domain name in the field Domain and choose Full for Impersonation e Choose DOMAIN for Retrieve Groups and leave Limit access to listed domains set to No 21 22 CHAPTER 4 TIPS AND TRICKS e click Save and log out again After you have done this you will need to enter your domain login details when you start the Pipeline Pilot Client
14. omponents s s sooo 4 1 1 lt Has Same File System gt ars is sw 244224122222 2284 4 2 Accessing Other Domains within Pipeline Pilot a 0 oO a NIN A 10 11 11 14 15 17 18 18 CONTENTS Quick Start Steps Download and Import current PipelinePilot Package Download current PipelinePilot package on http www biosolveit de download 2 Extract PipelinePilot package to a custom directory 3 From the custom directory import all components xml using PP Client via drag and drop to your components collection Install Tools Drag and drop BioSolvelT Web Installer component from the components collection to an empty protocol Set up Parameter Client Destination Dir Use the facility within filebrowser popup click on dektop icon and press select at the bottom right Using this setting the components will be downloaded to your local desktop Add your BioSolvelT license in the field for the parameter lt Run Tool gt on PP Server gt License Server or License File gt in the Implementation tab if your license is avail able from a license server simply type in the name of the server in this format server name or if you have a license file then you may browse for it using the facility Run protocol once it will result into html reports showing component download and installation success For alternative installations and more details se
15. res depend on the speed of your 2 4 RUNNING TOOLS IN PARALLEL IN PIPELINE PILOT 11 Implementation User Mode Expert Mi a E Run LeadIT on PP Server v E on PP Server Use Preinstalled LeadIT X License Server or License File riz A Executable leadit exe software BioSolvelT LeadIT leadit B Parallel Processing Options True Mi Batch Size 1 E Server localhost Processes 1 Preserve Order True v r Implementation Parameters Figure 2 4 The options for tuning parallel processing are found in the Implementation tab machines and network experiment with your set up if you intend to carry out large calcu lations often Also we advise you to read the Pipeline Pilot documentation about parallel processing to understand more fully how it works Note to set up large parallel processing jobs you will need Administrator rights to change one setting 2 4 1 Running in Parallel on the Pipeline Pilot Server You may have a multi processor machine as your Pipeline Pilot server If you also have the appropriate number of Pipeline Pilot and Tools licenses the simplest way to start a parallel calculation is to raise the number of processes to the number of processors of the machine You could also have more than one Pipeline Pilot server available in your network If so you can enter a list of the server names at the parameter lt Server gt Below that for the parameter lt Processes gt enter a list of the number of processes each serv
16. s with Tools in Pipeline Pilot is with the connection to the external Tools installation For one thing Tools itself must be correctly installed on the system independently of Pipeline Pilot it is essential first to make sure this is the case especially to make sure that Tools can locate the licenses Once Tools runs fine on your system the remaining key task is to make sure the path to the executable is correct within the Pipeline Pilot component When the error messages pop up they may contain an ToolError in the error message box as in figure B 1 Go to the Jobs tab below the Protocol workspace and check under the last run job for a file called ToolComponent Debug as in figure 2 Clicking on the link brings up HTML report with input and output data in a browser A correctly started Tools job outputs the following header if there is a problem you will see some of this header and the point where the problem occurs Lead Ii tT Copyright Prediction of Protein Ligand Interactions BioSolveIT GmbH Version 1 0 0 1 An der Ziegelei 79 Modules CDOCK FLEXE PHARM SCREEN PERMUTE 53757 St Augustin Germany Original Author Matthias Rarey www biosolveit de Contact leadit biosolveit de For information about additional contributors and copyright notes please consult the user guide or type help about gt gt Running on DELTA Windows 5 1 with 2 processors gt gt Lo
17. se the user would like to leave a copy of the calculation and result files on the Linux machine or for trouble shooting as mentioned above it is possible to set a parameter to tell Pipeline Pilot not to delete these files lt Run Tool gt via SSH gt Options gt Delete Results gt False 2 4 Running Tools in Parallel in Pipeline Pilot Tools in Pipeline Pilot takes advantage of the parallel computing options available in Pipeline Pilot to speed up longer calculations This section tells you how to adjust the op tions to your system and needs You will find the options in the Implementation tab as in figure The most important limitation to a parallel processing calculation is the number of licenses that you have If you only have a single license then parallel calculations will not be possible Further choices in the set up of the parallel computing computing calculation depend on the number of Pipeline Pilot licenses you have plus your choice of connection to the external Tools installation on PP Server or via SSH It is important to note that a balance must be achieved between the overhead caused by running several calculations instead of one and the size of the calculation there is a lot of overhead involved in sending all the essential data to different computers and collecting the results For the Tools component however we still think parallel computing will work also for parallel jobs with batch size 1 Of course these figu
18. st 1 Number of Processes Behavior The calculation will run as one complete job on the ssh host ssh PPServer PPServer Parallel Processing Options True Number of PP servers in list 1 Number of ssh Hosts in list Number of Processes 1 Behavior Default Induces the pipeline effect the job will run in chunks on the server processor s ssh PPServer PPServer Parallel Processing Options True Number of PP servers in list 1 Number of ssh Hosts in list Number of Processes gt 1 Behavior A true parallel effect the job will be run in chunks on the server processors ssh PPServer PPServer Parallel Processing Options True Number of PP servers in list Number of ssh Hosts in list 4 each with two processors 2525252 Number of Processes Behavior A true parallel effect the job will run in chunks in par allel split across 8 processors ssh PPServer ssh Parallel Processing Options True Number of PP servers in list 1 Number of ssh Hosts in list 5 each with 2 processors Number of Processes 10 Behavior A true parallel effect the job will run in chunks in par allel split across 10 processors 2 5 UNINSTALLATION 15 2 5 Uninstallation Use BioSolvelT Uninstaller component to uninstall all Tools Note that this kind of deinstallation only removes Tools installed by installer component or BioSolvelT web installer component 16 CHAPTER 2 INSTALLATION Trouble Shooting The most commonly seen problem
Download Pdf Manuals
Related Search