User manual CLC Server Command Line Tools
Contents
1. Do readmapping echo we echo Read Mapping SSERVERCMD A read_mapping soliddata destdir refdata O read_mapping_result txt check_return_code Read Mapping echo SPARSECMD f read_mapping_result txt p i c mapping readmap SPARSECMD f read_mapping_result txt p i c Reads check_return_code Read Mapping result parsing rm read_mapping_result txt Quality based Variant Detection echo we echo Quality based Variant Detection SSERVERCMD A quality based_variant_detection create table true min coverage 1 variant in forward rever check_return_code Quality based Variation Detection table SPARSECMD f quality_result txt p s c Reads check_return_code Quality based Variation Detection result parsing rm quality_result txt Export variant table to excel echo we echo Export variant table to excel SSERVERCMD A export e excel_2010_exporter d table O table_export_result txt check_return_code Export variant table to excel file SPARSECMD f table_export_result txt O File check_return_code Export variant table to excel result parsing rm table_export_result txt Workflow completed echo we echo Workflow completed succesfully echo Variant table file Chapter 4 Usage for all commands A complete overview of usage for all commands can be found at http www clcsupport com clcservercommandlinetools current index php manual Usage_all_commands h tml2. 17
3. CLC Server Command Line Tools User manual Manual for CLC Server Command Line Tools 1 7 Windows Mac OS X and Linux April 16 2013 This software is for research purposes only CLC bio Finlandsgade 10 12 DK 8200 Aarhus N Denmark tC bio Contents 1 Introduction 1 1 Installation 1 2 System requirements 2 Basic usage 2 1 Handling passwords 2 2 Data objects data files andthe CLC URL 2020048 22 1 The CLC URL the ID TOM seios amp 4 dow amp Ger Se eo hae bh SE eee we A 2 2 2 The CLC URL Name form sa c ses eke ee eee ee EER ee eG 2 2 3 Indicating local system files or folders 25 2 3 Result files and connecting analyses in pipelines 0 2 4 Executing workflows 2 5 Emptying the recycling bin fora CLC Server File Location 3 Example script 4 Usage for all commands ol Chapter 1 Introduction Welcome to the user manual of CLC Server Command Line Tools The CLC Server Command Line Tools is a command line client for the CLC Genomics Servert The CLC Server Command Line Tools and in particular provide the tools to start analyses and other tasks on CLC Servers including data import and export and utility data operations such as moving renaming and deleting data on the server A typical work flow using the CLC Server Command Line Tools might be 1 Import your sequence data 2 Run analyses such as read mapping SNP detection or
4. This is a shell script that will run on Linux and Mac OS bin bash HERE FE HERE EEE EE FE FE FE AE HEE HE EEE HE FE FE AE EEE EEE EHH EE EH REE ERE HE FE FE AE EE REE EEE EEE EEE EH REE HE HE E EH Example workflow script for CLC Server Command Line Tools 1 6 CLC bio June 2012 For full documentation please visit http clcbio com usermanuals FE HEHE HE HE FE E FE FE HE FE FEAE EEE HEE HEE FE HE E FE FE FE FE EE FE FE FE FE HEE HEH AE FE EE HE FE FE AE HE HEE REBAR E AE AE HE FE FE RHEE HERE E E E E E EE E EE E E SETTINGS Edit before use 444 4E tAE 4E AE AE dE AE EEE EEE EEE THEE HEE EEE AE FE FE AE EEE EEE FE E FE FE EEE E EEE EEE EEE REE HE FE E FE E H 1 Configure your server connection parameters SERVER localhost PORT 7777 USER root PASSWORD default 2 Configure the path to the CLC Server Command Line tools SERVERCMDPATH home user clcservercmdline clcserver PARSECMDPATH home user clcservercmdline clcresultparser 14 CHAPTER 3 EXAMPLE SCRIPT 15 3 Configure High throughput sequencing data import location for import data Use the server web interface to setup a data import location Copy the example files from the data directory to this location Edit the IMPORTPATH variable below to a CLC URL point to this location For more info please visit http www clcsupport com clcgenomicsserver current index php manual Accessing_files_on_writing_to areas_server_filesyst
5. 9 2 2 Data objects data files and the CLC URL In this section we refer to data already in a CLC Server data area as beingin a persistence model This technical term allows us to refer to any area that the CLC Servers or Workbenches recognize as CLC data areas Most relevant to this document are CLC Server File Locations and Database Locations Each Server File Location for example is a single persistence model When running the clcserver command one needs to be able to specify resources inside of or outside of a CLC Server persistence model as well as files and folders on the local machine Files residing on the local system are indicated by using the full or relative path to the file Data objects held in a CLC Server persistence model and files stored in areas configured as Import Export locations for the CLC Server are indicated using a CLC URL CLC data objects within a persistence model can be identified by using two different URL forms one based on the object s name or using its object ID For files in an Import Export area only the name based URL form can be used 2 2 1 The CLC URL the ID form Data resources within persistence models can be referred to using the object ID form of CLC URLs These look soomething like the following cle node04 7777 3123 2131uafda sads 213 sddsa123 5232 Getting the object ID form of a CLC URL There are several ways this can be done 1 Via the Workbench Copy the CLC URL by high
6. FE FE FE FE FE FE FE AE AE AE EEE EEE HH HHH HH HHH HH FE FE FE FE FE HHH Make a directory for DIR echo we echo Make a directory DIR SSERVERCMD A mkdir t DATAPATH n DIR O tmpdir_result txt check_return_code make dir tmpdir SPARSECMD f tmpdir_result txt c clc example check_return_code make dir result parsing rm tmpdir_result txt Make subdirectory in DIR folder for result and data echo we echo Make subdirectory SSUBDIR SSERVERCMD A mkdir t tmpdir n SUBDIR O mkdir_result txt check_return_code Make sub dir destdir SPARSECMD f mkdir_result txt p d c SSUBDIR CHAPTER 3 EXAMPLE SCRIPT 16 check_return_code Make sub dir result parsing rm mkdir_result txt Import solid data echo we echo Import solid data SSERVERCMD A ngs_import_solid f SIMPORTPATH solid_matepair_F3 csfasta f SIMPORTPATH solid_matepair_F3 _ check_return_code Import solid data soliddata SPARSECMD f ngs_import_solid_result txt p i c solid ignorelog true check_return_code Import solid data result parsing rm ngs_import_solid_result txt Import genome echo we echo Import genome SSERVERCMD A import f automaticimport s SIMPORTPATH reference fa destdir O import_result txt check_return_code Import genome refdata SPARSECMD f import_result txt p references c reference check_return_code Import genome result parsing rm import_result txt
7. RNA Seq 3 Optionally export the results to your local disk Another client available to run tasks on the CLC Genomics Server is the graphical CLC Genomics Workbench Below are recommendations for choosing which of these two clients the graphical or the command line to use for your work e For visualization and interpretation of data we recommend the CLC Genomics Workbench The only way to visualize and interpret data when you have worked using the CLC Server Command Line Tools is to export the results into file formats that can be imported into visualization tools e For explorative work we recommend using the CLC Genomics Workbench The numerous parameters are easier to interpret using the graphical interface and selection and man agement of data is more intuitive through this interface for most users In addition the graphical user interface has more constraints to help guide reasonable choices of parame ters and combination of parameters these constraints are not all present in the CLC Server Command Line Tools e For automation and consistency of particular utility in production environments the CLC Server Command Line Tools client is recommended In particular you can script pipelines Like any other client software the CLC Server Command Line Tools would most commonly be installed and used on systems other than the one that the CLC Server software is installed on although there is no restriction meaning that this must b
8. Benefits Human readable and easier for many people when first starting out working with the Command Line Tools or when just running a few commands directly as opposed to via a script CHAPTER 2 BASIC USAGE 11 Drawbacks Any changes to the names of data objects or folders in the persistence model will break the URL 2 2 3 Indicating local system files or folders When importing from a file or files on the local system i e the machinethe clcserver command is being run on or exporting to the local system then just the relative or full path to the relevant file or directory can be provided An example of such a path would be home username somefolder datafile gb In the case where the clcserver command is on the same machine that the CLC Server software is installed on one could choose whether to consider a file as local or as a file that the CLC Server software has access to via an Import Export location In other words one would have the choice whether to provide just the path or the clc serverfile URL form form to identify files on the server machine A key difference here would be that the first would allow for any file on the system for which one has appropriate privileges to be indicated The second allows only for access to files in areas configured as Import Export data locations for the CLC Server 2 3 Result files and connecting analyses in pipelines For each run of clcserver text information is returned providin
9. C URL on the clipboard You can then paste the URL into your command in the terminal 2 2 2 The CLC URL name form The object name form of CLC URLs can be used to refer to data resources within persistence models or to refer to files located on the machine the CLC Server software is running on The first section of a object name form of a CLC URL indicates whether it is referring to a data object in a CLC Server persistence model or to a file stored in an area configured as an Import Export location for the CLC Server These forms are clce server refers to a data object present in a persistence model This part of the URL is then followed by the name of the persistence model the data is located in For example the name of particular CLC Server File Location An example of this form is cle server CLC_Server_Project alignments myAlignment Note that for CLC data in a persistence model you need the name of the data object as seen via the Navigation area of the Workbench not the name of the file holding the data object as seen when listing files using system tools like the commands Is or dir cle severfile refers to a file in an area configured as an Import Export area for the CLC Server This form would commonly be used to point at files containing data that is about to imported into the CLC Server or to indicate a location to export data to An example of this form is cle serverfile mnt data projectl1 s_1_1 sequence txt
10. e the case CHAPTER 1 INTRODUCTION 5 of analyses on the CLC Genomics Server and then use these scripts for processing many data sets in a consistent manner For initial pilot runs it is often helpful to use the exploratory features of the CLC Genomics Workbench to determine quality control and parameter settings and then incorporate these settings into a script using the CLC Server Command Line Tools This user manual begins with installation instructions followed by an explanation of the basics of operating the CLC Server Command Line Tools Then we provide an example script which illustrates various aspects of how to use the analysis tools available on the server 1 14 Installation The CLC Server Command Line Tools can be downloaded from http www clcbio com products clc server command line tools direct download and is available for Windows Mac and Linux You can install the tools on any computer that can connect to your CLC Genomics Server but it makes sense to install them onto the computer that will be used to run the scripts or onto the server computer itself 1 2 System requirements The system requirements of CLC Server Command Line Tools are these e Windows XP Windows Vista Windows 7 Windows 8 Windows Server 2003 or Windows Server 2008 e Mac OS X 10 6 or later However Mac OS X 10 5 8 is supported on 64 bit Intel systems e Linux Red Hat 5 or later SUSE 10 or later e 32 or 64 bit e 256 MB RAM req
11. em html For information about CLC URLs please visit http www clcsupport com clcservercommandlinetools current index php manual Referring_files_CLC_URL html MPORTPATH clc serverfile tmp cmdline 4 Configure data paths for saving results Use the server web interface to configure a file system location for data storage Edit the DATAPATH variable below to a CLC URL pointing to this location For more info please visit http www clcsupport com clcgenomicsserver current index php manual Adding_file_system_location html For information about CLC URLs please visit http www clcsupport com clcservercommandlinetools current index php manual Referring_files_CLC_URL html DATAPATH clc server test DIR clc example SUBDIR workflow example FUNCTIONS AEE AE AE it tt tt ta a AE aE a aE aE EH FE AE AE aE EE aE AE E aE HE EE EE FE FE aE AE aE EEE aE EE aE aE HE a aE HEE EEE aE aE aE EEE EEE function check_return_code return_code cmdname 1 echo Return code Sreturn_code if return_code ne 0 then echo we echo Error during Scmdname echo Terminating script exit 1 fi COMMANDS iat AEAEE at tat AE AE FE FE aE aE aE aE aE aE AE aE HE aE aE HE aE FE aE aE AE FE FE HE EEE aE EE HE EEE EEE aE EEE EEE EEE EEE EEE H SERVERCMD SERVERCMDPATH S S SERVER P SPORT U SUSER W SPASSWORD PARSECMD PARSECMDPATH WORKFLOW SCRIPT tt dE et tt tt tt FE FE HF HEHEHE E E E E EE AE FE FE FE
12. eturns the CLC URL for files where a match to that text has been found in the Name field The Name field will contain the name of the input data along with a description of the type of data held in that file location In the case above you would probably search for the trimmed reads to use for further analysis which could be done with a command like this clcresultparser f result txt c trimmed Here the following text would be returned cle 127 0 0 1 7777 268177574 YCAAAAAAAAAAAAP C67 3b0db8c5e724f 5d66a991 12d75090d93 7f fF The options for the clcresultparser program are f lt name of result file to parse gt This option is required c lt text to search for gt Text to search for in the Name field of the result file If nothing is found the exit code is 1 n lt text that should not match gt Text that should not be contained in the Name field of the result file r lt regexp gt A Java regular expression used for matching the name of the output See http java sun com docs books tutorial essential regex index html ignorelogs lt boolean gt By default all analyses produce log files You can provide false as the argument to this option to stop log files from being returned This is equivalent to excluding all names ending with log or log with a number suffix The latter are generated when there is more than one log file in the same folder p lt prefix text gt When more than one match is found the data locat
13. g a summary of the steps taken and the locations in ID form of any files generated The file containing this information will by default be created in the current directory and will be called result txt You can use the 0 option for the clcserver command if you wish to specify an alternative file to be written to An example of contents in a typical result file is shown below In this case the file that was generated after running the the trim algorithm using a sequence list called reads as input The result file lists the three files that were produced CHAPTER 2 BASIC USAGE 12 if Name reads trimmed lcUrl cle 127 0 0 1 7777 268177574 YCAAAAAAAAAAAAP c67 3b0db8c5e724f 5d66a991 12d75090d93 7ffFE MN ame reads report lcUrl cle 127 0 0 1 7777 26817757 4 ADAAAAAAAAAAAAP C67 3b0db8c5e724f 5d66a991 12d75090d93 7ffF m o me Trim Sequences log lcUrl cle 127 0 0 1 7777 26817757 4 CAAAAAAAAAAAAAP C67 3b0db8c5e724f 5d66a991 12d75090d93 7ffF S20 2 Ss SQ SS SO When creating pipelines stitching together several analyses you parse the result file to get the location of the data produced which is needed as input for the next algorithm The result file is just a text file but it can still be a challenge to parse it to get the necessary CLC URLs Thus we provide a tool called clc_result_parser to help with this It searches the result file for a text expression you provide and r
14. he clcserverkeystore program has two other parameters that can be used d lt token gt This will delete the individual token provided as a parameter deleteAll This will delete all the tokens in the user profile The first section of the diagram below illustrates the process of logging into the server using a clear text password The second section illustrates the process of generating a password token and storing it in the keystore followed by a section showing how the token is substituted by the CLC Server Command Line Tools with the real password when initiating the connection to the server host server com PES SSS SSS user bob prea een SST nra F port 7777 Command password secret USER bobs SaaS SSeS gt Line l 4S SS SSS Seger e gt server com 7777 password secret Tool 4 4 Password token generation 4 generate 2 Set password secret Command Line Tool xxx password key xxx oe R Se CAIHMAAAAAAAAAPcb769377f4 hese Sse aos KR KR KR Rk e S see SSS KeyStore 4 Password token invocation host server com port 7777 user bob She RSH Ste SSH Ss password Command user bob PERSE SSR RSE ESE eSSE CAITHMAAAAAAAAAPcbh7 Line password secret 6937784 O gerea besos gt Tool 0 SSP SHS sees gt server com 7777 l Pelee Se Hee ne 4 CHAPTER 2 BASIC USAGE
15. he column width of the help output D lt boolean gt Enables debug mode when set to true providing more elaborate output and error messages H Display general help instructions V Display the version number of CLC Server Command Line Tools 2 1 Handling passwords To help you avoid sending your server login password in clear text across the network we provide the clcserverkeystore tool This enables you to convert your password to a token which is stored and can be interpreted by the CLC Server Command Line Tools when logging onto the server The token is encrypted and saved with the user profile on the computer running the CLC Server Command Line Tools You can generate a password token using the following command clcserverkeystor generat You will be prompted for the password After you have typed the password press the Enter key The password token is then returned on screen It will be a long string of text that you should save somewhere to refer to for future use So if we say that user bob has password secret and has generated a password token CATHMAAAAAAAAAPcb769377f4 then he could enter either of the following two commands to connect to his server The first passes the password in plain text The second passes it as an encrypted token CHAPTER 2 BASIC USAGE cleserver S server com U bob W secret clcserver S server com U bob W CAIHMAAAAAAAAAPcCb769377 4 If the token needs to be deleted t
16. ing bin for a CLC Server File Location Each CLC Server File Location has a recycling bin where files that users delete are put Only members of the administrator group as defined on the Genomics Server can empty the recycle bin associated with Genomics Server file locations This is because the recycle bin is a shared location for any given Genomics Server file location and many sites do not want all users to be able to access it directly that is to be able to view things or delete other people s data One can avoid the need to periodically go in and manually empty recycling bins by setting up a script that is run as a cronjob which includes a command of the following form cleserver S lt serverinfo gt P lt portnumber gt U lt adminusername gt W lt password or tok Above SLOCATIONNAME would be replaced byt the name of the CLC Server File Location you wish to empty the recycling bin of Chapter 3 Example script In this section we present an example script showing a typical work flow consisting of the following steps e Import of NGS data e Read mapping e Variant detection The result of the variant detection is then exported to the local file system in Excel format The script is intended only as an example Hopefully it will be possible for you to modify it to fit your purposes You can download the script including data from http www clcbio com files CLCServerCommandLineTools 1 7 example workflow zip
17. ions for all matches will be output as a space separated list By supplying a prefix string you can stipulate what character s to separate the list using E g If you need to send several files output by the clcresultparser command as arguments to i options for the next analysis simply provide i as as the argument for the p flag CHAPTER 2 BASIC USAGE 13 e lt integer gt The number of CLC URLs that are expected to be returned If this is not the number of results files that match the search string the command will return with exit code 10 This option is designed for use in scripts where you will wish to carry out validation steps are you proceed through the pipeline On the command line you check the error code returned by the previous command by typing echo C lt integer gt Specifies the column width of the help output 2 4 Executing workflows It is possible to execute workflows installed on the server Workflows are described in detail in the user manual of of CLC Genomics Workbench and CLC Genomics Server at http www clcbio com usermanuals Executing workflows is similar to executing algorithms and the installed workflows will be listed when the A is omitted Parameters that are open for change on execution are displayed when the workflow is specified for the A option Please note that the parameter names have name of the workflow element pre pended to make sure they are always unique 2 5 Emptying the recycl
18. lighting the data object by clicking on the object in the Workbench Navigation area to select it and then using the keyboard short cut Ctrl C Then use Ctrl V to paste the URL into a shell window text editor or similar See figure 2 1 2 Using the CLC Server web administrative interface Select a data object from the tree browser on the left hand side of the browser window and then select the Element info tab in the main area of the browser window Click on the link to CLC URL This shows two versions of the CLC URLA one using the name and one using the object ID 3 Take the object ID from within the text ouptut file generated using the O flag of the clcserver command This would be the common route when running a series of commands via a script Benefits The ID form of a CLC URL is impervious to changes to the name of a data object or the folders the data resides in That is such changes do not affect a data object s ID Drawbacks They are unreadable by humans CHAPTER 2 BASIC USAGE 10 1 A a a EA presentation E server _data H E Illumina GA H E 454 ay RNA seq Xc NC_000019_ selection braini 2E braint E show Ctrl O braink N k E brainf R H E old Show gt E brain2 Toolbox gt uhri mC the Cut Ctrl X Qy lt enter search term Si Copy Ctrl C ice 5 Paste Ctrl V S Alignments and Tr Delete Delete Figure 2 1 Copying a data object in the workbench will put the CL
19. ommand to be executed on server gt If you supply the A flag with a program name but do not provide the required flags for that program then a listing of the flags for that program will be returned For example a command of a form like CHAPTER 2 BASIC USAGE 7 clcserver S server com U bob W secret A read_mapping would return the full list of parameters for the read_mapping function including the pos sible values and descriptions This information for each command is also available in the online manual at http www clcsupport com clcservercommandlinetools current index php manual Usage_all_commands html1 An optional flag when working on the command line but important when working with scripts is 0 lt filename gt The name of a file to be created to hold a summary of steps carried out on the server and data locations of the results generated The data locations are of a form that can be used by downstream CLC commands See section 2 3 for information about parsing this file By default this file is placed in your working directory If you do not provide this flag this data will be written to a file called results txt For those working with the CLC Grid Integration Tool you can run import and algorithm commands through your grid nodes by adding the following flag to your clcserver command G lt grid preset name gt Other optional flags available for the clcserver command are C lt integer gt Specify t
20. uired e 512 MB RAM recommended e 1024 x 768 display recommended You will also need a running version of CLC Genomics Server No additional license is required for running the CLC Server Command Line Tools Chapter 2 Basic usage Once installed there will be three programs present in the installation folder e clcserver the key program It is used to run all the commands that communicate with the server e clcresultparser used to parse data locations from particular text files generated during clcserver runs This command is most useful when connecting analyses in a scripting pipeline See section 2 3 e clcserverkeystore a helper tool for enabling passwords to be handled securely see section 2 1 The clcserver program requires the following four flags which provide information about the connection to the server S lt hostname or IP address of the server gt P lt port the server runs on gt When omitted port 7777 is used which is the default for server installations U lt user name gt The username used to log into the server W lt password or token gt See section 2 1 for how to avoid entering passwords in clear text If you run the clcserver command with the above parameters and nothing else then a list of all commands that can be run on the server will be returned For example clcserver S server com U bob W secret The commands to be run on the server are supplied with the flag A lt c
Download Pdf Manuals
Related Search