Home

UNICORE Commandline Client: User Manual

1. 7 8 2 Specifying a project If the system you re submitting to does accounting you can specify the account or project you want to charge the job to using the Project tag Project my_project 7 8 3 Specifying the user email for batch system notifications Some batch systems support sending email upon completion of jobs To specify your email use User email foo bar org Hint if you want to explicitely switch off the email notification use NONE as email value This might be necessary because older UNICORE server versions try to use the email address from your certificate if present 7 8 4 Specifying the job name The job name can be set simply by Name Test job 7 8 5 Specifying the status check interval for batch mode Once a job is started it is often not useful to check its status every few seconds because the job might be running several minutes or more Especially in batch mode it can reduce the load on the servers if the update interval is chosen longer This can be achieved by using the following setting this only affects batch mode Update interval 60 only check once a minute default is one gt second 7 8 6 Specifying the lifetime of the job If you want to specify a lifetime of the job and not rely on the server default you can use the lifetime attribute Lifetime 12h sec min h d UNICORE Commandline Client User Manual 34 8 Data management functions UCC offers access
2. Property name Type Default value mandatory Description truststore crlDiskCachePath filesystem path Directory where CRLs should be cached after downloading them from remote source Can be left undefined if no disk cache should be used Note that directory should be secured i e normal users should not be allowed to write to it Not used for Openssl truststores truststore crlLocations list of properties with a common prefix List of CRLs locations Can contain URLs local files and wildcard expressions Not used for Openssl truststores runtime updateable truststore crlMode REQUIRE IF_VALID IGNORE IF_VALID General CRL handling mode The IF_VALID setting turns on CRL checking only in case the CRL is present truststore crlUp dateInterval integer number How often CRLs should be updated in seconds Set to negative value to disable refreshing at runtime runtime updateable truststore ocspCacheTtl integer number 3600 For how long the OCSP responses should be locally cached in seconds this is a maximum value responses won t be cached after expiration truststore ocspDiskCache filesystem path If this property is defined then OCSP responses will be cached on disk in the defined folder truststore ocspL ocalResponders lt NUMBER gt list of properties with a common prefix Optional list of local OC
3. UNICORE Commandline Client User Manual Table 4 continued Property name Type Default Description value mandatory truststore integer number 600 How often the truststore updatelnterval should be reloaded in seconds Set to negative value to disable refreshing at runtime runtime updateable Directory type settings truststore integer number 15 Connection timeout for directoryConnect fetching the remote CA ionTimeout certificates in seconds truststore filesystem path Directory where CA directoryDiskCac certificates should be hePath cached after downloading them from a remote source Can be left undefined if no disk cache should be used Note that directory should be secured i e normal users should not be allowed to write to it truststore direc PEM DER PEM For directory truststore toryEncoding controls whether certificates are encoded in PEM or DER truststore direc list of List of CA certificates toryLocations properties with locations Can contain a common URLs local files and prefix wildcard expressions runtime updateable Keystore type settings truststore string The keystore type jks keystoreFormat pkcs 12 in case of truststore of keystore type truststore string The password of the keystorePassword keystore type truststore truststore string The keystore path in case of keystorePath truststo
4. Recent servers do not require this trick and you should be able to directly execute your uploaded executable without requiring a helper script UNICORE Commandline Client User Manual 24 6 4 6 Scheduling job submission to the batch system Sometimes a user wishes to control the time when a job is submitted to the batch queue for example because she knows that a certain queue will be empty at that time Note This feature only works with server release 6 4 0 or higher To schedule a job you can either use the S option to the ucc run command tee pa S VIDS aa Alternatively you can specify the start time in your job file using the Not before key word Not before 12230 In both cases the specified start time can be given in the brief HH mm hours and minutes format shown above or in the full ISO 8601 format including year date time and time zone Not before 2011 12 28 112 30 00 OZN 6 5 Executing a command If you just want to execute a simple command remotely i e without data staging resource specifications etc you can use the exec command This will run the given command remotely similarly to ssh and print the output to the console You can specify the site with the s option If you do not specify the site a random site will be chosen UNICORE will run the command on the login node it will not be submitted to the batch system For example try ucc exec bin date Wa
5. use at your own risk Groovy example delete all your jobs x x remove all jobs import UNICORE X client classes import de fzj unicore uas client x import eu unicore security wsutil client authn DelegationSpecification helper kills a job with the given status SUCCESSFUL RUNNING def kill job statuscode i GOD mS chaUls r EOSTO gt ELSE cols yo Cestr rOy iterate over TSSs and remove all jobs def lister new de fzJ unicore uas lookup Sitelister registry configurationProvider lister each it jobs each messageWriter message Job at it address stringValue securityProperties configurationProvider getClientConfiguration it DelegationSpecification STANDARD kill new JobClient it securityProperties SUCCESSFUL Groovy example list available storages x x list available storages import de fzj unicore uas client x import eu unicore security wsutil client authn DelegationSpecification import javax xml namespace QName UNICORE Commandline Client User Manual 55 porttype of storage service def SMSPORT new QName http unigrids org 2006 04 services sms StorageManagement method to extract storage name from a storage client def findName epr securityProperties configurationProvider getClientConfiguration gt epr DelegationSpecification STANDARD sms new StorageClient epr securityProperties return sms resource
6. Gea Se a ee Ee ea 13 A6 Using IMYRIORY asni bane oe TA EPR ORES EERE YU 13 Aap ICHENCOPUONS s 2 25 64 kG oie oe Sa AAA A 14 4 8 Other options seios sies se ee Rhee a Re ee pe o os 17 5 SAML PUSH support 18 SA IMrOUCUON si she XA Segoe APA gs ee 18 3 2 Basie Usage ae ee ea le ee RA eh ae oe e ae 18 53 Attribute filtering se s ceci boa daa ee wr ek ea da 19 5 4 Rules for multiple filters 2 e esre ae ea ess 20 UNICORE Commandline Client User Manual 6 10 Running jobs 6 1 Introduction lt 2 s seie aci t A d Goals ail ae ee al eed 6 2 Options Overview i lt i 2256 ra o sha ee baee ent eee he te 6 3 Resource brokers oes a eG bak a a ee A RR a ie 6 4 Processing jobs asynchronously ooo 6 3 Executing acomma d ooo a AA Job description format OA SIE DAME ei e e A A e ic ada 7 2 Specifying the application or executable ooo 7 3 Arguments and Environment settings p s sec ecc pesi ap iate TA Application parameters sins e a aA a Le 10 Jobdata managements e ee a 224 a e a a A a e eG be TO RESOURCES eepo e a Gad A A A o eee Ba eo TI Execution environments s sica e e Aes ee osa E 7 8 Miscellaneous options s s s e e ee babe ee Ra eee Data management functions 8 1 Specifying remote locations s o se seses aae ee ee ee wl eee 3 2 Data movement saai s s eop dla a oo we a ea 83 gt Handling directories 04 5 8356 4 tr A bane be 8 84 Finding data 2 2 45 4080 ra baw ae B
7. and received by UCC In your properties file set log outgoing messages log outgoing true log incoming messages log incoming true which will log the messages on INFO level 17 1 5 How can generate a proxy cert and add it to my message in order to use e g GridFTP In your properties file add enable proxy cert out handler client outHandlers de fzj unicore uas security ProxyCertOutHandler which will add a handler that creates a proxy cert and adds it to the message 17 2 Usage 17 2 1 Can I use multiple registries with UCC 1 Yes Simply use a comma separated list of URLs for the c option However you may only use a single key truststore so all registries and sites listed in them must accept the same security credentials 17 2 2 Can l upload and execute my own executable Yes Check Section 6 UNICORE Commandline Client User Manual 57 17 2 3 Can I use UCC to list the contents of the registry Using the wsrf command and the UNIX grep utility this is very easy for example ucc wsrf getproperties https localhost 8080 DEMO SITE services 4 Registry res default_registry grep Address will list the addresses of all services registered in the registry 17 2 4 How can use plain JSDL files instead of a u JSON file for job submission Add the j option when submitting a job 17 2 5 I get strange errors related to security Please read the general UNICORE FAQ on www unicor
8. end point reference schema See below the contents of the sample endpoint reference file lt wsa EndpointReference xmlns wsa http www w3 org 2005 08 addressing gt lt wsa Address gt https localhost 8080 DEMO SITE services BESFactory res default_bes_factory lt wsa Address gt lt wsa EndpointReference gt In the above XML snippet under the Address tag you must specify the URL of a target BESFactory service For the sake of convenience here is an XML infoset representation taken from the WS Addressing specification lt EndpointReference gt lt Address gt xs anyURI lt Address gt lt ReferenceParameters gt xs any lt ReferenceParameters gt lt Metadata gt xs anyx lt Metadata gt lt EndpointReference gt 12 2 Running and monitoring OGSA BES jobs UCC provides an easy to use command for submitting jobs on OGSA BES complaint endpoints To send ajob read from a JSDL file ucc bes submit job 3 hellompi xml s bes 3 v Alternatively the job can be submitted using a BESFactory URL or endpoint reference file path ucc bes submit Job J hellompi xml s https example3 com services BESFactory v or UNICORE Commandline Client User Manual 48 uce bes submit job J hellompi xml s file tmp bes jugene xml v The JSON job description Section 7 can also be used although only a subset of JSON constructs are supported for the OGSA BES extensions Users c
9. executed server side and UCC is used only for submitting managing data and getting results 10 2 Command overview The following commands are provided More details and examples follow below e workflow submit submit a workflow file e workflow info list information about workflows e workflow trace gather performance data from the workflow execution 10 3 Basic use To check the availability of workflow services issue the following command ucc system info 1 This should show at least an accessible workflow engine and service orchestrator The distribution contains some example workflow files in the lt gt directory that you can edit and submit ucc workflow submit yourworkflow swf which will submit the workflow and print the address of the workflow to standard output To get the workflow status ucc workflow info lt workflow_address gt To list all your workflows you can use the lt gt command without an explicit workflow address ucc workflow info 1 UNICORE Commandline Client User Manual 43 10 4 Managing workflow data During workflow execution data files will be produced that the workflow system will move to a location on the Grid that is accessible for the individual jobs Usually this location is created automatically by UCC before the workflow is submitted using a special UNICORE service called a storage factory If you want to influence this decision UCC allows to select the storage fact
10. in a script activity lt jsdl DataStaging gt lt 3jsdl CreationFlag gt overwrite lt 3jsdl CreationFlag gt lt 3jsdl FileName gt input sh lt 3jsdl FileName gt lt jsdl Source gt lt jsd1 URI gt c9m WORKFLOW_ID input sh lt jsdl URI gt UNICORE Commandline Client User Manual 44 lt jsd1 Source gt lt 3jsdl DataStaging gt The workflow system will resolve the name at runtime and your file will be used This allows you to group your files by workflow ID 10 4 2 Downloading output files You can use the usual get file command to download files using the global IDs used by the workflow engine Hint the workf 1ow in fo command will list the files that are produced by the workflow 10 5 More 10 5 1 Tracing The trace functionality of the workflow engine allows to retrieve some performance data try ucc workflow trace lt your_workflow_address gt 11 Batch processing The batch command allows you to run many jobs without having to start UCC each time You can control how many jobs should go to which site This allows efficient job processing while putting some load on the client machine If you need to take the client offline you should consider using the workflow system instead which also allows efficient high throughput processing Assume you have a bunch of jobs in UCC s job description format Section 7 stored in a directory jobs The output should go to a directory out You can run them all through UCC u
11. lt job_desc gt lt job_desc_2 gt will retrieve the status of the given jobs If not given on the command line a job ID will be read from the console UNICORE Commandline Client User Manual 23 6 4 3 Download results To get stdout stderr and other files you have marked for export in your job description do ucc get output o lt outdir gt lt job_desc gt Here the option o specifies the directory where to put the output by default the current directory is used As before the job address can also be read from the console 6 4 4 Referencing a job by its EPR Endpoint reference In case you want to check on a job not submitted through UCC or in case you do not have the job descriptor file any more you can also refer to a job given its EPR For example the list jobs command will produce a list of all job EPRs that you can access Note that in this case UCC will only retrieve stdout and stderr files To download other result files you ll have to use the datamovement functions described in Section 8 6 4 5 Uploading and executing an executable To upload and execute a file on a remote server you might need a small helper script to make the uploaded file executable and run it bin sh chmod x myapp myapp Your ucc job description would then look as follows Executable bin sh Arguments helper sh eMC rates aa From helper sh To helper sh Hrom Maa Mos Mica l Note
12. set the timeout when establishing a connection to 3 seconds you would use client http connection timeout 5000 4 8 Other options The following table lists other options that are more rarely used Table 7 Other options for the UCC Property name Description contact registry Do not attempt to contact the registry even if one is configured UNICORE Commandline Client User Manual 18 5 SAML PUSH support 5 1 Introduction UCC supports the SAML PUSH mode of authentication In this mode the user is authen ticated by an attribute assertion signed by a trusted third party The third party is a Virtual Organisation VO SAML 2 0 server such as UVOS or VOMS 5 2 Basic usage First retrieve an attribute assertion from VO server scoped for a particular group you are in and save it into a file ucc save attributes J https uvos example com 2443 G vo group O assertion xml If the VO server has been registered in the registry you may use auto instead of server s URL this is possible only in case of UVOS ucc save attributes J auto G vo group 0 assertion xml Then you may use this assertion to get access to the server Most UCC commands support this for example you may connect to the server using this assertion ucc connect A assertion xml You also may combine those two steps into one ucc connect J https uvos example com 2443 G vo group or assuming that UVOS server is in the regis
13. setting by replacing the value by a sweep specification This can be either a simple list Arguments Pa eek oat aml ae fo ma A o Ce aul l or a range UNICORE Commandline Client User Manual 27 Arguments WOW Bronk I Wos 3 Sess a jt where the From To and Step parameters are floating point or integer numbers 7 4 Application parameters In UNICORE parameters for applications are often transferred in the form of environment variables For example the POVRay application has a large set of parameters to specify image width height and many more In UCC you can specify these parameters in a very simple way using the Parameters keyword ApplicationName POVRay Parameters WIDTH 640 HEIGHT 480 DEBUG Note that an empty parameter which does not have a value needs to be written with an explicit empty string due to the limitations of the JSON syntax 7 4 1 Parameter sweeps You can sweep over application parameters by replacing the parameter value by a sweep speci fication The replacement can be either a simple list Parameters WIDTH Values 240 480 960 ha or a range Parameters WIDTH From 240 To 960 Step 240 Ihe where the From To and Step parameters are floating point or integer numbers UNICORE Commandline Client User Manual 28 7 5 Job data management In general your job will require data files either from your client machine or fr
14. the only location required In case when credential is provided in two files it is the certificate file path UNICORE Commandline Client User Manual Table 3 continued Property name Type Default value mandatory Description credential format jks pkes12 der pem Format of the credential It is guessed when not given Note that pem might be either a PEM keystore with certificates and keys in PEM format or a pair of PEM files one with certificate and second with private key credential password string Password required to load the credential credential keyPath string Location of the private key if stored separately from the main credential applicable for pem and der types only credential keyPassword string Private key password which might be needed only for jks or pkcs12 if key is encrypted with different password then the main credential password credential keyAlias string Keystore alias of the key entry to be used Can be ignored if the keystore contains only one key entry Only applicable for jks and pkcs12 Table 4 Truststore properties Property name Type Default Description value mandatory truststore ALLOW ALLOW Controls whether proxy allowProxy DENY certificates are supported truststore type keystore mandatory The truststore type openssl to be set directory
15. to all the data management functions in UNICORE You can upload or download data from a remote server initiate a server to server transfer create directories and so on 8 1 Specifying remote locations Remote locations can be specified in two ways The first way is to use a URI that includes protocol storage server and filename for example BFT https mygateway 8080 SITE services StorageManagement res default_storage file which specifies a file named file on the storage instance https mygateway 8080 SITE services StorageManagement res using the BFT protocol Paths are relative to the storage root not the root of the actual file system This explicit format is sometimes inconvenient so you can use a shorter more intuitive format This is also a URI but you need to know only the name of the virtual site target system and the storage or job id For example unicore6 SITE Home file protocol PROTOCOL or shorter u6 SITE Home file protocol PROTOCOL This will resolve the current user s Home storage at the target system named SITE Note that if you do not specify the protocol the BFT protocol will be used as default You can also refer to ajob s working directory on a given site For this you will need the unique ID of that job which you can get for example using the list jobs command For example u6 SITE 1f3bc2e2 d814 406e 811d e533f8f7a93b outfile refers to the file outfile in t
16. will print a list of jobs actually their addresses with their respective status RUNNING SUC CESSFUL etc 4 Common options to UCC The following table lists the options understood by most UCC commands Most commands have additional options You can always get a summary of all available options for a command by calling UCC with the h or help option for example ues ipgiceld liSiho Since it is not possible to give all the required options on the commandline it is mandatory to create a preferences file containing e g your settings for keystore registry etc Table 1 Common options for the UCC Option short and long Description form 0 gt Properties file containing your preferences By default a configuration file userhome ucc preferences is checked lt Properties_file gt k authenticati Authentication method to use default X509 onMethod lt auth gt o output Directory for any output produced default is the current lt Output_dir gt directory r registry The comma separated list of URLs of UNICORE registries lt List_of_Registry _URLs gt v verbose Verbose mode h help Print help message y 7 with timing Timing mode UNICORE Commandline Client User Manual 6 4 1 User attributes and VOs If you have multiple user IDs or are a member of multiple Unix Groups on the target system you may wish to control the user atttributes that are used when
17. M directoryConnectionTimeout 100 directoryDiskCachePath tmp crlLocations 1 trust dir crl crlLocations 2 http caserver crl pem UNICORE Commandline Client User Manual 13 truststore crlConnectionTimeout 200 truststore crlDiskCachePath tmp Java keystore used as a trust store truststore type keystor truststore keystorePath some dir truststore jks truststore keystoreFormat JKS truststore keystorePassword xxxxxx OpenSSL trust store truststore type openssl truststore opensslPath truststores openssl truststore openss1NsMode EUGRIDPMA_GLOBUS_REQUIR truststore allowProxy ALLOW truststore updatelnterval 1234 truststore crlMode IF_VALID 4 5 Using Unity If your Grid installation is using the Unity identity management service see http www unity idm eu you can setup UCC to use Unity In this case you do not need a private key only a truststore UCC is configured for Unity using the following properties authenticationMethod unity unity address https lt host gt lt port gt unicore soapidp saml2unicoreidp soap AuthenticationService unity username lt your Unity username gt unity password lt your Unity password gt At minimum you must specify the authenticationMethod and unity address parameters Password and username are optional if you do not specify them they will be queried interac tively In some installations you can use a so called OIDC Bearer token instead of use
18. PropertiesDocument storageProperties fileSystem name list storages from registry registry listAccessibleServices SMSPORT each name findName it messageWriter message Storage lt name gt at it address stringValue list storages attached to target systems def lister new de fzJ unicore uas lookup Sitelister registry configurationProvider lister each it storages each name findName it messageWriter message Storage lt name gt at it address stringValue 17 Frequently asked questions 17 1 Configuration 17 1 1 Dolreally have to store my password in the preferences file Isn t this insecure Putting the password in a file or giving it as a commandline parameter can be considered inse cure The file could be read by others and the commandline parameters may be visible in for example in the output of the ps command Thus UCC will simply ask for the password in case you did not specify it UNICORE Commandline Client User Manual 56 17 1 2 How can I enable more detailed logging UCC uses log4j by default the configuration is done in lt UCC_HOME gt conf logging properties You can edit this file and increase the logging levels choose to log to a file or to the console etc 17 1 3 How can I set the HTTP connection timeout In your properties file set client http connection timeout lt timeout in milliseconds gt 17 1 4 How can I log the SOAP messages sent
19. RE Commandline Client User Manual 22 6 3 Resource brokering If no site is specified upon submission UCC will select a matching site where the requirements resources application and execution environments are met There are two types of brokers available which can be selected using the B or broker option e LOCAL brokering is done by UCC itself e SERVORCH default brokering is done by the service orchestrator component of the UNICORE Workflow system The SERVORCH broker requires an accessible UNICORE workflow system in version 6 6 0 or later If this is not available UCC will fall back to the LOCAL broker As usual you can set this option in your UCC preferences file 6 4 Processing jobs asynchronously In case of long running jobs you will want to run the job asynchronously i e just submit the job stage in any files and start it in order to get the results later UCC supports this of course The basic idea is that when submitting a job in asynchronous mode a job descriptor file is written that contains the job s address and any information about export files 6 4 1 Asynchronous submission Use the a flag when submitting a job es abia lt 3olo ze gt This will submit the job stage in any local files start the job and exit A job descriptor file ending in job will be written to your configured output directory 6 4 2 Get the status of particular jobs The command ucc job status
20. SA BES functions Assuming you have successfully installed UCC Section 2 this section shows you how to man age and monitor jobs on OGSA BES services using UCC The set of commands not only sup ports the UNICORE implementation but may also work with implementations in other Grid middlewares compliant with OGF s OGSA BES specification 12 1 OGSA BES Setup In UNICORE style users are required to provide a Registry URL inside the preferences file For BES users it is not always the case that an endpoint is advertised via a UNICORE Registry Therefore the configuration options allow user to modify this behaviour contact registry true false UNICORE Commandline Client User Manual 47 Users who whish to disable UCC calling the registry can set the contact registry option to false By default the contact registry option is true When setting contact registry to false OGSA BES users must provide at least one BESFac tory URL using the following format bes 1 https sitel com services BESFactory bes 2 https site2 com services BESFactory bes 3 https site3 com services BESFactory bes 4 file tmp bes jugene xml bes 5 tmp bes juropa xml If the contact registry option is set to false and no OGSA BES URL is specified UCC will report an error To use an XML endpoint reference EPR read from a file for contacting a BESFactory service the contents of a EPR file must validate against the WS Addressing s
21. SP responders UNICORE Commandline Client User Manual 12 Table 4 continued Property name Type Default Description value mandatory truststore REQUIRE IF_AVAIL General OCSP ckecking ocspMode IF_AVAILABLE ABLE mode REQUIRE should IGNORE not be used unless it is guaranteed that for all certificates an OCSP responder is defined truststore integer number 10000 Timeout for OCSP ocspTimeout connections in miliseconds truststore CRL_OCSP OCSP_CRL Controls overal revocation revocationOrder OCSP_CRL sources order truststore true false false Controls whether all revocationUseAll defined revocation sources should be always checked even if the first one already confirmed that a checked certificate is not revoked 4 4 Trust store examples Here are some examples for commonly used trust store configurations Directory trust store with a minimal set of options truststore type dire CEOHY truststore directorylLocations 1l trust dir x pem Directory trust store with more options truststore type dire E PUSUELOLEOS gt truststore updatelnt ELQEY allowProxy DENY rval 1234 TTUSUSCOTE CRUST CORS CAUSE LODE EXUstTSstores Crust store Lrustsrore LPuetetore Lrustarore truststore crl crlUpdate ode R Interval 400 EQUIRE directoryLocations 1 trust dir pem directoryLocations 2 http caserver ca pem directoryEncoding PE
22. The tell UNICORE that this is OK you must set a ReadOnly flag on the import MIKE S s I tell UNICORE that the file can be sym linked if possible UNICORE Commandline Client User Manual 29 From work data fileName To uspaceFileName ReadOnly lt lt Mitra Onur hy Note UCC supports simple wild cards and for importing exporting LOCAL files and for remote files if the server is version 7 x or later Wildcards do not work for server to server imports and exports on 6 x servers 7 5 2 Importing files into other storage locations A UNICORE site may supports multiple storages for example a TMP or SCRATCH directory To instruct the server to stage in a file into such a storage the Filesystem tag may be used For example to stage in a file into SCRATCH space the following Imports definition can be used mapos nl import a file from a remote storage into the SCRATCH space on the target resourc From u6 DEMO SITE Home work data fileName To fileName We iollbesysiceile O CRATE HA Sweeping over a stage in file You can also sweep over files i e create multiple batch jobs that differ by one imported file To achieve this replace the From parameter by list of values for example MISO SS From u6 DEMO SITE Home work data filel u6 DEMO SITE Home work data file2 u6 DEMO SITE Home work data file3 l To f
23. UNICORE Commandline Client User Manual UNICRE BE RA E EUROPEAN MIDDLEWARE INITIATIVE UNICORE COMMANDLINE CLIENT USER MANUAL UNICORE Team Document Version 1 0 0 Component Version 7 4 0 Date 13 07 2015 This work is co funded by the EC EMI project under the FP7 Collaborative Projects Grant Agreement Nr INFSO RI 261611 UNICORE Commandline Client User Manual Contents 1 Overview 1 2 Installation and configuration 1 2A PIETEQUISUES a oe oe ee eee EER ESE we ee ee e 1 22 Download serio pe a Se be ek ERR oes 4 Y 1 23 Installation and configuration 5 42424 pae 0456 e 2 24 Preferences MMe oos a rant ea ee au eae e e a ke e a AL ea 2 2 MOSS 24 40 66444 244 ea Ghia wet e ewan be pee Ph he oe 3 2 6 Installing UCC extensions so cce a Oe we ek ee Ee eS Le 3 2 7 Tes ng the installation s sc e e eb ea eae ee ee 3 3 Getting started with UCC 3 Sol Get ng help sab be See EA Sa DREHER S Ble Od bo 4 3 2 GOMMECUNE grag e hoe A RM as Ae ROR RO Re Bg ae 4 Sus Listavailable SES 4 sk ee ee ek Rw ee ee e aoe a 4 3 4 RUMMING your stjo s ua e A A ie 4 30 Listing YOUP IONS se pe a Seep so na AA Soe eee 5 4 Common options to UCC 5 4 1 Usetattributes and VOS sos c 500 maod ew e E e al Oe A eae 6 42 Configuration file cios src dr Rd E a 6 4 3 Credential and truststore Options ec s e smaa Eae e BY ee Re Se es 7 aA Muststore examples e 2 5 oe ee ra Ae BPS ee 12 A3 Usine Unity s 24 5 4 a eo Re pd Sk we Be
24. alse false Whether HTTP basic httpAuthnEnabled authentication should be used client string empty Password for use with httpPassword string HTTP basic authentication if enabled client httpUser string empty Username for use with string HTTP basic authentication if enabled client string empty Space separated list of inHandlers string additional handler class names for handling incoming WS messages UNICORE Commandline Client User Manual 15 Table 5 continued Property name Type Default Description value mandatory client integer number 3 Controls how many times maxWsCallRetries the client should try to call a failing web service Note that only the transient failure reasons cause the retry Note that value of 0 enables unlimited number of retries while value of 1 means that only one call is tried client true false false Controls whether messages messageLogging should be logged at INFO level client string empty Space separated list of outHandlers string additional handler class names for handling outgoing WS messages client true false true Controls whether security securitySessions sessions should be enabled client serverHos NONE WARN Controls whether server s tnameChecking WARN FAIL hostname should be checked for matching its certificate subject This verification prevents man in the middle attacks If enabled WARN will only print warni
25. an fetch the job status by specifying the descriptor job file This file is automatically generated after a successful execution of bes submit job command Example Fetch job status example UEcANESS AO status soObucdeaiob Job can be terminated using a job descriptor file ucc bes terminate Job jobid job To list BESFactory properties ucc bes list att s bes 1 The above command will result in BESFactory s properties without jobs information To see the list of the user s jobs on a BESFactory ucc bes list job s bes 1 12 3 Get OGSA BES job outputs The UCC OGSA BES extension allow users to fetch job s output through the descriptor job file The command is called bes get output It supports downloading standard error and output files The common usage scenario is as soon as user issues the command UCC will attempt to download the output files Before downloading these files UCC probes the job s status to check whether the job has successfully finished If it is then the output files will be fetched instantly otherwise UCC waits until the job approaches completion Get job output example ucc bes get output jobid job Note that at the moment the bes get output command only supports UNICORE based BES implementations However it can be used against other BES services which implement UNI CORE specific server extensions such as BES Activity and UNICORE s native StorageMan agementService UNICORE C
26. data C read s lt STORAGE gt test f out json The f option is optional 9 2 3 updating metadata Using update the given metadata is merged with any existing metadata Say we have a file x json containing X3 Y we can append this to the existing metadata ucc metadata C update s lt STORAGE gt test f x json Check that the metadata has indeed been appended 9 2 4 deleting metadata Explicitely deleting is also possible ucc metadata C delete s lt STORAGE gt test Check that the metadata has indeed been deleted 9 2 5 searching Searching requires a search string according to the rules of Apache Lucene and is triggered by the search command ucc metadata C search q foo s lt STORAGE gt 9 2 6 triggering metadata extraction To trigger the extraction of metadata on the server use the start extract command uce metadata C start extract s lt STORAGE gt In this case the denotes the base path from which to start the extraction process The extrac tion process is asynchronous so a Task service address will be returned which can be used to monitor the extraction process using the wsrf getproperties command UNICORE Commandline Client User Manual 42 10 Workflow extensions 10 1 Introduction UCC supports the UNICORE workflow system and allows to submit workflows to the workflow engine or single jobs to the service orchestrator broker The workflows are
27. e batch job on the target system However there is a UNICORE feature called parameter sweep which leads to the creation of multiple batch jobs from a single template job UCC can also create these sweep jobs as described in the relevant parts of the job description Note that a sweep job still is treated as a single job by UNICORE Sweep jobs are very useful if you need to run jobs that are highly similar and only differ by a parameter setting or even by a different input file 7 1 Site name You can optionally specify on which site if available the job should be run Site DEMO SITE If you do not specifiy anything UCC will select a site that will match your requirements at least those that UCC checks for UNICORE Commandline Client User Manual 26 7 2 Specifying the application or executable You can specify a UNICORE application by name and version or using a machine dependent path to an executable file using application name and version ApplicationName Date APP ARO Sas ton Note the comma separation and the curly braces To call an executable using an executable Executable bin date 7 3 Arguments and Environment settings Arguments and environment settings are specified using a list of String values Here is an example BxceutalolecmeU ouem aksiy Micciemess MUL EW Environment OO O A 7 3 1 Argument sweeps To create a sweep over an Argument
28. e eu the UNICORE website which contains descriptions of many common errors 17 2 6 Are the JSDL documents and workflow documents validated The JSDL documents passed to the run command and to submit workflow are validated and any errors are logged If you wish UCC to stop in case of validation errors you need to set a property ucc validation fail_on_errors true
29. e in UNICORE 9 1 Basics The metadata functions are all accessed via a single UCC command metadata The actual operation to be performed is given with the C i e command option The storage to be operated upon is given using the s option alternatively the m option can be used to directly give the metadata service URL In addition to the URL the name of the target file on the storage is required Metadata is represented in JSON format The metadata operations usually read metadata from a file or write results to file which is specified using the f option In the following examples lt STORAGE gt denotes the URL of a storage capable of handling metadata 9 2 Available commands 9 2 1 creating metadata To create metadata a file in JSON format is required containing key value pairs For example edit the file meta json to contain TOG A Say we have a file test on our storage then you can create metadata as follows ucc metadata C creat f meta json s lt STORAGE gt test If you now look at the file with Is l m ucc ls 1 m lt STORAGE gt test you should get something like this lt fw 3344 2011 06 27 22 32 test MIEDOS a resourceName test UNICORE Commandline Client User Manual 41 9 2 2 reading metadata Apart from the Is 1 m used above there is also an explicit read command which can write the metadata to a file as well ucc meta
30. e name absolute path of your configuration file For example to set your keystore truststore and registry the file would contain the following settings UNICORE Commandline Client User Manual 7 credential path lt your keystore gt credential password XXXXXXX truststore type keystor truststore keystorePath lt your keystore gt truststore keystorePassword XXXXXX registry https localhost 8080 DEMO SITE services Registry res default_registry Note To protect your passwords you should make the file non readable by others for example on Unix using a command such as chmod 600 preferences Note If the credential and or truststore passwords are not given in the properties file they will be queried interactively 4 3 Credential and truststore options In general you need a keystore containing your identity in order to use UNICORE as well as a truststore file or directory containing trusted certificates Note that there may be other options available for authentication try ucc help auth to find out A full list of options related to credential and truststore management is available in the following table You can also get them via the online help using ucc help auth Table 3 Credential properties Property name Type Default Description value mandatory credential path filesystem path mandatory Credential location In case to be set of jks pkcs12 and pem store it is
31. eatured client for the UNICORE Grid middleware UCC has client commands for all the UNICORE basic services the OGSA BES interface and the UNICORE workflow system It offers the following functions Job submission and management for both UNICORE native and OGSA BES interfaces Batch mode job submission and processing with many performance tuning options Data movement upload download server to server copy etc using the UNICORE storage management functions and available data transfer protocols Storage functions Is mkdir including creation of storage instances via storage factories Full UNICORE workflow system support including the possibility to run single jobs through the resource brokering system Support for the UNICORE metadata system Support for sharing UNICORE resources via ACLs Information about the Grid is provided via the system info and query cip commands Various utilities like a shell mode the ability to generate SAML trust delegations low level WSRE operations and others Extensibility through custom commands and the possibility to run scripts written in the Groovy programming language Built in help For more information about UNICORE visit http www unicore eu 2 Installation and configuration 2 1 Prerequisites To run UCC you need a Java runtime version 7 or later OpenJDK Oracle or IBM You might need to install the unlimited cryptography extensions for the Oracle or IBM Java 2 2 Dow
32. ecifying the site In the example above a random site will be chosen to execute the job To control it you can use the s option This will accept the name of a target system The target systems available to you can be listed by ucc list sites 6 1 3 Accessing a job s working directory Using the UCC s data management functions the job working directory can be accessed at any time after job submission Please see section Section 8 for details 6 2 Options overview The following options are available when running jobs see also the general options overview in Section 4 Table 8 Job submission options for UCC Option Short and long Description form a asynchronous Run asynchronously b brief Do not create a sub directory for output files B broker Select the type of resource broker to use LOCAL or SERVORCH d dryRun Only show candidate sites but do not submit the job s sitenam Site where the job shall be run lt SITE gt S schedul Schedule the submission of the job at the given time lt Time gt j jsdl Tell UCC that the job file is a JSDL document o output Directory for any output produced default is the current lt Output_dir gt directory O stdout specify a name for the exported standard out by default lt stdout_name gt stdout E stderr specify a name for the exported standard error by default lt stderr_name gt stderr UNICO
33. er data transfer Use the a option to run asynchronously i e ucc will not wait for the transfer to complete Instead a file containing the transfer reference will be written which can be passed to the copy file status command for status checking later In case the source and target file are on the same storage resource UCC will issue the remote copy command and return immediately as there is no need for an asynchronous mode Example ucc copy file s u6 OTHER SITE Home test txt t u6 DEMO SITE Home test txt Sometimes a user wishes to schedule the time when a server to server transfer is executed for example because she knows that more network bandwith will be available at that time Note This feature only works with server release 6 4 0 or higher To schedule the file transfer you can use the S option to the ucc copy file command UNICORE Commandline Client User Manual 39 tes corales S VILA so The format is simply HH mm hours and minutes Alternatively you can give the time in the full ISO 8601 format including year date time and time zone UCA PY CSS sued O AZ ADS OO 05 02 010 8 6 Using the StorageFactory service UNICORE sites may allow users to dynamically create storage resources which even can be linked to special back end systems like Apache HDFS iRODS or cloud storage like Amazon S3 You can find out if there are sites supporting this StorageFactory se
34. ers the cp command supports several additional options The S option allows to schedule a transfer for a certain time For example ucc cp S 23 00 u6 SHARE x pdf u6 Demo SITE Home The format is simply HH mm hours and minutes Alternatively you can give the time in the full ISO 8601 format including year date time and time zone es es 2QII 12 244111 2330 8 OORHOZOOU Another useful option is a which will execute the server server transfer asynchronously i e the client will not wait for the transfer to finish UNICORE Commandline Client User Manual 36 8 2 2 copy file status This will print the status of the given data transfer As argument it expects a file name contain ing the transfer reference or directly the reference Example for Unix which captures the reference into a shell variable export ID ucc cp a u6 OTHER SITE Home test txt u6 DEMO SITE Home test txt ucc copy file status SID 8 2 3 Specifying the file transfer protocol To use a different protocol from the default BFT you can use the P option to specify a list of preferred protocols UCC will try to match them with the capabilities of the storage and use the first match Your preferred protocols can also be listed in your preferences file using the protocols key protocols UFTP BFT Note If necessary you can specify additional filetransfer options in your preferences file as well F
35. erties lt Address gt To extend the lifetime of a resource ucc wsrf extend lt Address gt lt Days gt These commands can be abbreviated e g ucc wsrf d lt Address gt 16 Scripting UCC can execute Groovy scripts Groovy http groovy codehaus org is a dynamic scripting language similar to Python or Ruby but very closely integrated with Java The scripting facility can be used for automation tasks or implementation of custom commands but it needs a bit of insight into how UNICORE and UCC work 16 1 Script context Your Groovy scripts can access some predefined variables that are summarized in the following table Table 11 Variables accessible for scripts variable description Java type registry A preconfigured client for de fzj unicore uas client IRegistryQuery accessing the registry configurationProvider Security configuration de fzj unicore ucc authn UCCConfigurationProvider provider keystore etc registryURL the URL of the registry java lang String message Writer for writing messages to the de fzj unicore ucc Message Writer user commandLine the command line org apache commons cli CommandLine properties defaults from the user s java util Properties properties file UNICORE Commandline Client User Manual 54 16 2 Examples Some example Groovy scripts can be found in the samples directory of the UCC distribution Here is a script that will delete all your finished jobs
36. es that contain the word login and the admin value of all attributes containing the word role in their name ucc list attributes J auto E x xlogin x role admin You may also exclude attributes For example the following filter chooses all attributes but those ending with an a tes lisis tices J awto O Yo saw If you are content with the results of filtering you can obtain an assertion only with the filtered attributes ucc save attributes J auto G vo group 0 assertion xml I Loa sole acia 0 Y at You can also use it directly ucc connect J auto G vo group 1 xlogin role admin Q A xa UNICORE Commandline Client User Manual 20 5 4 Rules for multiple filters Each attribute is considered as a set of name value pairs containing the name of attribute and the value of attribute A pair matches a name filter i e a filter defined using a regular expression without equals sign if the name of the attribute matches the regular expression A pair matches a name value filter i e a filter defined using a regular expression with equals sign if the name of the attribute matches the left hand of the regular expression and the value matches the right hand For each pair the following steps are performed e if there is a matching exclusive name value filter the pair is rejected otherwise if there is a matching inclusive name value filter the pair passes otherwise if there
37. et registry https output tmp To set one or more properties add space separated key value strings ucc gt set output work registry https You can also clear a property set it to null by using unset ucc gt unset registry 14 Sharing resources Accessing UNICORE resources jobs storages is usually only possible when you own the resource or when there are special server side policies in place that allow you access Starting with server version 7 3 UNICORE supports ACLs on a per service instance basis This means that you can give other users access to your target systems jobs storages For example you might have access to an S3 cloud storage via UNICORE and you want to securely share data on this resource Or you want to allow others to check job status or even allow them to abort jobs Note that to access actual files the permissions on file system level still need to match Usually this is achieved by using Unix groups 14 1 Editing ACLs The ACLs are managed via the share command Use the basic ucc share lt URL gt to showe the current ACL for the given resource where URL is the full WSRF service URL of the resource e g uce share https localhost 8080 DEMO SITE services StorageManagement res demo Hom To add an ACL entry use UNICORE Commandline Client User Manual 51 uce share AGH ACHh2 32 lt URL gt where ACE is an access control ent
38. he working directory of the given job on the SITE target system And ucc ls 1 u6 SITE 1f3hbc2e2 d814 406e 811d e533f8f7a93b will list the job s working directory It is also possible to refer to storage services that are registered in the registry using their name for example u6 SHARE myfiles a_file UNICORE Commandline Client User Manual 35 can be used to refer to the shared storage named SHARE if it is registered in the registry Though convenient the method using unicore6 is much slower and will generate some network traffic If you do a lot of operations on the same resource you should use the resolve command to find out the URI of the resource and use that later 8 1 1 The resolve command This will figure out the real address for a unicore6 URL as defined above ucc resolve u6 SHARE 8 2 Data movement 8 2 1 cp The cp command is a generic command for copying source file s to a target destination where source and target can be remote locations or files on the local machine Wild card characters and are supported Examples for client server transfers ucc cp data x pdf u6 SHARE pdfs ucc cp u6 SHARE test pdf ucc cp u6 SHARE test1 u6 SHARE test3 data The R option allows to choose whether subdirectories are to be copied too Examples for server server transfers ucc cp u6 SHARE pdf u6 Demo SITE Home For server to server transf
39. ileName Ve Note that only a single stage in can be sweeped over in this way and that this will not work with files imported from your local client machine 7 5 3 Exporting result files from the job workspace To export files from the job s working directory to your local machine or to some remote storage use the Exports keyword Here is an example Exports section that specifies two exports UNICORE Commandline Client User Manual 30 Exports this exports all png files to a local directory From x png To home me images this exports a single file to a to local directory failure of this data transfer will be ignored From error log To home me logs error log FailOnError EOS ip this exports to a UNICORE storage From stdout To u6 DEMO SITE Home results myjob stdout Le As a special case UCC also supports downloading files from other UNICORE storages using the Exports keyword HecpOrmtesi all this exports a file from a UNICORE storage From u6 DEMO SITE Work somefile To home me somefile br The remote location can be given as a full UNICORE URI or using the more user friendly but slower u6 notation Read more on remote locations in Section 8 Local files can be given as an absolute or relative path in the latter case the configured output directory will be used as base directory The protocol to be used for
40. imports and exports can be chosen using the Preferred Protocols entry containing a space separated list of protocols Preferred protocols BFT RBYTEIO If not specified BFT will be used UNICORE Commandline Client User Manual 31 7 5 4 Specifying credentials for data staging Some data staging protocols supported by UNICORE require credentials such as username and password Currently these are ftp and scp In case you want to give username and password the syntax is as follows Imports From ftp someserver 25 some file To input_data Credentials Username myname Password mypassword Why and similarly for exports 7 5 5 Redirecting standard input If you want to have your application or executable read its standard input from a file you can use the following Stdin filename then the standard input will come from the file named filename in the job working directory 7 6 Resources A job definition can have a Resources section specifying the resources to request on the remote system For example Resources memory per node bytes you may use the common K M or G UA Memory 2G wal LEtme seconds se mia at ore a fer other units Runtime 86400 Total number of requested CPUs CPUs 64 you may optionally give the number of nodes Nodes 2 together with the CPUs per node CPUsPerNode 32 UNICORE Commandline Client User Manua
41. invoking UCC Also when using UCC with infrastructures that run a SAML enabled VO server such as UVOS there are additional options for specifying the VO server your VO etc Please see section for details about the VO support Here is a list of options related to user attributes and VOs Table 2 Security and VO options for the UCC Option short and long Description form U user User ID to use remotely if you have multiple Z preferenc Select from your remote attributes e g xlogin J VO VO server URL G VOGroup VO group A attributeAs File containing a VO attribute assertion sertion A SS Filter VO attributes includeAttributes Q Filter VO attributes excludeAttributes 4 2 Configuration file Note Since UCC 6 6 0 the use of a configuration file is mandatory since the possibility to specify options related to keystore and truststore on the command line have been removed By default UCC checks for the existence of a file lt userhome ucc preferences gt and reads set tings from there As shown above you can use a different file by specifying it on the comman dline using the c option The configuration file can contain default settings for many commandline options which are given in the form lt option name gt lt value gt where lt option name gt is the long which are auto matically replaced with the environmental variable values with the sam
42. ion 4 for a full description of available options Note If you are worried about security and do not want specify the password UCC will ask for it if it is not given in the preferences or on the commandline UNICORE Commandline Client User Manual 3 Note Windows only The preferences are usually searched in the c Users lt user_name gt ucc folder To create the ucc folder you might have to use the command prompt mkdir command When specifying paths in the preferences file the backslash character needs to be written using an extra backslash For example if you are using a local UNICORE installation for testing you could use registry https localhost 8080 DEMO SITE services Registry res default_registry Note If you wish to change the default property file location you can set a Java VM property in the UCC start script for example by editing the command that starts UCC java Ducc preferences lt preferences location gt 2 5 Logging UCC writes some messages to the console more if you choose the verbose mode v option If you need real logging e g when using the batch mode you can edit the lt UCC_HOME gt conf logging properties file which configures the Log4J logging infrastructure used in UNICORE 2 6 Installing UCC extensions UCC can be extended with additional commands It is enough to copy the libraries jar files of the extension into a directo
43. is a matching exclusive name filter the pair is rejected otherwise if there is a matching inclusive name filter or there are no inclusive name filters the pair passes otherwise the pair is rejected The result of filtering are the attributes containing only those values which passed after filter ing empty attributes are discarded 6 Running jobs 6 1 Introduction The UCC can run jobs specified in a simple job description format Section 7 In the following it is assumed that you have UCC installed Section 2 and tried some examples Section 3 For example assume the file myjob u looks as follows ApplicationName Date ApplicationVersion 1 0 To run this through UCC issue the following command ucc run myjob u This will submit the job wait for completion download the stdout and stderr files and place them in your default output directory The run command has many options to see all the possibilities use the built in help uee ciim n UNICORE Commandline Client User Manual 21 6 1 1 Controlling the output location and file names Output files will be placed in the directory given by the o option if not given the current directory is used Also file names will be put into a subdirectory named as the job id to prevent accidental overwriting of existing files This behaviour can be changed using the b option When b is given on the command line no subdirectory will be created 6 1 2 Sp
44. l 32 Custom resources site dependent StackLimitPerThread 262144 Operating system Operating system LINUX MACOS WINNT Resource reservation reference Reservation job1234 Note that you can also specify a reservation reference if your batch system supports this and you have made a resource reservation 7 7 Execution environments To run a job in a special execution environments as supported by the server you can use the following syntax Execution environment Nan Arguments ArgNamel valuel ArgName2 value2 ep Opr rons in soo ily Preconmands anne RO Sits OMmanic Si aa User precommand RunUserPrecommandOnLoginNode false User ypOsecommandse in a RunUserPostcommandOnLoginNode true The user pre and post commands can be run either separately on the login node default or be placed in the job script This is achieved using the RunUserPrecommandOnLoginNode and RunUserPostcommandOnLoginNode directives which can be set to true or false 7 8 Miscellaneous options 7 8 1 Selecting the remote login and or group In case you have multiple logins or Unix groups on the remote site mapped to the same creden tial you can select the user name and or group to use as follows User name yourlogin Group yourgroup UNICORE Commandline Client User Manual 33 Hint you can get a list of your logins groups on the site by executing ucc list sites s SITENAME 1
45. ng in log FAIL will close the connection client true false true Controls whether SSL sslAuthnEnabled authentication of the client should be performed client true false true Controls whether the sslEnabled SSL TLS connection mode is enabled client integer number 10000 Amount of milliseconds to wsCallRetryDelay wait before retry of a failed web service call HTTP clie nt settings client http allow chunking true false true If set to false then the client will not use HTTP 1 1 data chunking UNICORE Commandline Client User Manual 16 Table 5 continued Property name Type Default Description value mandatory client http true false false If set to true then the client connection close will send connection close header so the server will close the socket client http integer number 20000 Timeout for the connection connection timeout establishing ms client http maxPerRoute integer number 6 How many connections per host can be made Note this is a limit for a single client object instance client http integer number 3 Maximum number of maxRedirects allowed HTTP redirects client http integer number 20 How many connections in maxTotal total can be made Note this is a limit for a single client object instance client http socket timeout integer number 0 Socket timeout ms HTTP pr
46. nload You can get the latest version from the SourceForge UNICORE download page UNICORE Commandline Client User Manual 2 2 3 Installation and configuration To install unpack the distribution in a directory of your choice It s a good idea to add the bin directory to your PATH variable export PATH PATH lt UCC_HOME gt bin where UCC_HOME is the directory you installed UCC in Note Windows only Please do not install UCC into a directory containing spaces such as Program files Also avoid long path names this can lead to errors due to the Windows limit on command line length Setting environment variables can be done as administrator using the Control panel System Extras panel Though you can specify your keystore location and other parameters on the commandline it is easiest to place this information in a file so that you do not have to key in this information repeatedly 2 4 Preferences file UCC checks by default whether the file lt userhome gt ucc preferences exists and reads it A minimal example that specifies keystore password and your preferred UNICORE registry URL would look as follows credential path lt your keystore gt credential password lt your password gt truststore type keystor truststore keystorePath lt your keystore gt truststore keystorePassword lt your password gt client serverHostnameChecking NONE registry lt your registry gt Please refer to Sect
47. oe a be eae Be eA Ee So Deprecated commands lt 23 602 224i 4008 66 004 bade eee a 3 6 Using the StorageFactory Service coco ei a we So Metadata management functions OL WBASICS e s Sark Bow ee OS th Ge a Bite we kw ed 92 Available commands 0 ss sid doy a Re Sago e RR ew me a Workflow extensions 20 20 21 22 22 24 25 25 26 26 27 28 31 32 32 34 34 35 36 37 37 39 UNICORE Commandline Client User Manual 11 12 13 14 15 16 17 Batch processing 11 1 Options 11 2 Performance tuning Options s se ee rs a a 11 3 Resource selection in batch mode see cercos OGSA BES functions 12 1 OGSA BES Setup 12 2 Running and monitoring OGSA BES jobs 123 Get OGSA BES JOD QUIPUES e ecne ew AS eR ee ee Soe eH oy Y 12 4 Enabling username password authentication The UCC shell 13 1 Exiting the shell 13 2 Changing property settings sucios eRe ok ee ee ee ee EE a Sharing resources 14 1 Editing ACLs 14 2 Permission levels Admin use of UCC 15 1 Security considerations 15 2 Filtering lists 15 3 WSRF commands Scripting 16 1 Script context 16 2 Examples Frequently asked questions 17 1 Configuration FeZ Wsage nicol 50 50 51 51 51 52 53 53 53 54 UNICORE Commandline Client User Manual 1 1 Overview The UNICORE Commandline client UCC is a full f
48. om some remote location An important concept in UNICORE is the job s workspace also called Uspace which is the default location into which files are placed The same applies to result files by default files will be downloaded from the job s workspace However other remote storage locations are supported too 7 5 1 Importing files into the job workspace To import files from your local computer or from remote sites to the job s working directory on the remote UNICORE server there s the Imports keyword Here is an example Import section that specifies three imports INMSKONaIESIS import a local file into the job workspace From work data fileName To uspaceFileName import a set of PDF files into the Uspace mirom Arosa ale saeit Wes Wu jh import a remote file from a UNICORE storage From u6 DEMO SITE Home testfile To otherUspaceFile If for some reason it may happen that the local file does not exist and you want the job to run anyway there is a flag FailOnError that can be set to false LMP Owe Sia do not fail on errors for this import From work data fileName To uspaceFileName gt realice Weadlse Iy Recent UNICORE servers 7 3 and later support optimized input data handling When multiple jobs or workflows use the same input data files it may be possible to just create a symbolic link instead of physically copying the file
49. ommandline Client User Manual 49 12 4 Enabling username password authentication Some BES implementations support authentication using username and password To add user name and password to the messages sent by UCC the UCC preferences file must contain the following settings setup username password uas security out handler classname de fzj unicore bes security Username0utHandler de fzj unicore bes security UsernameOutHandler wsUserName lt your_username gt de fzj unicore bes security UsernameOutHandler wsPassword lt gt your_password gt de fzj unicore bes security UsernameOutHandler wsActivateUsernameProfile tru 13 The UCC shell If you want to run a larger number of UCC commands the overhead of starting the Java VM or checking the registry may become annoying For this scenario UCC offers a shell that allows the user to enter UCC commands interactively It is usually started by ucc shell If you want to process a list of commands from a file instead of typing them you can start the shell like this ucc shell f commandsfile or on Unix you can use the redirection features ucc shell lt commandsfile 13 1 Exiting the shell To exit type exit or press CTRL D UNICORE Commandline Client User Manual 50 13 2 Changing property settings To change a property setting in shell mode you can use the set command Without additional arguments current properties are listed USE s
50. or example to use the UFTP protocol you may need to specify the client host address and the number of parallel streams explicitely uftp client host your_client_ip_ address uftp streams 2 encrypt data at the cost of performance uftp encryption true compress data requires UFTP2 server uftp compression true You can even override the UFTP server host which can be useful in case the UFTP server is accessible via multiple network interfaces uftp server host myhost com UCC will try to use reasonable defaults for any missing parameters 8 3 Handling directories 8 3 1 mkdir This will create a directory including required parent directories remotely Example UNICORE Commandline Client User Manual 37 ucc mkdir u6 DEMO SITE Home testdirectory data pdfs 8 3 2 rm This will remove a file or directory remotely By default UCC will ask for a confirmation Use the quiet or q option to disable this confirmation e g when using this command in scripts Example ucc rm u6 DEMO SITE Home testdirectory data pdfs 8 4 Finding data 8 4 1 Is This will list a remote directory Useful options are 1 detailed output H human friendly and R recurse Example ucc 1s u6 DEMO SITE Home 1 H If the storage supports metadata you can get the metadata of a single file using Is 1 m ucc ls u6 DEMO SITE Home bashrc l m 8 4 2 find Thi
51. ory to be used via the f option to the workflow submit command ucc workflow submit f lt factory url gt lt workflow file gt You can check the available factories with the system info command If not specified UCC will use the fist storage factory it finds in the registry In general storages that are dynamically created will be deleted when the workflow is deleted To persistently store data you need to make sure to export important result files to a persistent location e g your Home on some Grid site or a persistent storage Alternatively you can directly specify a storage URL either using the convenient u6 notation or as a real network URL ucc workflow submit S u6 MY SITE Home lt workflow file gt ucc workflow submit S https my gateway SITE services StorageManagement res myuser Home lt workflow file gt 10 4 1 Importing local data for use by a workflow If you have local files that need to be imported before starting the workflow you have to specify this using a normal UCC job file that contains only an Imports section stage in specification Imports From local file sh To c9m WORKFLOW_ID input sh l When submitting the workflow add the u lt filename gt option to specify the imports file This will cause UCC to copy the local file local file sh to the workflow storage space You can refer to this file in your workflow using the global name c9m say
52. oxy settings client http string Space single separated list nonProxyHosts of hosts for which the HTTP proxy should not be used client http string Relevant only when using proxy password HTTP proxy defines password for authentication to the proxy client http string Relevant only when using proxy user HTTP proxy defines username for authentication to the proxy client http string If set then the HTTP proxy proxyHost will be used with this hostname client http proxyPort integer number HTTP proxy port If not defined then system property is consulted and as a final fallback 80 is used UNICORE Commandline Client User Manual 17 Table 5 continued Property name Type Default Description value mandatory client http string HTTP HTTP proxy type HTTP or proxyType SOCKS Table 6 HTTP options for the UCC Property name Description client http HTTP s proxy to use proxyHost client http Port of the HTTP s proxy to use proxyPort client http Space separated list of host name fragments for which NOT nonProxyHosts to go via the proxy If the target URL contains such a fragment it is accessed directly client http Timeout to use when establishing a HTTP connection connection timeout client http Timeout to use when reading writing from to HTTP socket timeout connection For example to
53. re of keystore type Openssl type settings UNICORE Commandline Client User Manual 10 Table 4 continued Property name Type Default Description value mandatory truststore opens true false false In case of openssl slNewStoreFormat truststore specifies whether the trust store is in openssl 1 0 0 format true or older openssl 0 x format false truststore GLOBUS_EUGRHDRMADPM In case of openssl openss1NsMode EU A_GLOBUS truststore controls which GRIDPMA_GLOBUS and in which order GLOBUS namespace checking rules EUGRIDPMA should be applied The GLOBUS_EUGRIDPMA_REQUR QUIRE settings will EU cause that all configured GRIDPMA_GLOBUS_REQUIRMiamespace definitions files GLOBUS_REQUIRE must be present for each EU trusted CA certificate GRIDPMA_REQUIRE otherwise checking will EU fail The AND settings will GRIDPMA_AND_GLOBUS cause to check both existing EU namespace files Otherwise GRIDPMA_AND_GLOBUS_REQIB REL found is checked IGNORE in the order defined by the property truststore filesystem path etc Directory to be used for opensslPath grid sec opeenssl truststore urity certific ates Revocation settings truststore crlCo nnectionTimeout integer number 15 Connection timeout for fetching the remote CRLs in seconds not used for Openssl truststores UNICORE Commandline Client User Manual 11 Table 4 continued
54. rname and password In this case specify only unity address and the token like this authenticationMethod unity unity address https lt host gt lt port gt unicore soapidp saml2unicoreidp soap oidc AuthenticationService oauth2 bearerToken lt your oauth token gt 4 6 Using MyProxy UCC can retrieve a short lived certificate from a MyProxy server To setup UCC for this use the following properties UNICORE Commandline Client User Manual 14 authenticationMethod MYPROXY myproxy host myproxy teragrid org myproxy port 7512 myproxy username lt your MyProxy username gt myproxy password lt your MyProxy password gt myproxy expires 12 At minimum you must specify the authenticationMethod and myproxy host parameters The default port of 7512 will be used if not otherwise specified Password and username are optional if you do not specify them they will be queried interactively The lifetime in hours of the short lived certificate can be set with the myproxy expires option if not set the lifetime is 12 hours 4 7 Client options The configuration file may also contain low level options for example if you need to specify connection timeouts http proxies etc Table 5 Client options Property name Type Default Description value mandatory client digitalSi true false true Controls whether signing of gningEnabled key web service requests should be performed client true f
55. rvice either by running the system info l command or better using gt ucc create storag al This will list the available StorageFactory services and also show which types of storage are supported and how much space is left on each of them UCC supports creating storages via the create storage command The simple gt ucc create storage will create a new storage resource using the default storage type at some site Usually you want to control at least where the storage is created Additionally the type of storage and some parameters can be passed to UCC As an example creating a storage of type S3 would look like this gt ucc create storag t S3 accessKey secretKey Note With UNICORE servers before 7 2 you ll need to know what parameters can be used with a certain storage types UNICORE v7 2 and later publishes this information You can also read parameters from a file Say you have your S3 keys in a file s3 properties then you can use the following syntax gt ucc create storag t S3 s3 properties You can also mix this with the normal key value syntax or mix it like this gt ucc create storag t S3 accessKey s3 accessKey secretKey s3 secretKey The last version key file causes just the value to be read from the named file UNICORE Commandline Client User Manual 40 9 Metadata management functions UCC offers a simple interface to access the metadata management servic
56. ry expressed in a simple format read modify DN VO GROUP UID value For example to give modify permission to a user whose UNIX user id on the target system is test you would use ucc share modify UID test lt URL gt To delete entries use the d option ucc share d modify UID test lt URL gt To delete all entries use the b option uce share D lt URL gt 14 2 Permission levels The permissions controlled by ACLs are as follows e read access resource properties e modify perform actions e g job submission or creating a file export Only the owner of a resource can edit the ACL or destroy the resource 15 Admin use of UCC You can use UCC to keep track of your jobs or with appropriate permissions to keep track of all the resources on a site UCC allows to list jobs Grid sites and applications including full details Using the scripting possibilities described in Section 16 UCC can be extended to other administrative tasks as well 15 1 Security considerations Usually each UNICORE user has only access to his or her own resources such as jobs For administrative use you will need to aquire administrator privileges There are two ways to achieve this UNICORE Commandline Client User Manual 52 create a dedicated certificate and map it to role admin in the XUUDB or whatever attribute source you are using This method is recommended if you want to remotely administrate UNICORE X
57. ry that is scanned by UCC in general these are the UCC lib 2 7 Testing the installation To test your UCC installation and to get information about the resources on the Grid you re connecting to do UCCeSyStem ioro iy 3 Getting started with UCC Assuming you have successfully installed UCC this section shows how to get going quickly UNICORE Commandline Client User Manual 4 3 1 Getting help Calling UCC with the h option will show the available options To get a list of available commands type mes n To get help on a specific command type uces lt command gt IZA See also here for a list of common options 3 2 Connecting First contact UNICORE and make sure you have access to some target systems ucc connect options 3 3 List available sites Then list the sites available to you using ucc list sites options 3 4 Running your first job The UCC distribution contains samples that you can run Let s run the date sample The v switch prints more info so you can see what s going on ucc run options yv UCC_HOME samples date u Note Look for UCC samples in the usr share doc unicore ucc samples directory This will run date on a randomly chosen site and retrieve the output To run on a particular site use the s option to specify a particular target system UNICORE Commandline Client User Manual 5 3 5 Listing your jobs The command ucc list jobs options
58. s command is a similar to the well known Unix utility however much less powerful It allows to do recursive listings and retrieve files matching certain conditions Currently only name match is available For example to get all PDF files on a storage ucc find r l u6 DEMO SITE Home N pdf The find command is currently implemented synchronously and may thus run into a network timeout when it takes too long This limitation will be overcome in future versions of this command 8 5 Deprecated commands While the old get file put file and copy file are still available you can use the simpler cp in stead UNICORE Commandline Client User Manual 38 8 5 1 get file Use get file to download remote files to your local machine Example ucc get file s u6 DEMO SITE Home test txt t my_test txt The s source and t target options are used to denote the source file s and the target file or directory Wild card characters and are supported For example ucc get file s u6 DEMO SITE Home x x pdf t pdfs will download all pdf files and write them to the pdfs directory which must exist 8 5 2 put file Use put file to upload a local file to a remote location Example ucc put file s test txt t u6 DEMO SITE Home test txt If you specify the a option data will be appended to an existing file 8 5 3 copy file This will initiate a server to serv
59. sing a single invocation as follows uce bach i obs o Out n As job files UCC will accept files ending in u jsdl or xml 11 1 Options You can run in follow mode where UCC will watch the input directory and will process new files as they arrive UNICORE Commandline Client User Manual 45 UCRANIA Ca i SOON mon UCC can also process JSDL files to batch process these use the j option UCA AECA OSO RO 11 2 Performance tuning options Getting the most performance out of UCC and your Grid installation can be a challenging task Sending too many jobs to a site might decrease throughput sometimes the client machine can be the limiting factor etc You should experiment a bit to get the best performance for your specific setup UCC has many options available for tuning Here is an overview Table 9 Tuning options for the UCC batch mode Option short and long Description form K keep Do not delete finished jobs on the server By default finished jobs are destroyed m Max Limit on jobs submitted by UCC at one time default 100 lt MaxRunningJobs gt t threads lt NumThreads gt u update lt Updatelnterval gt Number of threads to be used for processing default 4 Minimum time in milliseconds between status requests on a single job Default 1000 R Do not check if the necessary application is available on the noResourceCheck target sys
60. tch out to properly escape any arguments in order not to interfere with the arguments to UCC UNICORE Commandline Client User Manual 25 7 Job description format UCC uses a simple format that allows you to specify the application or executable you want to run arguments and environment settings any files to stage in from remote servers or the local machine and any result files to stage out A number of sample files can be found in the samples directory of your UCC distribution on Linux check also usr share unicore ucc samples The format used is called JSON and contains comma separated key value mappings where the values can be simple strings or lists of values or maps String values should be placed in quotes Comments are inofficially possible using the hash character as in Unix shell scrips Each JSON file must begin and end with curly braces Several complete job samples can be found in the samples directory of the distribution Note Note quotes are needed around the keys and values in case special characters like or appear if in doubt use quotes we To view an example job showing most of the available options simply run es sabia F most of the options shown are not mandatory of course Note You may alternatively specify jobs in the JSDL format that is used internally in UNICORE To do this run UCC with the j option Usually a UCC job file describe a singl
61. tem will increase performance a bit Xp Do not fetch standard output and error noFetch0ut come S submitOnly Only submit the jobs do not wait for them to finish M maxNewJobs Limit the number of job submissions default 100 S sitenam Specify which site to use W siteWeights Specify a file containing site weights de Sa Assume jobs are in JSDL format instead of the default JSON u files UNICORE Commandline Client User Manual 46 11 3 Resource selection in batch mode By default the UCC batch mode will select a random site for running a job You can modify the selection in different ways e using the s option or a Site lt sitename gt entry in the job file you can specify the site directly e use the W option to specify a file containing site weights Say you have two sites where one site is a big cluster and the other a small cluster To send more jobs to the big cluster you can use the site weights file example site weights file for use with ucc batch W BIG CLUSTER 100 SMALL CLUSTER 10 send no jobs to this site BAD CLUSTER 0 set default weight for any sites not specified here UCC_DEFAULT_SITE_WEIGHT 10 This would tell UCC to send 10 times more jobs to the BIG CLUSTER site and send no jobs to the BAD CLUSTER All other sites would get weight 10 i e the same as SMALL CLUSTER 12 OG
62. try ucc connect J auto G vo group However note that using lt lt A gt gt gt option is faster attributes are read from a file while in the combined scenario the attributes are fetched from a remote VO server prior to invocation of the intended UCC command UNICORE Commandline Client User Manual 19 5 3 Attribute filtering You might want to obtain an assertion with only handful of attributes and their values This is done by using attribute filters UCC can display a list of all attributes Names scopes values and descriptions if available will be displayed tee isr Atte meSS 0 ero The last column F contains a letter Y next to all attribute values that pass through attribute filters There are two kinds of filters inclusive filters which specify what attributes should pass and exclusive filters which specify what attributes should be filtered out If not specified the default inclusive filter approves of all attributes and their values and the default exclusive filter does not reject anything In this example an inclusive filter choose only those attributes whose name contains the word xlogin UCCAES ES ARAU ESSE UE SAO AS The value for the lt lt lt I gt gt gt option is a list of semicolon separated Java regular expressions The expressions that contain the equal sign are called name value filters those which do not are called name filters For example you may choose all attributes with nam
63. use the server keystore of the UNICORE X server you want to administrate as UCC key store This will also give you administrator privileges For this you will need to be logged on to the UNICORE X server 15 2 Filtering lists The UCC commands that list server side things list jobs etc accept a filtering option that can be used to limit the results of the operation Filtering works on the XML resource properties of the resource in question Filtering is enabled by the f or filter option of the form f XMLNAME OPERATOR VALUE where XMLNAME is the name of an XML Element from the WSRF resource properties docu ment For example to list all your running jobs ucc list jobs f Status equals RUNNING To list all jobs submitted on Nov 13 2007 ucc list jobs f SubmissionTime contains 2007 11 13 etc Table 10 Filtering options Operator long and Description short form equals eq String equality ignoring case notequals neq String inequality ignoring case contains c Substring match notcontains nc substring non match greaterthan gt Lexical comparison lessthan lt Lexical comparison UNICORE Commandline Client User Manual 53 15 3 WSRF commands UCC supports several low level WSRF operations using the wsrf command To destroy a resource ucc wsrf destroy lt Address gt To get a property listing i e print the XML resource property document ucc wsrf getprop

UNICORE Commandline Client: User Manual

Contents

Download Pdf Manuals

Related Search

Related Contents