GROWL - Old STFC e-Science Centre and CSED Web Server
Contents
1. installation of globus and MyProxy grid proxy init grid proxy info globus job submit globus job run gsissh gsiscp openssl myproxy init myproxy info myproxy logon GROWL wrapper scripts growl submit growl status growl get output growl sh growl cp growl mkdir growl rm growl pwd growl which growl get jobmanager NW GRID Seminar 20th March 2007 E s Rob Allan CCLRC Daresbury Laboratory as p ED r F NM de LH Firstly To add these various commands to your environment add the following toyour bashrc cshrc login profile or equivalent For bin sh or bin bash n Growl setup sh For bin csh or bin tcsh source Growl setup csh NW GRID Seminar a gt 20th March 2007 G cIiemce Rob Allan CCLRC Daresbury Laboratory You can inter mix Globus and GR WL but Before running any Globus commands or the GROWL wrapper scripts you must have a valid local proxy certificate grid proxy init Your identity C UK O eScience OU CLRC L DL CN john kewley Enter GRID pass phrase for this identity lt grid password gt Creating Proxy KEKE SE KK SS E kel O eee niine eee Done Your proxy is valid until Tue Jan 23 05 19 34 2007 NW GRID Seminar a Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory growl info growl info is a wrapper for grid cert info grid proxy info2. supported by the SAGA API e Asynchronous notification should be supported by the SAGA API e Support for messages on top of the streaming API should be considered by the SAGA API e Asynchronous notification should be supported by the SAGA API e Application level event generation and delivery should be supported by the SAGA API e Application steering should be supported by the SAGA API but more use cases would be useful e GridRPC should be supported by the SAGA API e Access to data bases does not currently require explicit support in the SAGA API NW GRID Seminar a Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory GROWL_jobRun GROWL_jobSubmit GROWL jobPoll GROWL_jobGetStatus GROWL jobCancel GROWL jobClean GROWL_jobGetStdoutAsString GROWL_jobGetStdoutAsFile GROWL_jobGetStderrAsFile GROWL_jobGetStderrAsString GROWL_jobPutFileToStdin GROWL_jobPutString ToStdin GROWL_getRemoteFile GROWL getRemoteFileAsstring GROWL putRemoteFile NW GRID Seminar a i Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory usual include files GROWL header file with function and server definitions include growl h int main int argc char argv char sessionld int res do authentication run a job get output free sessionld return O Seminar R 20t March 2007 G GE 5 GE AMM GE a Rob Allan CCLRC Daresbury Laboratory Authentic
3. GROWL Scripts and Web Services Rob Allan Grid Technology Group E Science Centre CCLRC Daresbury Laboratory r j allan dl ac uk m NW GRID Seminar N Rob Allan 20t March 2007 Stet lt CCLRC Daresbury Laboratory EEE UNIVERSITY OF JIS LANCASTER J CAMBRIDGE i UNIVERSITY JN Collaborative project JISC VRE I programme between CCLRC Daresbury Laboratory and the Universities of Cambridge and Lancaster Project Objectives to produce a lightweight client side Grid connection toolkit Primarily aimed at application developers as a Grid programming library Project competed end January 2007 but related work is continuing NW GRID Seminar a i Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory Outline 1 Background 2 Avoiding firewall problems 3 Use of GROWL Scripts for authentication job submission and file management 4 GROWL Web Services the GROWL Grid Programming Library NW GRID Seminar a R Rob Allan 20t March 2007 Stet lt CCLRC Daresbury Laboratory GROWL tries to address the three barriers that newcomers find when using the Grid for the first time 1 Setting up the client side middleware 2 Handling of certificates 3 Job submission in the presence of firewalls We covered the first 2 of these in the previous NW GRID training event see http www nw grid ac uk q seminar dares 250107 NW GRID Seminar a Rob Allan 20t March 2007 ial CCLRC Dar
4. and myproxy info growl info Certificate Information including validity subject C UK O eScience OU CLRC L DL CN john kewley notBefore Jun 15 16 10 35 2006 GMT notAfter Jun 15 16 10 35 2007 GMT Local proxy certificate s subject C UK O eScience OU CLRC L DL CN john kewley issuer C UK O eScience OU CLRC L DL CN john kewley identity C UK O eScience OU CLRC L DL CN john kewley type Proxy draft pre RFC compliant impersonation proxy strength 512 bits path tmp x509up ul3445 timeleft 11 57 19 Rob Allan NW GRID Seminar W Science CCLRC Daresbury Laboratory 20th March 2007 1 Grid certificate handling is easier 2 Grid job submission Help with transparency user shouldn t really need to know Machine s jobmanager Home directory location Location in your path of executable Firewall problems minimised 3 File manipulation on client and remote resources NW GRID Seminar a ja Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory If you need to upload your certificate to MyProxy and generate a local proxy growl login is provided with the GROWL scripts grid proxy init lt grid password gt myproxy init Your identity C UK O eScience OU CLRC L DL CN john kewley Enter GRID pass phrase for this identity lt grid password gt Enter MyProxy pass phrase lt myproxy pass gt Verifying Enter MyProxy pass phrase lt myproxy pass
5. ation printf Testing old session key n char uid rja char passPhrase my passwd preferably input from keyboard int ttl 2 char oldld f a6d3fe 1b36 4131 8dbe dd29a5c483c5 Il check validity res growl_testDelegation oldld amp session d if stremp oldid sessionld strcpy oldld sessionld printf Got new session key sin session d printf Need to renew session n get new proxy from MyProxy server onto GROWL server res grow getDelegation uid passPhrase ttl oldld amp sessionld else printf Existing session still OK n Seminar 20th March 2007 N Rob Allan MER WB gt CCLRC Daresbury Laboratory unning an i char response W set up parameters char target Iv1 nw grid ac uk char directory panfs lv1 home rja SABRE char environment char executable usr local sge6 0 bin lx24 amd64 gsub char arguments parallel sh submit the job res growl runClient sessionld target directory environment executable arguments amp response printf Service returned sin response free response NW GRID Seminar a N Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory char list get stdout back from GROWL server res growl getOutput sessionld amp list ii res Q printf n s n list else printf WS call failed n W get stderrr back fro
6. dules e GROWL has a modular structure so only the parts needed have to be installed e GROWL uses several different servers for different purposes in future it will use a registry to locate services fitting a functional description e Current modules are Authentication test and renew proxy on the server s 2 Jobs submit interactive and batch jobs upload stdin and download stdout and stderr via server 3 Files manage workspace on GROWL server stdin stdout stderr and other named files in a sandbox 4 Transfer transfer named files between client and or Grid resources Uses GSI SCP or Grid FTP where available 5 SRB manage virtual file store using SRB uses an SRB server 6 Rcommands meta data management some used internally Others being added your i lla here NW GRID Seminar 20t March 2007 Rob Allan Science CCLRC Daresbury Laboratory Comparison with SAGA SAGA Simple API for Grid Applications is an OGF activity to define a Grid programming API e Job submission and management should be supported by the SAGA API e Resource discovery should be supported by the SAGA API e Data management should be supported by the SAGA API e Efficient data access should be supported by the SAGA API e Data replication should be supported by the SAGA API e Persistent storage of application specific information should be supported by the SAGA API e Streaming of data should be
7. esbury Laboratory Globus Firewalls NA GHD Client Grid Resource globus job submit 4 R globus job get result jobmanager Results gsiscp gsissh GSI SSHTerm NW GRID Seminar 20th March 2007 Rob Allan CCLRC Daresbury Laboratory Web i HPC Portal i Applications GROWL SERVER i Cl l l l z S O 7 r b G N LI S t j a t GROWL I I A KIEN 7 Applications SH LIBRARY E Uguhs l HPC Applications NW GRID Seminar Rob Allan 20 March 2007 CCLRC Daresbury Laboratory GROWL Firewalls 3 tier Client GROWL Server Grid Resource N j ob manager N NW GRID Seminar 20th March 2007 Rob Allan CCLRC Daresbury Laboratory GROWL Components Simple GROWL Little Full GROWL Eagle e Linux e Linux Only e Window interface via Cygwin e Supporting scripts for e API uses only WS interfaces proxy management Authentication remote filestore manipulatic Job Submission Globus e API has additional non WS Simple indirect file transfer components Depends only on gSOAP SRB client remote applications Condor wrapped as WS SRB Direct bulk file transfer SABRE Other dependencies can be extended NW GRID Seminar a Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory Certificate helper scripts mk cert growl info growl login growl logout VDT client
8. gt growl login Password to protect MyProxy certificate lt myproxy pass gt Enter GRID pass phrase for this id lt grid password gt NW GRID Seminar a N Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory unning a grid job 1 growl submit dll nw grid ac uk hostname https dll nw grid ac uk 64010 792 116475 growl status https dll nw grid ac uk 64010 792 116475 PENDING growl status https dll nw grid ac uk 64010 792 116475 DONE growl get output https dll nw grid ac uk 64010 792 116475 comp023 nw grid ac uk growl submit c dll nw grid ac uk hostname growl status PENDING growl status DONE growl get output comp021 nw grid ac uk NW GRID Seminar a N Rob Allan 20t March 2007 srerce CCLRC Daresbury Laboratory Advantages of using GR WL wrapper scripts growl submit uses growl get jobmanager to obtain default parallel queue rather than defaulting to jobmanager fork uses growl which to get full path of executable ensuring it is in your path growl get output uses gsissh to do remote retrieval avoiding client firewall problem NW GRID Seminar W Rob Allan 20th March 2007 G cIiemmce CCLRC Daresbury Laboratory Remote filestore me tion nipula Equivalents of many of the standard unix command tools are provided for remote filestore manipulation growl 1ls contents of directory growl mkdir s
9. m GROWL server res growl getError sessionld amp list s NW GRID Seminar a R Rob Allan 20t March 2007 Stet lt CCLRC Daresbury Laboratory Contains papers presentations and other information Project Web site is being updated JISC project funding finished end of January 2006 User manual tutorials are being added http www growl org uk NW GRID Seminar a R Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory
10. ub directory creation growl rm file removal growl cp remote file copying including 3 party growl which finds executable in your path growl pwd prints your home directory on the grid resource growl sh gsissh wrapper using default ports They have an additional parameter of the remote resource on which they should run NW GRID Seminar a N Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory Rei note file copying usinc growl cp can be used to stage and retrieve files Ihe syntax is the same as that of scp It can also be used for 3 party file transfers growl cp my input file txt dll nw grid ac uk growl cp dll nw grid ac uk my output txt growl cp lvl nw grid ac uk my file txt dll nw grid ac uk For 3 party transfers to work there has to be a route through all firewalls between the 2 remote resources in one direction or the other NW GRID Seminar a N Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory Analogous interface to those we have seen on previous slides Authentication Job submission Remote file manipulation Interfacing to SRB Others indevelopment Wrappers being developed for access from R Fortran C and C Python and Perl later possibly Users at both Lancaster and Bristol using GROWL Web Services although not yet available for widespread release NW GRID Seminar a Rob Allan 20t March 2007 ial CCLRC Daresbury Laboratory GROWL Mo
Download Pdf Manuals
Related Search