GSRM - GARUDA Access Portal
Contents
1. dpm_putdone exit 1 j for 12 0 1 lt nbreplies i if pdfilestatuses i gt status amp DPM FAILED DPM FAILED printf state d 96s serrno 96d errstring lt s gt n 1 f stat pdfilestatuses 1 5status gt gt 12 pdfilestatuses 1 gt status amp OxFFF pdfilestatuses 1 errstring pdfilestatuses 1 gt errstring else printf state d s n 1 f stat pdfilestatuses 1 5status gt gt 12 GSRM v 1 0 User Manual 16 6 10 Page 17 of 26 SSDG CDAC KP Bangalore CDEC if pdfilestatuses 1 gt surl free pdfilestatuses 1 gt surl if pdfilestatuses 1 gt errstring free pdfilestatuses 1 gt errstring free pdfilestatuses I HHHHHBHBHHHHHHHHHHHHHHHHHHHHHHHHHHE end of put done exit 0 int thirdparty_file_transfer_cb void user_arg globus_ftp_client_handle_t handle globus object t err char tmp if err GLOBUS SUCCESS tmp globus object printable to string err globus libc fprintf stderr MM Error in callback s n tmp else globus libc printf n t t Status file transfer successful A globus_mutex_lock amp lock done GLOBUS_TRUE globus_cond_signal amp cond elobus mutex unlock amp lock return 0 Compilation gcc g o dpm_testput2 dpm_testput2 c I opt Icg include dpm I opt globus 4 0 7 include gcc32dbgpthr L opt Icg lib Idpm ldl L opt globus 4 0 7 lib lglobus f2. corresponding functionality with orange boxes GSRM v 1 0 User Manual 16 6 10 Page 4 of 26 SSDG CDAC KP Bangalore o O Jg CDAC Space Mgmt Persistence Space Manager StoRM Bestman Request User Request Interoperability Data Availability i Gsiftp DB i server SRM server l NN Metadata Performance DB GARUDA SRM Admin Request Global Namespace Figure 1 GSRM Component Features Mapping 1 4 GSRM Test Bed Deployment Figure 2 shows the GSRM deployment scenario across four GARUDA centers CDAC Bangalore CDAC Pune CDAC Chennai and CDAC Hyderabad with approximate storage of 1 2TB Bangalore Hyderabad oJ GSRM server GSRM server un E CPE o oco WJ Bangalore xn05 Hyd srmhyd 192 168 63 95 144 16 194 139 k 1TB 200GB Space manager e Space manager Namespace e Namespace manager manager Srm server e Srm server Data base e Data base SRM clients SRM clients Chennai srmchennai 144 16 194 41 Bangalore gridfs 192 168 60 40 Figure 2 GSRM Deployment Above shown is the current test bed setup with servers being installed Bangalore 192 168 63 95 GSRM v 1 0 User Manual 16 6 10 Page 5 of 26 SSDG CDAC KP Bangalore 1 5 CDAC and Hyderabad 144 16 194 139 and clients installations at Bangalore gridfs 192 168 60 40 and Chennai 144 16 194 41 Users can access GSRM services through gridfs 1
3. or pd GSRM v 1 0 User Manual 16 6 10 Page 8 of 26 SSDG CDAC KP Bangalore GDA ReleaseFiles or rf BringOnline or bol StatusBringOnline or sbol GetRequestTokens or grt GetRequestSummary or grs AbortRequest or ar AbortFiles or af ExtendFileLifeTime or efl iV Space management functions ReserveSpace Or rs GetSpaceMetadata or gsm ReleaseSpace or rsp GetSpaceTokens or gst Type clientSRM lt requestName gt h to get help about lt requestName gt options Global options h help Print this help message V Version Print version information Example 1 Ping to GSRM server xn05 ctsf cdac org in Command Usage clientSRM ping e httpg xnO5 ctsf cdac org 1n 8446 Request status statusCodez SRM SUCCESS 0 explanationz SRM server successfully contacted SRM Response version nfo v2 2 otherInfo size 2 0 keyz backend type 0 value DPM 1 keyz backend version 1 value 1 7 2 4 Example 2 Upload a file in GSRM storage It involves 3 steps step 1 ptp prepare to put request request to GSRM server step 2 Invoke the file transfer step 3 Ensure the file has transfered 1 e pd putdone GSRM v 1 0 User Manual 16 6 10 Page 9 of 26 SSDG CDAC KP Bangalore CDAC step 1 ptp request Command clientSRM ptp e httpg xn05 8446 s srm xn05 dpm ctsf cdac org in home test fastainputfile p Output Polling request status Current statu
4. 3 Note 1 There is a bug requires selection of file to be uploaded within 10 seconds in the file upload option of the DPM web interface 2 A sample screenshot of the DPM https browser is shown below DPM HTTPS BROWSER Windows Internet Explorer e x gt eo A 2 Hera Google LOE File Edit View Favorites Tools Help x Google ns Search t D A Y d Share S x Sidewiki 7 Bookmarks I4 Check aj Translate 1 3 AutoFill E e SignIn x Y Q Web Search n 325 Mail v i My Yahoo Y E3 Answers v 1J Anti Spy v R Bookmarks v Messenger v il Movies Spots v News v g Mobile gt X Favorites 523 M Gmail Inbox 7 payalsa New Tab 3 admind usage manual G di DPM v Suggested Sites v e Get More Add ons Linux v DPM HTTPS BROWSER e d v Pagev Safetyv Toos DPM HTTPS BROWSER lel CERN2007 Author Andreas Joachim Peters cern ch 2007 IT GD New File Dirname post mkdir rm chmod Mode 755 permissions nlinks owner group size mtime filename parent dir post mm chmod stat Mode 775 drwxrwxr x 2 0 103 0 Feb 02 2010 cdac rm eee stat Mode 775 drwxrwxr x 4 0 106 Mar22 2010 garuda post hmod stat drwxrwxr x 0 0 102 0 Decl52009 ssdg ig med Mode 775 post hmod tat drwxrwx 81 0 101 0 Mar 192010 test iim chmod stat Mode 770 Interne
5. 6 10 Page 22 of 26 SSDG CDAC KP Bangalore CDAC printSummary putDone java lang String siteUrl releaseFiles java lang String siteUrl removeFiles java lang String surls abortRequest SRMRequestStatus Show the status of the getReturnStatus getRequestToken setPathDetails java util HashMappathDetails getPathDetails getFileStatuses SrmRequestReturnStatus getStatus getExplanation SRMRequestSummary getSummaryArray Table 3 List of Bestman Java classes and APIs Example 1 A sample Java Program using BestMan Java APIs to Show get a file from GSRM server Pseudo Code Initialization SRMServer cc new SRMServer log4jlocation logPath debug cc connect serverUrl cc ping uid SRMRequest req new SRMRequest req setSRMServer cc req setAuthID uid req setRequestType get put req addFiles surl null null req setStorageS ystemInfo storageInfo req setFileStorageType fileType req submit for int j 0 j lt 2 j boolean status status req checkStatus System out printIn req checkstatus status SRMRequestStatus response req getStatus if response null System out printIn nStatus code response getReturnStatus getStatusCode System out printIn nStatus exp response getReturnStatus getExplanation if response getReturnStatus getStatusCode TStatusCode SRM SUCCESS I response getReturnStatus getS
6. 7 2009 Lawrence Berkeley National Laboratory All rights reserved Support at SRM LBL GOV and documents at http datagrid Ibl gov bestman SRM CLIENT Connecting to service httpg xn05 ctsf cdac org in 8446 dpm ctsf cdac org in home garuda SRM PING Wed Feb 10 14 44 23 IST 2010 Calling SrmPing Request versionInfo v2 2 Extra information Key Value backend_type DPM backend_version 1 7 2 4 Example 2 Upload a file into GSRM server GSRM v 1 0 User Manual 16 6 10 Page 19 of 26 SSDG CDAC KP Bangalore CDAC Uploading a file can be done using srm copy command Usage srm copy srcurl targeturl serviceurl srm gsrmserver Output srm copy file tmp filel srm xn05 dpm ctsf cdac org in home test inputfile1 serviceurl srm xn05 8446 dpm ctsf cdac org in home test srm copy 2 2 1 3 6 Mon Nov 30 09 10 48 PST 2009 SRM Clients and BeStMan Copyright c 2007 2009 Lawrence Berkeley National Laboratory All rights reserved Support at SRM LBL GOV and documents at http datagrid lbl gov bestman SRM CLIENT Thu May 06 13 56 58 IST 2010 Connecting to httpg xn05 8446 dpm SRM CLIENT Thu May 06 13 56 59 IST 2010 Calling SrmPrepareToPutRequest now request token 200850ef 2def 40ab bebe da3d0107e11d Request status SRM_REQUEST_QUEUED explanation null SRM CLIENT Next status call in 10 seconds SRM CLIENT Thu May 06 13 57 10 IST 2010 Calling Status at Thu May 06 13 57 10 IST 2010 SRM CLIENT Result Status fr
7. 92 168 60 40 We also provide users a test node pvfs2 172 20 1 81 at Bangalore where all SRM clients has been installed and users can practice using SRM client interfaces The three daemons running at each of the GSRM servers address the requests coming from any of the SRM client installations Space Manager is the main daemon performing space allocation space release and cleanup It also keeps track of each and every user request received from SRM server Namespace Manager takes care of the security of the system by controlling the hierarchical namespace the file permissions and the mapping between SEN Site File Name and physical names SRM server convert the SRM requests into GARUDA SRM requests and distribute the SRM requests workload to the required component Disk Servers provides secure file transfers between the disks and the client Request database stores the metadata about each user data request and Namespace database stores the mapping between physical file name and logical file name Disk Pools are the collection of various disks or file systems part of GSRM system These are the virtual entities where the actual data gets stored GSRM Accessibility in GARUDA GARUDA services are available to users at GARUDA head node currently only at Bangalore eridfs 192 168 60 40 The storage systems supported in this version of GSRM v1 0 are disk based storage Users can either access GSRM storage through command line clients installed at er
8. CDAC GARUDA Storage Resource Manager GSRM Version 1 0 User Manual 16 June 2010 CDAC System Software Development Group C DAC KP Bangalore GSRM v 1 0 User Manual 16 6 10 Page 1 of 26 SSDG CDAC KP Bangalore CoFC GARUDA Storage Resource Manager Version 1 0 User Manual Project No CDAC B SSDG GDS 2010 033 Document No SSDG GDS 2010 User manual 1 0 Control Status Controlled Internal Circulation Only Author Shashidhar V and Payal Saluja Distribution List Project 1 GDS Team 2 Dr B B Prahlada Rao 3 Dr Subrata 4 GARUDA application Users Approved By Designation Date of Approval Name Signature Dr B B Prahlada Rao Program Co ordinator SSDG CDAC Bangalore Release By SSDG C DAC Knowledge Park Bangalore Date of Release 16 06 2010 Copy No 1 Contents Page No GSRM v 1 0 User Manual 16 6 10 Page 2 of 26 SSDG CDAC KP Bangalore CICAC 1 GSRM Introduction 4 7 INNEN H ESI ago Tom 4 1 2 Purpose and Scope of the Document ooo coran RETIRER UR PIER EDU Pe vere 4 1 3 GSRM Highlights amp Supported features ccc cece eee e cece ee eeeeeees 4 1 4 GSRM Test Bed Deployment orsi orooio reo pERETO 2012109 riti ii in E ENESE 5 LIPS PE odi Bee RARO 6 1 6 GSRM Client Interfaces 0 0 e cece IRI II e emen 7 2 GSRM Interface Usage 8 27 PARC SES EE Aute 7 Z2 2 Pees te TOF We USR cco nn rcgnnosssteanianertes Er TAREE NE EENEN 7 2 2 o LORM command BIG disc
9. RM storage GSRM v 1 0 User Manual 16 6 10 Page 20 of 26 SSDG CDAC KP Bangalore GSRM v 1 0 U Bangalore CDAC Downloading a file can be done using srm copy command Usage srm copy srcurl targeturl serviceurl srm gsrmserver srm copy srm xnO05 dpm ctsf cdac org in home test inputfilel file tmp getfile 1 serviceurl srm xn05 8446 dpm Output srm copy 2 2 1 3 6 Mon Nov 30 09 10 48 PST 2009 SRM Clients and BeStMan Copyright c 2007 2009 Lawrence Berkeley National Laboratory All rights reserved Support at SRM LBL GOV and documents at http datagrid lbl gov bestman SRM CLIENT Thu May 06 13 58 09 IST 2010 Connecting to httpg xn05 8446 dpm SRM CLIENT Thu May 06 13 58 10 IST 2010 Calling SrmPrepareToGet Request now request token 686b6b39 8 bf 497f a645 65e7f1 1 b69ae Request status SRM_REQUEST_QUEUED Request explanation null SRM CLIENT Next status call in 10 seconds SRM CLIENT Thu May 06 13 58 21 IST 2010 Calling Status at Thu May 06 13 58 21 IST 2010 Request status SRM_SUCCESS SRM CLIENT RemainingPinTime 7 190 SRM CLIENT SURL srm xn05 dpm ctsf cdac org in home test inputfile1 SRM CLIENT FileStatus code from server SRM_FILE_PINNED SRM CLIENT Explanation from server null SRM CLIENT received TURL gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storage dat a0 1 test 2010 05 06 inputfile1 1331 0 SRM CLIENT Thu May 06 13 58 21 IST 2010 start file transfer
10. S printf failed to load the globus ftp module M GSRM v 1 0 User Manual 16 6 10 Page 16 of 26 SSDG CDAC KP Bangalore CICAC ifglobus ftp client handleattr init amp handle attr GLOBUS SUCCESS printf client handleattr init failed n ifglobus ftp client operationattr init amp attr 2 GLOBUS SUCCESS printf operation attr init failed n ifglobus ftp client handle init amp handle amp handle attr GLOBUS SUCCESS printf client handle init failed n for 120 1 nbfiles 14 4 done GLOBUS_FALSE if globus_ftp_client_third_party_transfer amp handle source_array 1 GLOBUS_NULL dest_array i GLOBUS NULL GLOBUS_NULL thirdparty_file_transfer_cb GLOBUS_NULL GLOBUS_SUCCESS printf globus ftp client third party transfer failed n else printf success M globus mutex lock amp lock while done globus cond wait amp cond amp lock j globus mutex unlock amp lock j if globus module deactivate all GLOBUS SUCCESS printf Failed to deactivate the globus ftp module Mn I HHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE end of globus C apis for a party transfer THHHHHHHHHBHHHHBHHBHBHHHBHHHHHHHBRHHBHHHBHHHHBHHHBHBHHHBHHHBHHHBHHHBHBHBEE for 120 1 nbfiles 14 4 free dest_array 1 I HHHHHHBHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE check for put done surls surl array if status dpm putdone r token nbfiles surls amp nbpdreplies amp pdfilestatuses 0 sperror
11. SRM CLIENT Source gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storage 1 dat a01 test 2010 05 06 inputfilel 1331 0 srm xn05 dpmvctsf cdac org in home test inputfile1l SRM CLIENT Thu May 06 13 58 23 IST 2010 Calling releaseFile SRM CLIENT Calling srmReleaseFiles statuszSSRM SUCCESS explanation null status SRM_ SUCCESS explanation null SRM CLIENT Request completed with success SRM CLIENT Printing text report now SRM CLIENT REQUESTTYPE get SRM CLIENT TOTALFILES 1 SRM CLIENT TOTAL_SUCCESS 1 SRM CLIENT TOTAL FAILED O SRM CLIENT REQUEST_TOKEN 686b6b39 8 bf 497f a645 65e7f1 1b69ae SRM CLIENT REQUEST_STATUS SRM_SUCCESS SRM CLIENT SOURCEURL 0 srm xn05 dpm ctsf cdac org in home test inputfilel SRM CLIENT TARGETURL 0 file tmp getfile1 SRM CLIENT TRANSFERURL 0 gsiftp pvfs4d ctsf cdac org in pvfs4 ctsf cdac org in pvfs2st ore storage1 data01 test 2010 05 06 inputfile1 1331 0 SRM CLIENT ACTUALSIZE 0 225 SRM CLIENT FILE_STATUS 0 SRM_FILE_PINNED CDAC NOTE put get a file to from GSRM server can be done using same command srm copy in a single step 2 7 BestMan Java APIs Usage BestMan Java APIs are available under the samples directory of the client installation directoryCheck the example programs for the basic operations like ping ls get put dir etc under the samples directory of bestman java api folder and add or extend your own functions to perform the req
12. an be accessed from here using StoRM command line interface If the user wants to use the clientSRM StoRM Clients from gridfs machine 1 Create a valid user proxy erid proxy init Enter GRID pass phrase for this identity RTCA DEO a Done Your proxy is valid until lt Date Time gt 2 Set the env variable for Globus location path export GLOBUS_LOCATION GLOBUS_LOCATION usr local GARUDA GLOBUS 4 0 7 export PATH PATH opt gsrm client srmv2storm bin export LD_LIBRARY_PATH LD_LIBRARY_PATH opt gsrm client cgsi_soap lib 3 Run the clientSRM command 2 pvfs2 172 20 1 81 pvfs2 is the GSRM testing node with the following client interfaces installed e StoRM Bestman and dpm command line interfaces e Bestman Java APIs e DPMC APIs The usage of above mentioned interfaces has been discussed on the following sub sections 3 GSRM Web Client is accessible from any of the user machines reachable to GSRM server xn05 ctsf cdac org in using URL https xn05 ctsf cdac org in 2 3 StoRM command line usage The usage of StoRM command line client is as follows clientSRM h Usage clientSRM lt requestName gt lt requestOptions gt lt requestName gt The various functions supported by StoRM are i Discovery functions Ping GetTransferProtocols or gtp ii Directory functions Mkdir Rmdir Rm Ls Mv in Data transfer functions PtP PtG Copy or cp StatusPtP or sptp StatusPtG or sptg StatusCopy or scp PutDone
13. const char source url elobus ftp client operationattr t source attr dest attr attr const char dest url globus object t error if argc 2 1 fprintf stderr usage os SURLs SOURCE TURLs Ww argv 0 exit 1 if argc 2 0 nbfiles argc 2 if reqfiles calloc nbfiles sizeof struct dpm putfilereq NULL perror calloc exit 1 j for G 0 1 nbfiles i regfiles 1 to surl argv 141 nbprotocols sizeof protocols sizeof char if status dpm put nbfiles reqfiles nbprotocols protocols NULL 0 0 r token amp nbreplies amp filestatuses 0 sperror dpm put exit 1 j printf dpm put returned r token s n r token wait for request status Done or Failed while status DPM QUEUED ll status DPM ACTIVE for 12 0 1 lt nbreplies i if filestatuses 1 gt to_surl free filestatuses 1 gt to_surl if filestatuses 1 gt turl free filestatuses 1 gt turl if filestatuses 1 gt errstring free filestatuses 1 gt errstring free filestatuses sleep r 0 1 DEFPOLLINT if status dpm getstatus putreq r token 0 NULL amp nbreplies amp filestatuses lt 0 sperror dpm_getstatus_putreq exit 1 GSRM v 1 0 User Manual 16 6 10 Page 15 of 26 SSDG CDAC KP Bangalore CDAC j printf request state s n status DPM DONE Done Failed if stat
14. e CDAC Execution export DPM_HOST xn05 ctsf cdac org in export DPNS_HOST xn05 ctsf cdac org in ecc o testprotocols dpm testgetprotocols c I opt lcg include dpm L opt lcg lib Results Jtestprotocols Supported protocols are rfio gsiftp https Example 2 A sample C Program using DPM C APIs to Upload a File into GSRM Storage Code include lt stdio h gt include lt stdlib h gt include lt sys types h gt include dpm_api h include serrno h define DEFPOLLINT 10 include globus_common h include globus ftp client h elobus mutex t lock elobus cond t cond elobus bool t done main argc argv int argc char argv static char f stat Success Queued Active Ready Running Done Failed Aborted struct dpm putfilestatus filestatuses struct dpm filestatus pdfilestatuses int 1 J int nbfiles int nbprotocols int nbreplies nbpdreplies static char protocols gsiftp int r 0 char surls char r token CA MAXDPMTOKENLEN 1 struct dpm putfilereq reqfiles int status char source_array 10 char dest array 10 GSRM v 1 0 User Manual 16 6 10 Page 14 of 26 SSDG CDAC KP Bangalore CDAC char surl array 10 elobus ftp client handle t handle elobus ftp client handleattr t handle attr int thirdparty file transfer cb void globus ftp client handle t globus object t
15. icccnosc Deko pO FR ER EDU DEDOS RU PIENE DESIT RS 8 24 DPM Web Interia TEC 12 PX IQUESO Du 13 2 0 Bosi Man command EE eror conce t erooE Ee teo utet enetetei bs ros E 18 2 7 BestMan Java APIS 0 ccc ccc cece cece e cence nec e cee encensenseesecceccuceuceneess 22 o a AE EA ES A O EE EEEE EEEE E a a GSRM v 1 0 User Manual 16 6 10 Page 3 of 26 SSDG CDAC KP Bangalore CDEC 1 GSRM Introduction 1 1 Overview Data grid is a system that manages and controls large amounts of distributed data Heterogeneous Grid environments demand a global namespace in order to present a Uniform storage space providing transparency of location as well as access to data Grid applications interact with a variety of storage systems Hence there is a need to standardize and streamline the uniform access interface to grid storage system Such a system not only manages storage allocation in a transparent manner but also dynamically manages the spaces on Grid GARUDA Storage Resource Manager is a peer to peer data grid solution for SOA based GARUDA An SRM can be defined as a middle ware component that manages the dynamic use of storage resources on the Grid This means that space can be allocated dynamically to a client and that the decision of which files to keep in the storage space is controlled dynamically by the SRM GSRM is a disk based SRM implementation following SRM version 2 2 Specifications https sdm
16. idfs So users can download the input files from GSRM before submitting the job and can also store the generated output files in it after the completion of the job GSRM will also accessible through GARUDA portal in future Figure 3 depicts the GSRM integration and interaction with other GARUDA middleware components with respect to the job submission by the user Myproxy Bio ud Head Node Voms i 7 proxy Schedule clients Space manager name space m manager Srm server GARUDA Federated Information system GSRM Information server Figure 3 GSRM Usage by GARUDA Users GSRM v 1 0 User Manual 16 6 10 Page 6 of 26 SSDG CDAC KP Bangalore CDAC Steps to be followed to access GSRM from gridfs Login to gridfs 192 168 60 40 Upload your IGCA user certificates under lt user home dir gt globus Initialize proxy with grid proxy init Set environmental variables mentioned in section respectively for whichever client to be used Run the SRM commands Details discussed in section 2 for each interface d ab em 1 6 GSRM Client Interfaces GSRM services can be accessed by the following client interfaces 1 Command Line The various command line interfaces available to access GSRM services are StoRM clientSRM commands BestMan and dpm command line interfaces i APIs GSRM supports DPM C APIs for C based applications and BestMan Java APIs for Java based applicati
17. lbl gov srm wg It adheres to Open Grid Forum standards It is based on open source Disk pool Manager DPM Version 1 7 2 1 2 Purpose and Scope of the Document This document is intended for the GARUDA users The purpose of this document is to help GARUDA users to use GSRM storage services It describes the step by step procedure to access GARUDA storage resources using different SRM client interfaces 1 3 Highlights and Features The GSRM version1 0 supports the following features e Provides single access point to distributed and heterogeneous storage resources for users by providing global namespace e Provides File and Directory management e Supports GSI Access Control Lists and VOMS Security mechanism and supports secure data transfers e GSRM provides dynamic space management reserving amp release of storage spaces and setting lifetime for the files e User Quota Allocation Each user is allocated 10GB of default which can be increased based on the user request e Provides user friendly interfaces to access its services e Provides interoperability with other SRM implementations based on SRMv2 2 specifications like Best Man Berkeley Storage Manager Castor SRM dCache SRM StoRM Storage Resource Manager Users can access GSRM storage services using any of the above mentioned client interfaces Figure 1 shows the various GSRM components and their feature mapping GSRM components are depicted with green boxes and their
18. ng the urls c setSourceUrl from c setDestinationUrl to c setUseThirdPartyCopy true c copy Jcatch Exception e e printStack Trace j j Execution set the classpath All jar files which are present under lib folder of bestman root path should be added to CLASSPATH eg export CLASSPATH home test bestman java api lib Berkeley StorageResourceManager client ap1 jar javac SRMGetTest java GSRM v 1 0 User Manual 16 6 10 Page 24 of 26 SSDG CDAC KP Bangalore C ODEAC java SRMGetTest serviceurl httpg xnO5 ctsf cdac org in 8446 surls srm xnO5 ctsf cdac org in dpm ctsf cdac org in home garuda file logpath logfile desturl file SPWD getsrmfile Results File will be transfered from sourceurl to desturl 1 e from GSRM server to the target location Check the logs for more information Explanation In the above code first the command line parameters are parsed to check for correct values using those it will connect to the srm server and processes the request based on the request type eg get put and checks for the request status once the status returned 1s SRM SUCCESS ll SRM FILE PINNED we can retrieve the TURL and can be passed to CoG api s to transfer the file from source turl returned by srm server to the destination desturl passed as parameter 3 GSRM Support To avail access and support to GARUDA SRM Storage System send a mail to rt gds cdacb ernet in 4 Glossa
19. om SRM srmStatusOfPutRequest SRM_SUCCESS SRM CLIENT RemainingPinTime 7190 SRM CLIENT RemainingFileLifeTime 0 SRM CLIENT received TURL gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storage1 data01 test 201 0 05 06 inputfile1 1331 0 SRM CLIENT Thu May 06 13 57 10 IST 2010 start file transfer SRM CLIENT Source file tmp file1 SRM CLIENT Target gsiftp pvts4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storage data0 L test 2010 05 06 inputfilel 1331 0 SRM CLIENT Thu May 06 13 57 12 IST 2010 end file transfer for file tmp file1 SRM CLIENT Thu May 06 13 57 12 IST 2010 Calling putDone for srm xn05 dpm ctsf cdac org in home test inputfile 1 Result status SRM_SUCCESS SRM CLIENT Request completed with success SRM CLIENT Printing text report now SRM CLIENT REQUESTT YPE put SRM CLIENT TOTALFILES 1 SRM CLIENT TOTAL_SUCCESS 1 SRM CLIENT TOTAL_FAILED 0 SRM CLIENT REQUEST_TOKEN 200850ef 2def 40ab bebe da3d0107el1d SRM CLIENT REQUEST_STATUS SRM_SUCCESS SRM CLIENT SOURCEURL O0 file tmp filel SRM CLIENT TARGETURL 0 srm xn05 dpm ctsf cdac org in home test inputfile1 SRM CLIENT TRANSFERURL 0 gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2 store storage1 data0 1 te st 2010 05 06 inputfilel 1331 0 SRM CLIENT ACTUALSIZE 0 0 SRM CLIENT FILE STATUS 0 ZSRM SUCCESS SRM CLIENT EXPLAN ATION 0 ZSRM CLIENT PutDone is called successfully Example 3 Download a file from GS
20. ons 111 Web Interface DPM web interface is available to access GSRM services through browser 2 GSRM Interface Usage 2 1 GSRM Access points Currently GSRM services can be accessed logging into GARUDA Bangalore head node gridfs 192 168 60 40 or pvfs2 172 20 1 81 SRM client installations are available on both the above mentioned nodes 1 pvfs2 172 20 1 81 node should be used to just test all the available SRM client interfaces like StoRM DPm BestMan 2 gridfs 192 168 60 40 node should if the user wishes to use GSRM storage for job execution Users can download Upload input output files into GSRM while submitting jobs from gridfs 2 2 Pre requisites for using the GSRM services e A valid user IGCA certificate Valid user certificate can be obtained through IGCA web site https ca garudaindia in e A SRM endpoint to contact httpg xn05 ctsf cdac org in 8446 e Unix Login for GARUDA Grid Head node gridfs mail to grid help cdacb ernet in or pvfs2 mail to rt gds cdacb ernet in e Authorization to access SRM storage resources mail to rt gds cdacb ernet in e SRM client interface Installation Any of the above mentioned SRM client installation GSRM v 1 0 User Manual 16 6 10 Page 7 of 26 SSDG CDAC KP Bangalore CDAC gt Following Access mechanisms are available at above mentioned nodes to access GSRM 1 gridfs 192 168 60 40 gridfs is the Bangalore GARUDA head node GSRM services c
21. r Command clientSRM ptg e httpg xn05 8446 s srm xnO05 dpnvctsf cdac org in home test testfile2 p Output Polling request status Current status SRM REQUEST QUEUED Ctrl c to stop polling Request status statusCodeZ SRM SUCCESS 0 SRM Response request Token 8003ef88 5624 47 10 970a 549867 1ad08c arrayOfFileStatuses size 1 0 status statusCodeZ SRM FILE PINNED 22 0 sourceSURL srm xn05 dpm ctsf cdac org in home test testfile2 0 fileS1ze25990933 0 remainingPinTime 7199 0 transferURL gsiftp pvfs4A ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storagel data0 1 test 2010 01 09 testfile2 771 0 Step 2 File transfer Command globus url copy gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storagel data0 test 201 0 01 09 testtile2 771 0 file home shashi get_testfile2 2 4 DPM Web Interface Usage DPM Web interface provides user friendly GUI to access GSRM services It can be accessed using the url https lt dpm head node gt Test Bed URL https xn05 ctsf cdac org in DPM provides a basic GUI with minimal functions like file amp directory management file upload download etc Steps to Access DPM Web interface GSRM v 1 0 User Manual 16 6 10 Page 11 of 26 SSDG CDAC KP Bangalore CDAC Upload the user certificate pkcs12 format file into the browser refer link http ca garudaindia in index php information faq 2 https lt dpm head node gt
22. ry 1 Storage Element SE Grid service that allow users to store and manage files on assigned Space 2 Logical File Name LEN alias created by a user to refer to some item of data Ifn grid lt My VO gt lt MyDirs gt lt MyFile gt 3 Site File Name SFN is a file name assigned by a site to a file Normally the site file name will consist of a machine port directory LFN but the site can choose to use another string instead of the LFN 4 Grid Unique Identifier GUID A non human readable unique identifier for data entity 5 Site URL SURL Physical file name or the location of an actual piece of data on a storage system lt srm gt lt SE_hostname gt lt path gt 6 Transport URL TURL Temporary location of a replica access protocol understood by a SE lt protocol gt lt path gt GSRM v 1 0 User Manual 16 6 10 Page 25 of 26 SSDG CDAC KP Bangalore
23. s SRM_REQUEST_QUEUED Ctrl c to stop polling Request status statusCode SRM_SUCCESS 0 SRM Response requestToken 635388bb 2e92 4a9f b590 57c99dfcdb38 arrayOfFileStatuses size 1 0 SURL srm xn05 dpm ctsf cdac org in home test fastainputfile 0 status statusCode SRM_SPACE_AVAILABLE 24 0 fileSize 0 0 remainingPinLifetime 7199 0 remainingFileLifetime 0 TURL gsiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store stor age1 data01 test 2010 05 05 fastainputfile 1312 0 Step 2 Globus File transfer File actually gets transferred to SRM location globus url copy lt source gt lt target gt i e globus url copy source TURL returned by step 1 Command globus url copy file home shashi em rel std pln esiftp pvfs4 ctsf cdac org in pvfs4 ctsf cdac org in pvfs2store storagel dataO1 t est 2010 05 05 fastainputfile 1312 0 Step 3 pd putdone use the token returned in step 1 Command clientSRM pd e httpg xn05 8446 s srm xn05 dpm ctsf cdac org in home test fastainputfile t 635388bb 2e92 4a9f b590 57c99dfcdb38 Output Request status statusCode SRM_SUCCESS 0 GSRM v 1 0 User Manual 16 6 10 Page 10 of 26 SSDG CDAC KP Bangalore SRM Response arrayOfFileStatuses size 1 0 SURL srm xn05 dpnvctsf cdac org in home test fastainputfile 0 status statusCodeZ SRM SUCCESS 0 Example 3 Download a file from GSRM storage Step 1 ptg prepare to get request to GSRM serve
24. t Protected Mode On fav 150 ES EC AEO N ENL ne Lacs N i cc Mater Mena Ss ppmitiss e PS DOMUI GSRM v 1 0 User Manual 16 6 10 Page 12 of 26 SSDG CDAC KP Bangalore CDAC 2 5 DPM C APIs Usage ref url https twiki cern ch twiki bin view LCG DpmApi Table 1 shows the list of DPM C APIs currently supported by GSRM dpm copy copy a set of existing files dpm get make a set of existing files available for I O getstatus copyreq get status for a dpm copy request pm getstatus putreq get status for a dpm put request dpm put make a set of existing files available for I O dpm putdone mark a set of files as complete pr pm_getstatus_getreq get status for a dpm_get request Table 1 List of DPM Example 1 A sample C Program using DPM C APIs to Show the supported transfer protocols by the GSRM Code include lt sys types h gt include opt lcg include dpm dpm api h include opt lcg include dpnvserrno h main i 1nt 1 int nb supported protocols char supported protocols if dpm getprotocols amp nb supported protocols amp supported protocols 0 A Sperror dpm getprotocols exit 2 FERE printf Supported protocols are for 12 0 1 nb supported protocols 1 printf 96s supported protocols 1 free supported protocols 1 _ free supported protocols exit 0 i GSRM v 1 0 User Manual 16 6 10 Page 13 of 26 SSDG CDAC KP Bangalor
25. tatusCode TStatusCode SRM FILE PINNED HashMap map response getFileStatuses Set set map entrySet Iterator 1 set iterator while i hasNext GSRM v 1 0 User Manual 16 6 10 Bangalore Map Entry me Map Entry i next String key String me getKey Object value me get Value if value null FileStatus fileStatus FileStatus value Page 23 of 26 SSDG CDAC KP CDAC org apache axis types URI uri fileStatus getTransferSURL 0 System out println nTransferSURL uri turl uri toString System out println turl is turl String temp turl split 2811 turl_final temp 0 temp 1 System out println final turl is turl_final j V end while V end if end if if releaseFile before release file is called user will do the file transfer either with globus url copy or with other tool to the SRM server ie copy the SRM server s transfer url to the local machine and when the transfer file is successful user will call release file req releaseFiles surl response req getStatus req printResults j cc disconnect GlobusURL from new GlobusURL turl final GlobusURL to new GlobusURL file home os pfs getfile txt System out printIn globusurl copy info System out printIn source url from System out printIn destination url to System out printIn globusurl copy infofHHHHBE UrlCopy c new UrlCopy Setti
26. tp client gcc32dbgpthr Execution usage dpm_testput2 SURLs SOURCE TURLs dpm_testput2 srm xn0O5 dpm ctsf cdac org in home test capi_putfile4 gsiftp g11 home shashi env srm Result dpm put returned r token e5al0f76 2a28 4742 8d64 55a0f6135298 request state Pending request state Failed state O Ready TURL gsiftp pvfs4 ctsf cdac org eao ctsf cdac org in lovfs2store storage data0 test 2010 05 05 capi putfile4 1324 0 nfiles 1 source array 0 gsiftp g11 home shashi env srm globus c api SUCCESS GSRM v 1 0 User Manual 16 6 10 Page 18 of 26 SSDG CDAC KP Bangalore CDAC Status file transfer successful state 0 Done Example 2 To Download a File from GSRM Storage Its is similar to put a file to GSRM server APIS to be used are dpm_get dpm getstatus getreq velobus ftp client third party transfer NOTE All the above sample programs are available on pvfs2 node NOTE Refer the list of available apis of DPM and DPNS and also the corresponding manpages ref url https twiki cern ch twiki bin view LCG DpmApi http grid deployment web cern ch grid deployment documentation LFC_DPM dpm 2 6 BestMan Command line Usage Example 1 ping to gsrm server xn05 Command Usage srm ping serviceurl httpg xn05 ctsf cdac org in 8446 dpm ctsf cdac org in home garuda Output srm ping 2 2 1 3 6 Mon Nov 30 09 10 48 PST 2009 SRM Clients and BeStMan Copyright c 200
27. uired tasks using the available api s from the Bestman java api s document under betman java api docs javadoc folder Table 2 shows the list of Bestman Java classes and the methods supported by them Class Methods SRMServer connect java lang String serverUrl SRMServer class is responsible for disconnect establishing and maintaining connections ping java lang String uid with the SRM Server SRMRequest setRequestType java lang String requestType The SRMRequest object holds all outstanding request information and status addFiles java lang String surls java lang String copyTargetSurls long fileSize setAllLevelsRecursive boolean allLevelsRecursive setAuthID java lang String authId setCount int count setFileLifeTime int fileLifeTime setFileStorageType java lang String tokenType setNumberLevels int numLevels setPinLifeTime int pinLifeTime setRequestToken java lang String reqId setRetentionPolicy java lang String retentionPolicy setSRMServer SRMServer srmServer setStorageSystemInfo java lang String storageInfo setTargetSURLLifeTime int fileLifeTime srmLs srmkm submit checkStatus checkStatus java lang String surl java lang String copyTargetSurl getAllLevelsRecursive getCount getFiles getFullDetailedList getNumberLevels getRequestSummary getRequestToken getRequestType getSRMServer getStatus printResults GSRM v 1 0 User Manual 16
28. us DPM FAILED exit 1 for 12 0 1 lt nbreplies i if filestatuses 1 gt turl dest array 1 char malloc strlen filestatuses 1 gt turl if dest array 1 NULL printf malloc error M strcpy dest_array 1 filestatuses 1 gt turl printf state d s TURL Ps n 1 _stat filestatuses i gt status gt gt 12 filestatuses 1 gt turl else if filestatuses i gt status amp DPM FAILED DPM FAILED printf state d s serrno d errstring lt s gt n 1 _stat filestatuses 1 gt status gt gt 12 filestatuses i gt status amp OxFFF filestatuses i gt errstring filestatuses 1 gt errstring else printf state d s n 1 f_stat filestatuses 1 gt status gt gt 12 if filestatuses 1 gt to_surl free filestatuses 1 gt to_surl if filestatuses 1 gt turl free filestatuses 1 gt turl if filestatuses 1 gt errstring free filestatuses 1 gt errstring free filestatuses for i2nbfiles1 j20 1 argc 14 4 printf nfiles d nbfiles source_array j argv 1 printf source_array d s n j l source array j 1 for 120 1 nbfiles 14 4 surl_array i argv i 1 THHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHE globus C apis for third party transfer THHHHHHHBHHHBHHHBHHHHHHHHHHHHHBHHHBHHBHHHBHHHBHHHHHBHHHBHHHBHE ifglobus module activate GLOBUS FTP CLIENT MODULE GLOBUS_SUCCES
Download Pdf Manuals
Related Search