Stork User Manual
Contents
1. stork transfer petashare stork transfer globus url copy stork transfer unitree stork globus url copy stork globus url copy64 stork transfer irods stork transfer srb local log local log sbin stork_server Please note that Stork uses Globus components for file transfers using the GsiF TP protocol GridFTP Therefore you need to have Globus installed and configured in your system with user host certificates Stork will perform the requested operation using stork globus url copy command in the libexec directory It is basically an extended version adding some extra features to be used by Stork of globus url copy from Globus 7 You can replace that command by globus url copy in your system Simply delete the file and put a link named as stork globus url copy pointing to the gridF TP client utility in your system We have Stork commands in bin directory and Stork server in sbin Sam ple submit files and templates for Stork configuration can also be found in etc directory Stork log files are kept in local log by default Transfer modules are in libexec and installation script will create appropriate symbolic links for each lhttp www globus org grid_software data gridftp php http www globus org toolkit transfer pair i e gsiftp to file stork transfer file file stork transfer ftp file gt stork transfer globus url copy stork transfer file ftp gt stork transfer globus url copy stork trans2. Stork support recursive directory transfers Recursive directory transfers are specified in the URL by ending the URLs with a to represent a directory Wild Card Support Currently all the transfer modules supported by Stork support transferring files with a wild cards such as txt stork stork or st rk Checkpointing File Transfers Currently the Petashare and GridF TP transfer modules supported by Stork support checkpointing of transfer and provide the capability of resuming trans fers in the event of an error These Stork modules checkpoint the transfers during various stages and thus enable Stork to resume the transfer at the last checkpoint in the event of a network outage or crash 6 Job Submission Since Stork is designed to work on a heterogeneous computing environment one of its goals is to support as many storage systems and file transfer protocols as possible Currently the following protocols and storage systems are supported by Stork e file e FTP e GridFTP e HTTP 10 e iRODS e PetaShare e SRB The protocol to be used is determined by the Stork server according to the URL signatures of the files to be transfered URLs supported The format of the URL for various supported protocols is as below file URL file path to file e FTP URL ftp user password host port path to file e HTTP URL http user password host port path to file GridFTP URL gsiftp user password host port path to fi
3. stork_q h usage stork_q option stork_server stork_server stork server deprecated help print this help information version print version information debug print debugging information to console name stork_server stork server Sample output from the stork q command dest_url file home user1 stork data10M_48 src_url file home user1 stork data10M remote_user user1 dsl turtle06 cct lsu edu status request_rescheduled dap_id 264 use_protocol 0 stork_server qbi loni org dap_type transfer error_code port not accessible num_attempts 1 owner useri cluster_id 264 timestamp absTime 2008 05 28T14 52 22 0500 generic_event Rescheduling Stork server runs as a persistent daemon process It consistently listens to requests from the clients The clients send their requests to the Stork server over the network using stork_submit command line tool in form of a ClassAd Classified Advertisement Since Stork is designed to work on a heterogeneous computing environ ment one of its goals is to support as many storage systems and file transfer protocols as possible Another important characteristic of Stork is reliability It makes sure that the requested transfers are completed successfully even in case of server or net work failures Stork source and destination URLs have a naming convention All URLs ending with a slash are assumed to be directories and th
4. 0 IKI II I I I I IK KEK k 10 23 16 55 24 Using config source tmp etc stork_config 10 23 16 55 24 DaemonCore Command Socket at lt 208 100 92 21 47661 gt 10 23 16 58 21 FORRES SIO I I I I I kK kk 10 23 16 58 21 stork_server STORK STARTING UP 10 23 16 58 21 tmp sbin stork_server 10 23 16 58 21 Version 6 9 4 Sep 11 2008 10 23 16 58 21 Platform I386 LINUX_RHEL3 10 23 16 58 21 PID 14336 10 23 16 58 21 Log last touched 10 23 16 55 24 10 23 16 58 21 kkk kk ak 3k ak k 3k 3k 3k I I II I I I I I I I 3K 3K 2K 3K 21 K 21 3K K 21 K K a K K 10 23 16 58 21 Using config source tmp etc stork_config 10 23 16 58 21 DaemonCore Command Socket at lt 208 100 92 21 47712 gt 10 23 16 58 21 10 23 16 58 21 10 23 16 58 21 10 23 16 58 21 DaP log file storkserver log 10 23 16 58 21 Userlog file null 10 23 16 58 21 XML log file null 10 23 16 58 21 Client Agent host null 10 23 16 58 21 10 23 16 58 21 STORK_TEST_MODE 10 0 do testing only 10 run as server 10 23 16 58 21 STORK_MAX_NUM_JOBS 1 10 23 16 58 21 STORK_MAX_RETRY 1 10 23 16 58 21 STORK_MAXDELAY_INMINUTES 0 10 23 16 58 21 STORK_AGGR_LEVEL 0 0 no aggregation 1 according to dest url 2 according to src url 3 if hostnames match 10 23 16 58 21 STORK_MAX_COUNT 1 10 23 16 58 21 STORK_RECURSIVE_COPY FALSE 10 23 16 58 21 STORK_VERIFY_CHECKSUM FALSE 10 23 16 58 21 STORK_VERIFY_F
5. ILESIZE FALSE 10 23 16 58 21 STORK_NETWORK_CHECK FALSE 10 23 16 58 21 STORK_TRANSFER_CHECKPOINT FALSE 10 23 16 58 21 STORK_SYNC_ONLY FALSE 10 23 16 58 21 STORK_TMP_CRED_DIR tmp 10 23 16 58 21 STORK_MODULE_DIR tmp libexec 10 23 16 58 21 modules will execute in LOG directory tmp local log 10 23 16 58 21 Getting monitoring info for pid 14336 4 Stork Components stork_server The stork_server is the main component of the Stork scheduler The stork_server runs as a persistent daemon process and performs all management scheduling execution and monitoring of data placement activities The Stork server accepts the following parameters defined by STORK_ARGS sbin stork_server help USAGE stork_server t output to stdin p port on which to run Stork Server help stork help screen Config stork config file Serverlog stork server log in ClassAds Xmllog stork server log in XML format Userlog stork userlog in XMLformat Clientagent host where client agent is running stork_submit The stork_submit is a client side tool used to submit stork jobs to the stork_server bin stork_submit usage stork_submit option stork_server submit_file stork_server specify explicit stork server deprecated submit_file stork submit file lognotes notes add lognote to submit file before processing stdin read submission from stdin instead of a fil
6. ISTORY SPOOL history Where is the Stork binary installed STORK SBIN stork_server STORK_ADDRESS_FILE LOG stork_address STORK_LOG_BASE Stork server job queue classad collection journal file STORK_LOG_BASE history Used to track completed jobs STORK_LOG_BASE user_log User level log also used by DAGMan STORK_LOG_BASE LOG Stork STORK_LOG LOG StorkLog STORK_DEBUG D_FULLDEBUG MAX_STORK_LOG 4000000 Stork startup arguments Start Stork on a well known port STORK_PORT 9621 STORK_ARGS p STORK_PORT f Serverlog STORK_LOG_BASE Limits the number of retries for a failed data placement default 10 STORK_MAX_RETRY 1 Limits the run time for a data placement job after which the placement is considered failed by default 0 infinite STORK_MAXDELAY_INMINUTES 10 Temporary credential storage directory used by Stork STORK_TMP_CRED_DIR tmp Directory containing Stork modules STORK_MODULE_DIR LIBEXEC max number of jobs running at the same time default 1 STORK_MAX_NUM_JOBS 1 Temporary credential storage directory used by Stork STORK_TMP_CRED_DIR tmp aggregating jobs default is 0 no aggregation 1 aggregation according to dest_url 2 src_url 3 if src_hostname and dest_hostname matches 4 dest_hostname src_hostname STORK_AGGR_LEVEL 0 set max number of job that can be aggregated comb
7. Stork User Manual version 1 0 0 1 Quick Start tar xvfz stork_binary_1 0 0 tar gz cd stork_binary_1 0 0 1s release tar gz stork install ksh stork install ksh export STORK_CONFIG usr local stork etc stork_config export PATH PATH usr local stork bin usr local stork sbin Extract Stork binary package and execute the installation script Set envi ronment variable STORK_CONFIG pointing to your configuration file Start the server by running stork_server command Check sample submit files in etc under Stork installation directory Prepare your job submission file and submit using stork_submit 2 Installation First extract the Stork binary package to get the release tar gz and the instal lation script stork install ksh tar xvfz stork_binary_1 0 0 tar gz cd stork_binary_1 0 0 In the current directory run Stork installation script by giving the installa tion directory as argument by default it will try to install into usr local stork stork install ksh user1 stork Installation directory is home user1 stork Proceed yes no yes stork install ksh extracting binaries release_file tmp release tar gz bin bin stork_rm bin stork_submit bin stork_status bin stork_q local local log stork_home etc etc stork_config sample etc submitArgument sample etc stork_config template etc stork_config etc submit sample ibexec ibex
8. e help print this help information version print version information debug print debugging information to console name stork_server stork server stork_status The stork_status is a client side tool used to query regarding the status of jobs submitted to the stork_server The dap_id is used by the stork_status commmand to query the Stork server The dap _id is generated and assisgned to a job when it is submitted to Stork using the stork_submit command The stork_status command accepts the following parameters where host_name is optional The host_name is used to specify a Stork server on a remote host stork_status h usage stork_status option stork_server job_id stork_server specify explicit stork server deprecated job_id stork job id help print this help information version print version information debug print debugging information to console name stork_server stork server stork_rm The stork_rm is a client side tool used to delete any jobs that are currently queued with the stork_server stork_rm h usage stork_rm option stork_server job_id stork_server specify explicit stork server deprecated job_id stork job id help print this help information version print version information debug print debugging information to console name stork_server stork server stork_q The stork_q is a client side tool used to retrieve a listing of jobs that are currently queued with the stork_server
9. e rest are assumed to be files 5 New Features Here is a possible submit file including extended features etc submitArgument sample dap_type transfer src_url gsiftp src loni org home balman tests srcfile dest_url gsiftp dest loni org home balman tests dest destfile output out err err arguments p 10 set_permission 066 sync_only true checkpoint_transfer true network_check true verify_filesize true recursive_copy true File size Verification Support Currently all the transfer modules supported by Stork support file size ver ification File size verification can either be turned ON OFF by specifying the corresponding option in the Stork configuration file When switched ON Stork determines the filesizes of the files at the source and the files at the destination and compares them If the filesizes differ an error message is logged in the Stork log file Checksum Verification Support Currently all the transfer modules supported by Stork support checksum verification Checksum verification can either be turned ON OFF by specifying the corresponding option in the Stork configuration file When switched ON Stork computes the checksums of the files at the source and the files at the destination and compares them If the checksums differ an error message is logged in the Stork log file Recursive Transfers Currently all the transfer modules supported by
10. ec stork transfer castor_srm ibexec stork globus url copy ibexec stork transfer srb ibexec stork transfer globus url copy ibexec stork transfer dcache_srm ibexec stork transfer petashare ibexec stork transfer irods ibexec stork transfer file file ibexec stork transfer unitree sbin sbin stork_server ln Se Maa Shae ata a Ga SSN SS stork install ksh creating configuration file Please set STORK_CONFIG environment variable export STORK_CONFIG home user1 stork etc stork_config By default stork_server will search usr local stork etc for stork_config file if STORK_CONFIG environment variable is not defined Please set PATH environment variable to Stork binaries export PATH PATH home user1 stork bin home user1 stork sbin Installation Complete The STORK_CONFIG environment variable should point to the stork_config file Installation script prepares the configuration file and puts under etc in the installation folder By default stork_server will search usr local stork etc for stork_config file if STORK_CONFIG environment variable is not defined In the installation folder we have the following directory structure bin etc libexec local sbin bin stork_q stork_rm stork_status stork_submit etc stork_config stork_config template stork_config sample submit sample submitArgument sample libexec stork transfer file file stork transfer dcache_srm stork transfer castor_srm
11. fer ftp ftp gt stork transfer globus url copy stork transfer gsiftp file gt stork transfer globus url copy stork transfer file gsiftp gt stork transfer globus url copy stork transfer gsiftp ftp gt stork transfer globus url copy stork transfer ftp gsiftp gt stork transfer globus url copy stork transfer gsiftp gsiftp gt stork transfer globus url copy stork transfer http gsiftp gt stork transfer globus url copy stork transfer file petashare gt stork transfer petashare stork transfer petashare file gt stork transfer petashare stork transfer petashare petashare gt stork transfer petashare stork transfer irods file gt stork transfer irods stork transfer file irods gt stork transfer irods stork transfer file srb gt stork transfer srb stork transfer srb file gt stork transfer srb Default values can be edited inside the stork_config file HHHHHHHAHHHHAHHHHHHHEHHEHEHHAHE HEHEHE E HAHAHAHAHAHA RAH HH HE stork_config 4 HHHHHHHHHHHHHHHHHHHHHHHHHHRHEHEAEAE AR AHA EAHA HHH HHH HHA RELEASE_DIR tmp user1 stork HHFHEHHRHREHRERRARRAR AREA REER HR RER ARR AR HERA RRR AREA HERRERA ARERR RHE LOCAL_DIR RELEASE_DIR local HHHHHHHHHH Pathnames LOG LOCAL_DIR log SPOOL LOCAL_DIR spool EXECUTE LOCAL_DIR execute BIN RELEASE_DIR bin LIB RELEASE_DIR lib INCLUDE RELEASE_DIR include SBIN RELEASE_DIR sbin LIBEXEC RELEASE_DIR libexec H
12. ined into a single job STORK_AGGR_MAX_COUNT 0 transfer module specific default values STORK_RECURSIVE_COPY FALSE STORK_VERIFY_CHECKSUM STORK_VERIFY_FILESIZE STORK_NETWORK_CHECK FALSE STORK_TRANSFER_CHECKPOINT STORK_TRANSFER_CHECKPOINT STORK_SYNC_ONLY FALSE STORK_SET_PERMISSIONS FALSE STORK_TEST_MODE 0 FALSE FALSE FALSE FALSE 3 Running Stork If installation is made with root privileges the installation script will try to create a stork user The server will switch to stork user for security purposes The command below starts the stork_server connected to port 10000 The stork logs are named with the prefix Stork such as StorkLog Stork history etc stork_server p lt port gt Serverlog lt stork log directory prefixForLogFiles gt The Stork server generates a log file which is used for logging the activities of the Stork server Below is a sample of the log file cat local log StorkLog 0 23 16755224 0000k kkk kk kk IO IO ak 3k 3k k 3k 3k 3K k K k Kk K k K k k k IC IC ICI ICA I AR AIK 10 23 16 55 24 stork_server STORK STARTING UP 10 23 16 55 24 tmp sbin stork_server 10 23 16 55 24 CondorVersion 6 9 4 Sep 11 2008 10 23 16 55 24 CondorPlatform I386 LINUX_RHEL3 10 23 16 55 24 PID 13697 10 23 16 55 24 Log last touched time unavailable No such file or directory 10 23 16 55 24 090000000000000
13. le SRB URL srb user mdasDomain zone password host port path to file e iRODS URL irods user zone password host port path to file e PetaShare petashare path to file Assuming that a service SRB iRODS GridFTP etc is running Please note that in the URLs shown above the parameters denote e user the username e password password corresponding to the username e host the host on which the service is running e port the port the service listens on e path to file path to the location of the file you would like to transfer e path to directory path to the directory you would like to perform transfers Sample Stork Job Requests i file to file transfer 11 dap_type transfer src_url file path to file dest_url irods user zone password host port path to file dap_type transfer src_url irods user zone password host port path to file dest_url file path to file ii file selection using wild cards dap_type transfer src_url file path to file dest_url irods user zone password host port path to directory dap_type transfer src_url file path to file dest_url irods user zone password host port path to directory dap_type transfer src_url file path to fi le dest_url irods user zone password host port path to directory dap_type transfer src_url file path to dest_url ir
14. ods user zone password host port path to directory dap_type transfer src_url irods user zone password host port path to file dest_url file path to directory dap_type transfer src_url irods user zone password host port path to file dest_url file path to directory dap_type transfer src_url irods user zone password host port path to fi le dest_url file path to directory dap_type transfer src_url irods user zone password host port path to dest_url file path to directory iii recursive transfer from local directory to SRB collection 12 dap_type transfer src_url file path to directory dest_url irods user zone password host port path to directory dap_type transfer src_url irods user zone password host port path to directory dest_url file path to directory Any of the supported protocols may be invoked by simply replacing the URLs shown above by those of the protocols required The new URLs should however conform to their URL format as described in the supported URL format section above 13
Download Pdf Manuals
Related Search