Home

CAIO User`s Manual

1. 44 11 2 E 45 Page 3 49 CAIO User s Manual Agency Europear Date 03 02 2015 Issue 1 Rev 8 Agence spa uropeenne ESA UNCLASSIFIED Releasable to the Public 1 INTRODUCTION Within the ESA s Science Operations Department the SAT in ESAC Spain is responsible of developing and maintaining ESA Scientific Archives For the Cluster Science Archive CSA the standard way to access to this ESA archive is through a powerful J ava interface http csa esac esa int csa csa jnlp To allow access to external applications and expert or normal users that need to retrieve data directly from the archive bypassing the J ava User Interface an integrated Archive Inter Operability system AIO has been developed for CSA This AIO system can be accessed using standard HTTP requests This subsystem shares the core system of the standard CSA User Interface It accepts three kinds of requests e Metadata request To be used when we want to retrieve Inventory information lists of Datasets or Files that fulfil a certain criteria or the metadata details of dataset s and file s e Data request To be used when we want to retrieve files of the data observed or dataset metadata header files e Asynchronous Data request To be used when we want to retrieve asynchronously files of the data observed e
2. 5 DATE 2002 02 12T23 30 00Z END DATE 2002 02 13T01 00 00Z O fgmDir Retrieve from all EDI Electro Gyrotime datasets and from all RAPID 3D distribution of Electrons datasets the data between the time range 2001 02 01T00 00 00Z 2001 02 01T12 00 00Z and leave the files at the clusterFiles directory java cp AioWEBClient S csa esac esa int csa L ALL DATASET DATASET ID 96 CP EDI EGD ORDATASET DATASET ID RAP_ START 2001 02 01 00 00 007 END DATE 2001 02 01T12 00 00Z O clusterFiles Asynchronous Data retrieval from multiple datasets Retrieve asynchronously all EDI Electro Gyrotime and all RAPID 3D distribution of Electrons data between the time range 2001 02 01T00 00 00Z 2001 02 01T12 00 00Z and leave the files at the asyncClusterFiles directory java cp savot jar AioWEBClient S csa esac esa int csa L ASYNC DATASET DATASET ID 96 CP EDI EGD ORDATASET DATASET START DATE 2001 02 01100 00 00Z END DATE 2001 02 01T12 00 00Z O asyncClusterFiles The response to every asynchronous request will generate a text file containing information about the Request ID and the link where the data will be ready when finished All those text files will be stored in the folder asyncClusterFiles Page 35 49 CAIO User s Manual European Space Agency Date 03 02 2015
3. csa aio product action USERNAME username amp PASSWORD password2 amp DATASET ID C3 CP ASP IONC amp START DATE 20 04 06 18T11 35 00Z amp END DATE 2004 06 19T18 35 00Z amp NON BROWSER Authentication cookie wget load cookies cookies file gt txt content disposition http csa esac esa int csa aio product action DATASET ID2C3 CP ASP IONC amp START DATE 2004 06 18T11 35 00Z amp END 2004 06 19T18 35 00Z amp NON BROWSER stores the retrieved data at a tar gzip file with a random name for example f73edec3 Ocd9 4cd7 a203 bd413de5fd9e tar gz If a user cannot be authenticated wget will output an authentication log n file instead 10 1 2 Asynchronous download See list of parameters at section 5 1 Asynchronous download is necessary in particular for data request greater than 100MB compressed i e between 200 and 900 MB once files are uncompressed There are two ways of executing the second example presented in the Async Data tab Credentials within the request Simply replace lt username gt and lt password gt by your CSA username and password in the example below wget content disposition http csa esac esa int csa aio async product action USERNAME lt username gt amp PASSWORD lt password gt amp DATASET ID C3 ASP IONC amp START DATE 20 04 06 18T11 35 OOZ amp END_DATE 2004 06 19T18 35 00Z amp NON_BROWSER Page 39 49 CAIO User s Manual
4. Edit View History Bookmarks Tools Help caro cluster Final archive Inter Operabilt cfaint esac esa int cfa sio product action DATASET 10 3 5 IONCSSTART DATE 2004 06 18T11 35 0028END 200 cluster Archive Inter pe Opening bmartine2094 tar gz You have chosen to open L Download re Do this automatically for fles like this from now on lose this page a _ 5 AAS Q f FOLLOW ESA SCIENCE 0 COPYRIGHT 2000 2013 EUROPEAN SPACE AGENCY ALL RIGHTS RESERVED TERMS AND CONDITIONS Figure 3 HTTP Product Request e Use of user credentials for the previous request Simply replace lt username gt and lt password gt by your CSA username and password in the example below http csa esac esa int aio product action2USERNAME username amp PASSWORD password2 amp DATASET ID C3 _CP_ASP_ION amp START_DATE 2004 06 18T11 35 00Z amp END_DATE 2004 06 19T18 35 00Z e Request DOUBLE STAR FGM products observed at D1 with dataset identifier D1 FGM SPIN for time range 2004 01 08 04 00 00 2004 01 08 06 00 00 17 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public http csa esac esa int csa aio product action DATASET ID D1 CP SPIN amp START DATE 2004 01 08T04 00 00Z amp END DATE 2004 01
5. Example To download full resolution FGM data from Cl and C2 22 50 23 50 UT 7 August 2004 including caveats in cdf format credentials csa_login username password filename csa_product credentials Cl1_CP_FGM_FULL C2_CP_FGM_FULL 2004 08 07T22 50 002 2004 08 07T23 50 002Z2 delivery format CDF REF DOC 1 Andrew Walsh 2013 05 30 awalsh AT rssd DOT esa DOT int Page 47 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public define URL host and path csa_host csa esac esa int csa product path csa aio product action construct URL query from supplied parameters and keywords csa product query for i 0 n elements csa dataset id 1 do begin csa product query csa product query DATASET ID csa dataset 1 amp endfor csa product query csa product query amp START start DATE csa end amp BROWSER if keyword_set csa_delivery_format then csa_product_query csa_product_query amp DELIVERY_FORMAT csa_delivery_format if keyword_set csa_delivery_interval then csa_product_query csa_product_query amp DELIVERY_INTERVAL csa_delivery_interval if keyword set csa ref doc then csa_product_query csa_product_query amp REF_DOC csa_ref_doc if keyword set csa ingested since then csa product
6. CP EDI OR DATASET DATASET ID like Equivalent URL http csa esac esa int csa aio metadata action SELECTED FIELDS DATASET DATASET ID DATASET START DATED ATASET END DATE DATASET TITLE DATASET DESCRIPTION amp RESOURCE LASS DATASET amp QUERY DATASET DATASET ID9620like9620 C9625 CP EDI EGD 96200R9620DATASET DATASET ID9620like9620 9625RAP 9625 amp RETURN _ TYPE CSV List File name Start Date End Date and Ingestion Date in CSV format of files that fulfil the criteria QUERY DATASET DATASET ID like CP EDI DATASET DATASET ID EFW L3 OR DATASET DATASET ID like FULL AND FILE START DATE lt 2001 02 05 AND FILE END DATE gt 2001 02 01 Equivalent URL http csa esac esa int csa aio metadata action SELECTED FIELDS FILE FILE NAME FILE START DATE FILE END D ATE ILE CAA INGESTION DATE amp RESOURCE CLASS FILE amp QUERY DATA SET DATASET ID9620like9620 C1 CP EDI EGD 96200R9620DATASET DATASE T ID9G20like9620 C1 EFW L3 P 20OR 20DATASET DATASET_ID 20lik 20 1 FULL 9620AND9620FILE START 20 lt 20 2001 02 05 20AND 20FILE END_DATE 20 gt 20 2001 02 01 SPAGE_SIZE 1000 amp PAGE 1 amp RETURN_TYPE CSV Page 12 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 3 INVENTORY RE
7. 38 10 11 Direct 39 10 1 2 Asynchronous download 39 101 3 Data SIWaming u L aaa 40 10 2 Inventory request ekini ineine E NES 41 10 3 Metadata 41 10 4 Howto specify the output 41 10 5 Howto create an authentication 42 10 6 Hint on descriptive error MESSAGES ssccsseseeseseeseseeseseeeessesesceceseesesecsesecseeceseecesseaeeecsaeseeseseeseseeaesecaeeeeseeeeaeeeeseeaeeeeaeeeed 43 11 APPENDIX HOW TO LINK WITH MATLAB IDL 44 11 C Matlab C D 44 1 11 For users with Matlab 20 13a or later releases 44 1 12 For users with Matlab 2012b or
8. 20AND 20DATASET INVENTORY END TIME 20 gt 20 2008 02 01T00 00 00Z amp RETURN TYPE J SON Page 7 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 2 METADATA REQUESTS Metadata requests are used to retrieve the metadata details of one or many cluster datasets or cluster files They are requests that the user will have to build using several parameters Once this query is sent the CSA backend will return the metadata that matches the given criteria in the requested format It is important to highlight that the default return type for all metadata requests is the VOTable format 2 1 Metadata Request Parameters All metadata requests have the following HTTP prefix http csa esac esa int csa aio metadata action After the HTTP prefix the following parameters are used to build the request Parameter Description Mandatory Possible Values Name s SELECTED FIELDS Single Entity Entity or list of Any CSA Entity or any Attributes requested Entity combination of CSA Attribute list separated by attributes See List of CSA commas entities and attributes RETURN TYPE Output format of the response The VOTABLE default CSV default response formal is VOTable JSON Other formats are J SON or CSV RESOURCE CLASS Single Entity which defines the Any CSA Entity See List of granularity level of the metadat
9. 00 00Z amp DELIVERY INTERVAL hourly6 amp NON BROWSER 112 UC2 General query to be executed asynchronously This use case retrieves 3 CEF products for 4 days split into day files The request will be executed asynchronously and will send an email when complete Type of request Asynchronous data request Example Simply replace username and lt pwd gt by your CSA username and password in the example below http csa esac esa int csa aio async product action USERNAME username amp PASSWORD pwd amp DATASET ID C1 CP EDI EGD amp DATASET I D C1 CP EFW L3 P amp DATASET ID C1 CP FGM FULL amp START DATE 2001 02 01100 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Daily amp NON BROWSER 113 UC3 List files This use case retrieves the list of files that match a search criteria The response will be in CSV format Note that the value of QUERY parameter should be URL encoded Type of request Metadata request Example http csa esac esa int csa aio metadata action SELECTED FIELDS FILE FILE NAME FILE START DATE FILE END DATE FILE CAA INGES DATE amp RESOURCE _ CLASS FILE amp QUERY DATASET DATASET ID 201like 20 EDI EG D 200R 20DATASET DATASET ID 20like 20 C1 EFW L3 P 200R 20DATASET DATASET I D9620like9620 C1 CP FULL 20AND 20FILE START DATE 20 lt 20 2001 02 05 20AND 20FILE END_ DATE 20 gt 20 2001 02 0 SIZE 1000 amp PAGE 1 am
10. 1 FEEN DATE FECSA INGESIION DATE 1 INGESTION DATE HLEEXTENMION o PRHLEACHVE INTEGER R sain 7 DATASET INVENTORY DATASET INVENTORY LEE 3 P DATASET INVENTORY DATASET STRING P DATASET INVENTORY START TIME DATE P DATASET INVENTORY END TIME OT DATE P DATASET INVENTORY NUM INSTANCES INTEGER _ DATASET INVENTORY VERSION CSL INTEGER Page 32 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 9 APPENDIX HOW TO RUN THE DEFAULT CLIENT We have implemented a code example using the J ava programming language to be used as a simple AIO client Download it and extract the content of the tar file on your local disk Client example http csa esac esa int csa aio aioclient 1 0 tar You will find both the source and the compiled code of our example We have implemented three different actions retrieval from a single dataset retrieval from multiple datasets and asynchronous retrieval from multiple datasets Forthe single dataset option the unique identifier of the dataset i e its dataset id is required and optionally other restrictions can be added see Table 4 Product request parameters This option simply makes and http request to the CSA and retrieve the file s he mult
11. TD gt http cfaint esac esa int cfaaio product action DATASET ID C1 CQ ASP CAVEATS lt TD gt lt TD gt http cfaint esac esa int cfa aio metadata action DATASET DATASET_ID C1_CQ_ASP_CAVEATS amp RESOURCE_CLASS DATASET amp SELECTED_FIELDS DATASET amp RETURN_TYPE VOTABLE lt TD gt lt TR gt lt gt lt gt 2 ASP lt gt lt TD gt 2001 01 01 00 00 00 0 lt TD gt El lt TD gt 2008 07 01 01 00 00 0 lt TD gt Figure 2 Returned VOTable file When requesting Datasets RESOURCE_CLASS DATASET in VOTABLE or JSON format additional information will be provided for each dataset e Data Link to retrieve the files for this dataset An Asynchronous Data Link to asynchronously retrieve the files for this dataset A Details Link to retrieve more precise information about this dataset 2 2 User of QUERY parameter In the metadata request the Keyword value paradigm creates simple queries using the AND argument as logical operator If a user wants to create a complex query by using gt Page 10 49 User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public lt gt lt like including SQL wildcards ana or operators he she will have to use QUERY parameter The rest of the keyword parameters will be seen as other AND conditions The value of this QUERY field should be UR
12. a browser Simply replace lt username gt and lt password gt by your CSA username and password in the example below http csa esac esa int csa aio login action USERNAME username amp PASSWORD password2 amp SUBMIT LOGIN Boo utar Pra rim omne 3 em cluster Archive Inter Operability subsystem AIO BMARTINE has successfully logged in 22 f FOLLOW ESA scence 7 COPYRIGHT 2000 2112 EUROPEAN SPACE AGENCY ALL RIGHTS RESERVED TERMS AND CONDITICNS Figure 7 Successful login Page 28 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 22 h F esa 7 2 Logout requests When a user logs out an anonymous user is set by default After the HTTP prefix the following parameters are used to build the login request Parameter Name s Description Mandatory Possible Values SUBMIT LOGOUT as the request is for a LOGIN or LOGOUT logout NON_BROWSER Whether the request is performed NOT from a Browser If not specified the requests is done from a browser and proper HTML pages wil be displayed Example e Logout of a user with no browser http csa esac esa int csa aio login action 9620SUBMIT LOGOUT amp NON BROWSER IIO Ele Edit History Bookmarks
13. future requests with wget 10 6 Hint on descriptive error messages When wget gets a response with an error status it just displays the error code with the standard error message In some cases the error response contents some text explaining the reason of the failure This text can be extracted with curl using the syntax below curl b cookies file txt R lt url of failing request gt For example a request with an incorrect date format like curl b lt cookies_file gt txt http csa esac esa int csa aio streaming action DATASET 4 CP PEA LERL PSD amp START DATE 2004 09 18T10 00 00Z amp END DATE 2004 10 2022 00800Z amp NON BROWSER returns the following response ERROR 400 The string 2004 10 2022 00 00Z does not match an ISO Date format Page 43 49 CAIO User s Manual European Space Date 03 02 2015 Issue 1 Rev 8 Agence spatiale e ESA UNCLASSIFIED Releasable to the Public APPENDIX HOW TO LINK CAIO WITH MATLAB OR IDL In this appendix some recommendations for users that want to access the Cluster Science Archive directly from Matlab and or IDL are given 11 Matlab Please note that the Authentication option in urlwrite m is ONLY available since Matlab release 2013a You can check your matlab version by typing ver at the Matlab prompt Hence two examples are proposed here 11 11 For users with Matlab 2013a or later releases Simply replac
14. in All Loo TenMin time By default Daily Loo hourly3 iae hourly12 When this is set it delivers the files NOT Oor1 corresponding to the dataset requested AND the files of the available datasets that are referenced in the metadata of the requested dataset Referenced documents are typically CAVEAT files If the reference documents are CEF files only the requested interval is delivered if they are non CEF files the file s containing the requested interval will be delivered By default 0 INGESTED_ SINCE Only download files that have been NOT ISO time format ingested since a particular date ddTHH mm ssZ The size of the maximum amount of NOT A positive integer uncompressed data that can representing the downloaded in Kbytes If the query maximum file size in results in a delivery exceeds the Kbytes specified filesize the user will receive an error message NO NOTIFY Indicates that the user does not want NOT to be notified by email when the data is ready for download If not specified an email will be sent to the user NON BROWSER Indicates that the request is not NOT performed from a Browser If not specified it means that the requests are done from a browser and proper HTML pages will be displayed in order to present information to the user Table 6 Asynchronous Product request parameters Although we strongly recommend the use of authentication session cookies it is also pos
15. load the cookies file recommended For each type of data request direct asynchronous or data streaming two ways of executing the same data request will be detailed For CAA users upgrading their scripts please note that the CAA uname keyword is simply USERNAME on CSA while pwd becomes PASSWORD Any data retrieval request has as output a file See How to specify the output file section 10 4 to get a meaningful filename Bear in mind that content disposition is not yet bug free and sometimes the requests can store the output in a filename with the webpage name and query string parameters default behaviour of wget For example Page 38 49 User s Manual European Space A Date 03 02 2015 Issue 1 Rev 8 Agence spatiale ESA UNCLASSIFIED Releasable to the Public Simply replace lt username gt and lt password gt by your CSA username and password in the example below async product action USERNAME username amp PASSWORD password2 amp DATASET ID C3 CP ASP IONC amp START DATE 2004 06 18T11 35 00Z amp END DATE 2004 06 19T18 35 00Z amp NON BROWSER 10 11 Direct download See list of parameters at section 4 1 Please find below two ways of executing the first example presented in the Data tab Credentials within the request Simply replace username and password by your CSA username and password in the example below wget content disposition http
16. query csa product query amp INGESTED SINCE csa ingested since if keyword set csa max file size then csa product query csa product query amp MAX FILE 517 max file size construct crendentials string for http header csa cookie Cookie creds Create IDLnetURL object and set properties csa product obj obj new IDLnetUrl csa product obj SetProperty VERBOSE 1 csa product obj SetProperty url scheme http csa product obj SetProperty url host csa host csa product obj SetProperty url path csa product path csa product obj SetProperty url query csa product query csa product obj SetProperty headers csa cookie send request to CSA AIO system saving response in csa buffer dat print Getting data from CSA please wait csa product response csa_product_obj gt get filename csa_buffer dat csa product obj getproperty response_header csa_product_header check a tar gz file was downloaded and if so rename buffer to correct filename and return correct filename otherwise return 0 csa filestart strpos csa product header filename if csa filestart ne 1 then begin csa fileend strpos csa product header gz csa filename strmid csa product header csa filestart 10 csa fileend csa filestart 8 csa dir end strpos csa product response csa buffer dat csa working dir strmid csa product response 0 csa dir end file move csa product response csa work
17. 00Z http csa esac esa int csa aio product acion DATASET ID C3 CP RAP L3DD R amp START DATE 2001 02 01T00 00 00Z amp END_ DATE 2001 02 01T12 00 00Z http csa esac esa int csa aio product action DATASET 4 CP RAP E3DD R amp START DATE 2001 02 01100 00 00Z amp END DATE 2001 02 01T12 00 00Z this sequence of actions is equivalent to Page 36 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public java cp AioWEBClient S csa esac esa int csa L GET ALL DATASET DATASET ID 96 CP EDI EGD ORDATASET DATASET ID RAP_ START DATE 2001 02 01T00 00 00Z END DATE 2001 02 01T12 00 00Z O clusterFiles All retrieved files will be stored in the folder clusterFiles Asynchronous Data retrieval from multiple datasets In the same way as for a data retrieval it is a sequence of actions First a Metadata request to search for the datasets that fulfil the criteria afterwards an Asynchronous Data Product request per dataset found to retrieve its files Look for datasets that contains files matching this particular criterion http csa esac esa int csa aio metadata action RESOURCE CLASS DATASET amp SELECTED FIELDS DATASET DATASET ID amp QUERY DATASET DATASET ID Hike49627C9625 EDI EGD 27 OR DATASET D ATASET IDHike 27 25RAP_ 25DD_R 27 GRETURN_TYPE VOTAB
18. 08T06 00 00Z Request DWP products observed at Cluster 1 with dataset identifier 1 CORR SI for time range 200 06 10 22 12 14 2001 06 11 06 12 14 in CDF format and file interval 3 hours http csa esac esa int csa aio product action DATASET ID C1 CP CORR ST amp START DATE 2001 06 10T22 12 14Z amp END 2001 06 11T06 12 14Z amp DELIVERY FORMAT CDF amp DELIVERY INTERVAL hourly3 Request non CEF products from WBD observed at Cluster 4 with dataset identifier CG WBD GIFPLOT for time range 2008 01 30 05 55 00 2008 01 30 05 57 00 http csa esac esa int csa aio product ID C4 CG WBD GIFPLOT amp START DATE 2008 01 30T05 55 00Z amp END_DATE 2008 01 30T05 57 00Z Request RAPID products observed at Cluster 3 with dataset identifier CP RAP and its caveats for time range 2003 04 01 00 00 00 2003 04 02 00 00 00 http csa esac esa int csa aio product action DATASET ID C3 CP RAP DEG amp START DATE 2003 04 01100 00 00Z amp END DATE 2003 04 02T00 00 00Z amp REF DOC 1 Request EDI products observed at Cluster 1 with dataset identifier 1 CP EDI for time range 2001 02 01 00 00 00 2001 02 05 00 00 00 that have been ingested since 2010 03 10T10 21 00Z http csa esac esa int csa aio product action DATASET ID C1 CP EDI EGD amp START DATE 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVER
19. 15 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 10 APPENDIX HOW TO USE CAIO WITH WGET Below some instructions on how to use wget when interacting with the CSA Instructions for different kind of requests are explained including two ways for user authentication data download Although username and password can be used directly on the http request for authentication we strongly recommend the use of cookies that provide a higher level of security Recommendations For a smooth execution add the flag BROWSER to all CAIO requests when using wget Use the wget option content disposition When making requests using otherwise the output filename will be valid but meaningless in terms of syntax If your request contains special chars e g password with exclamation mark use bash shell and add single quote 7 instead of the usual double quotes around the http request Go to last section of this chapter to know how to get more descriptive error messages These are the sections within this appendix Datadownload o Direct download o Asynchronous download Data streaming Inventory request Metadata request How to specify the output file Howto create a cookie Hint on descriptive error messages 10 1 Data download Data retrieval requires an authenticated user Thus it is necessary either to provide user credentials in the request or to
20. ASET ID Unique identifier of the dataset YES Any of the Cluster whose data we want to retrieve It can datasets appear more than once and can contain wild cards Page 15 49 User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Cesa RETRIEVALTYPE Type of retrieval By default NOT PRODUCT or PRODUCT cluster data file HEADER START_DATE Start time of the time range that the NOT but ISO time format retrieved files should cover By recommended yyyy MM default Cluster Mission start time ddTHH mm ssZ END_DATE End time of the time range that the NOT but ISO time format IN default Cluster Mission end time ddTHH mm ssZ p ee CEF files By default CEF DELIVERY INTERVAL Specifies the delivery file interval in 2 2 TenMin time By default Daily Hourly hourly3 hourly6 hourly12 When this is set it delivers the files NOT Oor1 corresponding to the dataset requested AND the files of the available datasets that are referenced in the metadata of the requested dataset Referenced documents are typically CAVEAT files If the reference documents are CEF files only the requested interval is delivered if they are non CEF files the file s containing the requested will be delivered By default INGESTED SINCE Only download files that have been NOT eh M format ingested since a particular
21. ATASET MAX_WIDTH P QUICK LOOK DATASET MAX HEIGHT INTEGER P QUICK LOOK DATASET DATASET OID NTEGER QUICK LOOK DATASET DATASET ID NG QUICK LOOK DATASET START DATE P QUICK LOOK DATASET END DATE P QUICK LOOK DATASET DESCRIPTION NG P QUICK LOOK DATASET DATASET VERSION P QUICK_LOOK_DATASET RELEASED P QUICK LOOK DATASET ACTIVE NTEGER Po QUICK LOOK DATASET INVENTORY NTEGER P QUICK LOOK DATASET DISPLAY ORDER QUICK LOOK DATASET FILE EXTENSION F s o QU o QU QU o QU 221 o QU o QU S 2 o QU DATA TYPE eee eae eee eee QUICK LOOK DATASET TITLE ICK LOOK DATASET TIME RESOLUTION OUBLE QUICK LOOK DATASET MIN TIME RESOLUTION g LOOK DATASET MAX TIME RESOLUTION g LOOK DATASET DATASET CAVEATS QUICK LOOK DATASET GUI NAME LOOK DATASET MAIN GROUP I G G 445 DATA NTEGER Loro 1 DATA TYPESHORT NAME LONG 4855 G G Er C Er 2129 Page 31 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public FILE IDC STRING PoC FILE VERSION NUMBER INTEGER VERSION CS STRING FILE METADATA VERSION CL STRING STHNG EHE TYE STHNG HBLESTART DATE CCAS
22. DATASET INVENTORY amp RESOURCE CLASS DATASET INVENTORY amp QUERY DATA SET DATASET_ID 20like 20 CM_CP_DWP_UT_PIOR 20AND 20DATASET_INVENTORY START_TIME 20 lt 20 2013 03 02T02 00 00Z 20AND 20DATASET_INVENTORY END_ TIME 20 gt 20 2013 03 01T00 00 00Z amp RETURN TYPE CSV amp NON BROWSER stores the result at a file named CSA Results csv n where n is a number that increments in order to not overwrite the file 10 3 Metadata request See list of parameters at section 2 1 No authentication needed please find below how to execute the last example presented in the Metadata tab wget content disposition http csa esac esa int csa aio metadata action SELECTED FIELDS FILE FILE NAME FILE START DATE FILE END DATE FILE CAA INGESTION DATE amp RESOURCE CLASS FILE amp QUERY DATASET DATASET ID 201ike 20 C1 CP EDI EGD 96200R9620DATASET DATASET_ID 20like 20 C1_CP_EFW_L3 9020 2020 10 2011 20 1 CP FULL 20AND 20FILE START_DATE 20 lt 20 2001 02 05 20AND 20FILE END_DATE 20 gt 20 2001 02 O1 G amp PAGE_SIZE 1000 amp PAGE 1GRETURN_TYPE CSV amp NON_BROWSER stores the result at a file named CSA Results csv n where n is a number that increments in order to not overwrite the file 10 4 Howto specify the output file Be aware of the following indication at the wget manual content disposition If this is set to on experimental not fully functional suppo
23. Date 03 02 2015 Issue 1 Rev 8 ESA UNCLASSIFIED Releasable to the Public Authentication cookie wget load cookies lt cookies_file gt txt content disposition http csa esac esa int csa 1 async product ID2C3 CP ASP IONC amp START_DATE 2004 06 18T11 35 00Z amp END_DATE 2004 06 19T18 35 00Z amp NON BROWSER stores the request information at a txt file with a random name for example 8cf8ae7f 7ab7 4af5 8557 11ff6498b4c8 txt Sometimes it can store the output in a filename with the following syntax of product action USERNAME lt username gt S amp PASSWORD lt password gt SDATASET_ID C3_CP_A SP IONC amp START DATE 2004 06 18T11 35 00Z amp END DATE 2004 06 19118 35 00Z amp NON BROWSER Example of content of the file Your request has been sent Your request ID is bmartine1060 The file will be available at http csa esac esa int ftp public bmartine1060 bmartine1060 tar gz You will be notified by email when the data is ready for download If a user cannot be authenticated wget will output an authentication log n file instead 10 1 3 Data streaming See list of parameters at section 6 1 Please find below two ways of executing the last example presented in the Streaming tab Credentials within the request Simply replace username and password by your CSA username and password in the example below wg
24. ESA UNCLASSIFIED Releasable to the Public DOCUMENT Y N S C esac European Space Astronomy Centre 0 Box 78 28691 Villanueva de la Ca ada Madrid Spain T 34 91 8131 100 F 34 91 8131 139 www esa intO CSA Archive Inter Operability AIO System User s Manual Prepared by Reference Issue Revision Date of Issue Status Document Type Distribution Science Archives Team 1 8 03 02 2015 Under review European Space Agency Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Title CSA Archive Inter Operability AIO System User s Manual Issue 1 Revision 8 Author Science Archives Team Date 03 02 2015 Approved by Date Head of the ESAC Science Archives Team Reason for change Issue Revision Date Add information about Inventory Requests 1 1 21 06 2013 Added wild card option to product requests and 1 2 04 07 2013 asynchronous product requests Added missing information 1 3 26 07 2013 Archive renamed from Cluster Final Archive to Cluster 1 4 25 09 2013 Science Archive CSA Correct typos at product request and replace cfa 1 5 31 10 2013 instance at URLs by csa Included new functionality data streaming 1 6 17 01 2014 Improve description of data download specially with 1 7 15 09 2014 WGET Include Double Star example and updated IDL login 1 8 03 02 2015 routine Issue 1 Revision 8 Reason for change Paragraph s Updated IDL login routine to deal with special cha
25. Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 9 3 Equivalence to data requests The equivalent command line syntax to the data product request http csa esac esa int csa aio product action DATASET_ID C3 CP ASP IONC amp START DATE 2004 06 18111 35 OOZ amp END _ DATE 2004 06 19T18 35 00Z is java AioWEBClient S csa esac esa int csa L GET DATASET ID C3 CP ASP IONC P START DATE 2004 06 18T11 35 00Z DATE 2004 06 19T18 35 00Z Data retrieval from multiple datasets A data retrieval from multiple datasets consists on a sequence of actions First a metadata request to search for the datasets that fulfil the criteria afterwards a Data Product request per dataset found to retrieve its files Look for the datasets that contains files matching this particular criterion http csa esac esa int csa aio metadata action RESOURCE CLASS DATASET amp SELECTED FIELDS DATASET DATASET ID amp x QUERY DATASET DATASET_IDHike 27C 25_CP_EDI_EGD 27 OR DATASET D ATASET _ID Hike 27 25RAP_ 25DD_R 27 amp RETURN_TYPE VOTABLE amp PAGE 1 amp PAGE SIZE 1000 For each dataset returned in VOTABLE format retrieve the data files for the specific time range if specified http csa esac esa int csa aio product action DATASET ID C1 EDI EGD amp START DATE 2001 02 01T00 00 00Z amp END DATE 2001 02 01T12 00
26. J INTEGER _ 1 DATE DATE OL DATE 22 INSTRUMENLNAME SIRHNG 2 INSTRUMENLDESCHIPTION SCS STRING INSTRUMENTACIVE INTEGER o INSTRUMENT _ STHNG MEASUREMENT TYPETYEE o SIHNG DATASET DATASETID CS STRING DATASET START_ DATE 1 DATASETEND DATE 1 DATASET DESCRIPTION SIHNG VERSIONS STRING 2 DATASELRELEASED 7 INTEGER 2 DATASETACHVE INTEGER 2 DATASETINVENIORY INTEGER DATASET DISPLAY ODER INIEGER DATASELHLE EXTENSION CS STRING Page 30 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public E _ Ld DATASET MAX TIME RESOLUTION DOUBLE CAVEATS 2 DATASETGU GROUP DATASELCATECORY DATASETINGESTION_DATE DATASET UNIT NAME EP pO QUICK LOOK DATASET DATASET MULTI_SC I INTEGER NG NTEGER QUICK LOOK DATASET P QUICK LOOK DATASET QL DISPLAY ORDER INTEGER P QUICK LOOK DATASET WIDTH INTEGER LU _ QUICK LOOK DATASET HEIGHT INTEGER P QUICK_LOOK_D
27. L encoded Inside the value of the QUERY parameter we allow any J ython valid query expression which makes use of CSA attributes see List of CSA entities and Attributes J ython query operations are slightly different from the SOL ones as stated in this table QUERY Operators Type Precedence Symbol PARENTHESIS Grouping 1 0 EQUALS jReeiond 2 p LOWEROREQUALSTHAN Relationd 2 lt 02 LIKE Relational 2 LIKE gt MULTIPLE CHARACTERS gt ONE CHARACTER OR 3 OR Table 2 QUERY parameter operators Let s see the use of all the above through some examples First 100 datasets that fulfil the criteria QUERY DATASET DATASET ID like CP EDI OR DATASET DATASET ID like Equivalent URL http csa esac esa int csa 1 metadata action SELECTED FIELDS DATASET amp RESOURCE CLASS DATASET amp QUERY DATASET DATASET ID9620like9620 C9625 EGD 96200R9620DATASE T DATASET ID9620like9620 9625RAP 9625 amp PAGE SIZE 100 amp PAGE 1 e Dataset ID Start Date End Date Title and Description of Datasets that fulfil the criteria 2 URL encoding reference http www w3schools com tags ref urlencode asp 3 http www jython org Page 11 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public QUERY DATASET DATASET ID like
28. LEG amp PAGE 1 amp PAGE SIZE 1000 For each dataset returned in VOTABLE format request asynchronously the data files for the specific time range if specified http csa esac esa int csa aio async product ID C1 CP EDI EGD amp START DATE 2001 02 01T00 00 00Z amp END_ DATE 2001 02 01T12 00 00Z http csa esac esa in aio c product acion DATASET ID C3 CP RAP 1300 R amp START DATE 2001 02 01T00 00 00Z amp END_ DATE 2001 02 01T12 00 00Z http csa esac esa int csa aio async product acion DATASET 4 CP RAP E3DD R amp START DATE 2001 02 01T00 00 00Z amp END_ DATE 2001 02 01T12 00 00Z this sequence of actions is equivalent to java cp AioWEBClient S csa esac esa csa L ASYNC DATASET DATASET ID CP EDI EGD ORDATASET DATASET ID RAP P START DATE 2001 02 01T00 00 00Z END DATE 2001 02 01T12 00 002Z O asyncClusterFiles The response to every asynchronous request will generate a text file containing information about the Request ID and the link where the data will be ready when finished All those text files will be stored in the folder asyncClusterFiles Please feel free to take this code example and modify it to suit your application needs This is aJ ava example but any other language could be used for the same purpose Page 37 49 CAIO User s Manual European Space Agency Date 03 02 20
29. ODIF RPA Use of user credentials in the previous request Simply replace username and password by your CSA username and password in the example below http csa esac esa int csa aio product action2USERNAME username amp PASSWORD password amp RETRIEVALTYPE HEADERGDATASET ID C CIS CODIF RPA e Request dataset metadata header files for EDI and RAPID Page 20 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public http csa esac esa int csa aio product action RETRIEVALTYPE HEADERGDATASET ID C CP EDI EGD amp DATASE T ID RAP 5 ASYNCHRONOUS DATA REQUESTS Asynchronous data requests are used to retrieve files asynchronously These files are files that belong to a given dataset and that cover a given timespan They are HTTP requests that the user will have to build using several parameters Once this request is sent the CSA backend will return a Request ID and a link were the data will be available once the retrieval has finished All the asynchronous data requests have the following HTTP prefix http csa esac esa int csa aio async product action Only one type of data requests is available in asynchronous mode e Asynchronous Product requests Please note that for asynchronous data requests Itis mandatory that the user is logged in see Chapter HTTP prefix and para
30. QUESTS Inventory requests are a special case of Metadata Requests This section explains how to use them to retrieve inventory information The default return type is the VOTable format but it can return J SON and CSV as well The criteria for Inventory Requests are Dataset Id s Start time and Stop time Below are outlined the parameters to create a basic inventory request Parameter Name s Values SELECTED FIELDS DATASET INVENTORY RESOURCE CLASS DATASET INVENTORY Complex restriction using e DATASET INVENTORY DATASET ID Unique identifier of the dataset It can appear more than once if so include them between parenthesis and separated with OR operator It can contain wild cards using together LIKE operator and DATASET INVENTORY START TIME Set it to lower or equals than the end ofthe time interval you are searching for DATASET INVENTORY END TIME Set it to greater or equals than the beginning ofthe time interval you are searching for RETURN TYPE optional VOTABLE default CSV J SON NON_BROWSER No value needed When specified indicates that the request is not performed optional from a Browser When not specified requests done from a browser proper HTML pages will be displayed in order to present information to the user Table 3 Inventory request parameters Note that the value of the QUERY parameter must be URL Examples of retrieving inventory info e Dataset s DWP
31. Streaming Data request To be use when we want to retrieve files of the data observed via the streaming functionality Please note that for a synchronous data retrieval there is a limit of 1GB while for an asynchronous data retrieval the limit is 50GB Ideally this interface should be used by a script application code or command line tools First we retrieve a list of datasets through a metadata request and then we issue a data request for each one of them in order to retrieve the file s Examples of using interface from ava client wget Matlab or IDL can be found at the appendix section Meanwhile at section 1 1 some use cases of CAIO pointing to its corresponding type of request are presented 11 Use Cases 111 General query to be executed immediately This use case retrieves 2 CEF products for 1 day split into 6 hour files and the request will be executed immediately Type of request Data product request Page 4 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Example Simply replace lt username gt and lt pwd gt by your CSA username and password in the example below http csa esac esa int csa aio product action USERNAME lt username gt amp PASSWORD lt pwd gt SDATASET_ ID C3 ASP IONCSDATASET I D C1 CP EFW L1 P2 amp START DATE 2004 06 18T00 00 00Z amp END DATE 2004 06 19T00
32. Tools http cfaint esac LOGOUT amp NON BROWSER gt cfaint esac esa int cfa aio login action SUBMIT LOGOUTSNON_BROWSER Google fr 40 anonymous has successfully logged in Figure 8 Successful logout Page 29 49 User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 8 LIST OF CSA ENTITIES AND ATTRIBUTES Below you can find the list of all the CSA entities and attributes They can be used in the metadata request related to certain parameters Entity name can appear as SELECTED FIELDS only one at a time Entity name can appear as RESOURCE CLASS only one at a time Attribute names can appear as SELECTED FIELDS one or a combination of them Attribute names can appear as ORDER BY one or a combination of them Simple attribute restrictions can be added as Attribute Name value Entity Name Attribute Name Attribute Type 5 STRING 1 SPAN PoC OBSERVATORY ABBREVIATION OL STRING PoC OBSERVATORY NAME SIHNG OBSERVATORY DESCRIPTION CST STRING SPAN JSIHNG O _ NAME o EXPERIMENT DESCRIPTION SCS STRING 2 o
33. UT Time stamped PIOR Time interval 2013 03 01T00 00 00Z 2013 03 02T02 00 00Z Format CSV http csa esac esa int csa aio metadata action SELECTED FIELDS DATASET INVENTORY amp RESOURCE CLASS DAT ASET INVENTORYGQUERY DATASET 4 URL encoding reference http www w3schools com tags ref urlencode asp Page 13 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public _INVENTORY DATASET _ID 20like 20 CM_CP_DWP_UT_PIOR 20AND 20 DATASET INVENTORY START TIME 20 lt 20 20 13 03 02T02 00 00Z 20AND 20DATASET INVENTORY END TIME 20 gt 20 201 3 03 01T00 00 00Z amp RETURN_TYPE CSV Dataset s CP FULL and C1 FGM _ SPIN Time interval 2008 02 01T00 00 00Z 2008 02 04T00 00 00Z Format JSON http csa esac esa int csa 1 metadata action SELECTED FIELDS DATASET INVENTORY amp RESOURCE CLASS DAT ASET INVENTORY amp QUERY DATASET INVENTORY DATASET ID 20like 2 CP 1 200 20 INVENTORY DATASET ID9620like 20 C1_CP_FGM_SPIN 20AND 20DATASET_INVENTORY START_TIME 2 0 lt 20 2008 02 04T00 00 00Z 20AND 20DATASET_INVENTORY END_TIME 20 gt 20 200 8 02 01T00 00 00Z amp RETURN_TYPE SON Dataset s EDI Time interval 2004 12 06T23 50 00Z 2004 12 07T01 00 00Z Format VOTABLE http csa esac esa int csa 1 m
34. Y_INTERVAL DailySINGESTED SINCE 2010 03 10T10 21 00Z Request FGM products observed at Cluster 1 with dataset identifier 1 FULL for time range 2001 02 01 00 00 00 2001 02 05 00 00 00 that have been ingested since 2007 01 01T00 00 00Z Even when there are data for this time range NO RESULTS are obtained since the ingestion date is previous to the one established at the parameters Page 18 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public http csa esac esa int csa aio product action DATASET ID C1 CP FULL amp START DATE 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Daily amp INGESTED SINCE 2007 01 01T00 00 00Z e Request EDI products observed at Cluster 1 with dataset identifier 1 CP EDI for time range 2001 02 01 00 00 00 2001 02 05 00 00 00 whose total filesize is not greater than 9216 Kb 9 2Mb http csa esac esa int csa aio product ID C1 CP EDI EGD amp START DATE 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Daily amp MAX FILE SIZE 9216 Request ASPOC and EFW products for the same time range http csa esac esa int csa aio product action 2DATASET ID C3 CP ASP IONC amp DATASET ID2C1 CP EFW L1 P2 amp S TART DATE 2004 06 18T11 35 00
35. Z amp DELIVERY INTERVAL Daily Request ASPOC products observed at Cluster 3 with dataset identifier CP ASP IONC for time range 2004 06 18 11 35 00 2004 06 19 18 35 00 http csa esac esa int csa aio async product action DATASET ID C3 ASP IONC amp START DATE 2004 06 18T11 35 00Z amp END_DATE 2004 06 19T18 35 00Z Request ASPOC and EFW products for the same time range http csa esac esa int csa aio async product acion DATASET ID C3 CP ASP IONC amp DATASET ID C1 EFW L1 P2 amp STA RT DATE 2004 06 18T11 35 00Z amp END DATE 2004 06 19T18 35 00Z amp NO NOTIFY Page 23 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 6 STREAMING DATA REQUESTS Data Streaming allows a faster delivery of the data It indeed enables immediate streaming of one dataset to the users machine instead of a gzip file being created on the CSA server and then sent However the following constraints apply Only CEF products can be downloaded using these requests Only one dataset can be requested Only one file is delivered for the time period requested i e delivery interval option is not available Header only cannot be requested All the streaming data requests have the following HTTP prefix http csa esac esa int csa aio streaming action Only one type of data requests is available in s
36. Z amp END 2004 06 19 18 35 007 Page 19 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Ma sesa 4 2 Header requests After the HTTP prefix the following parameters are used to build the header request Parameter Name s Description Mandatory Possible Values RETRIEVALTYPE Type of retrieval For this request YES PRODUCT or HEADER HEADER DATASET_ID Unique identifier of the dataset YES Any of the Cluster whose metadata header we want to datasets retrieve It can appear more than once and can contain wild cards NON BROWSER Indicates that the request is not performed from a Browser If not specified it means that the requests are done from a browser and proper HTML pages will be displayed in order to present information to the user Table 5 Header request parameters Although we strongly recommend the use of authentication session cookies it is also possible to include the user credentials in the request Authentication Description Mandatory Parameters USERNAME CSA user name NOT recommended use of cookie instead PASSWORD CSA password NOT recommended use of cookie instead Examples e Request dataset metadata header files for all CIS Dataset for RPA mode http csa esac esa int csa aio product action RETRIEVALTYPE HEADERGDATASET CIS C
37. a CSA entities and attributes query response PAGE SIZE amp PAGE Number of records per page amp page NOT but Any integer value number 5 mmended a restrictions of the request ython valid query a see below a BY Criteria to sort 2 results Any CSA attribute See List of CSA entities and attributes NON_BROWSER Indicates that the request is not NOT performed from a Browser When not specified requests done from a browser proper HTML pages will be displayed in order to present information to the user Table 1 Metadata request parameters 1 International Virtual Observatory standard for the interchange of data http www ivoa net Documents VOTable Page 8 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public In addition to all these parameter names simple attribute restrictions can be added as Entity Attribute value to the metadata request for example EXPERIMENT NAME ASPOC Once again the possible values for this Entity and Attribute fields need to be compliant with the List of CSA entities and attributes Please note that e Any value that contains special chars or hyphens should be contained inside apostrophes For example dataset id C1 CIS CODIF HS H1 CS e Dates are expressed in yyyy MM dd format and are contained inside apostrophes e HTTP pre
38. age 44 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public These set of commands creates in your local folder CSA Dowload type folder with the following folder file inside C2 CP RAP PAD L3DD C2 RAP PAD L3DD 20100101 000000 20100101 2 35959 V110630 cef 112 IDL Below you can find two IDL routines developed by Andrew Walsh that use the CAIO to download data directly from IDL They work with Linux Mac and Windows enabling login and product actions The error handling will be improved in the near future together with the actions for unpacking the downloaded tar files csa login function csa login user pass Function that logs in to the CSA AIO system Parameters USER String containing your RSSD LDAP user identifier PASS String containing your RSSD LDAP password Fr Return Value If login is successful returns a string containing JSESSIONID cookie If login is unsuccessful returns 0 Example H credentials csa login username password Andrew Walsh 2013 05 30 awalsh AT rssd DOT esa DOT int Updated to add support for reserved characters in passwords Andrew Walsh 2015 01 28 awalsh AT sciops DOT esa DOT int Check for reserved characters in password and replace with URL Encoded equivalents reserved Jo m
39. date iam mm ssZ The size of the maximum amount of NOT A positive integer uncompressed data that can be representing the downloaded in Kbytes If the query maximum file size in results in a delivery exceeds the Kbytes specified filesize the user will receive an error message NON BROWSER Indicates that the request is not NOT performed from a Browser If not specified it means that the requests are done from a browser and proper HTML pages will be displayed in order to present information to the user Table 4 Product request parameters Although we strongly recommend the use of authentication session cookies it is also possible to include the user credentials in the request Authentication Description Mandatory Parameters USERNAME CSA user name NOT recommended use of cookie instead PASSWORD CSA password NOT recommended use of cookie instead Page 16 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public esa Examples e Request ASPOC products observed at Cluster 3 with dataset identifier CP ASP ION for time range 2004 06 18 11 35 00 2004 06 19 18 35 00 http csa esac esa int csa aio product action DATASET ID C3 CP ASP ION amp START DATE 2004 06 18T11 35 00Z amp END_DATE 2004 06 19T18 35 00Z Final Archive Inter Operability subsystem Mozilla Firefox Emi
40. e lt username gt and lt password gt by your CSA username and password in the example below Example URL http csa esac esa int csa product action fileName tempname gzFileName fileName gz gzFileName st urlwrite URL gzFileName Authentication Basic Get Username lt username gt password lt password gt DATASET CP WHI ELECTRON DENSITY START 2011 11 10718 00 007 DATE 2011 11 10T21 00 00Z NON BROWSER gunzip gzFileName fileNames untar fileName for iFile 1 numel fileNames disp fileNames iFile end This set of commands creates in your local folder a CSA_ Download type folder with the following folder file inside C1 WHI ELECTRON DENSITY C1 CP WHI ELECTRON DENSITY 20111110 180000 20111110 210000 V120703 cef 1L12 For users with Matlab 20 12b or earlier releases Simply replace username and password by your CSA username and password in the example below Example URL CSA http csaint esac esa int csa aio product actionPUSERNAME lt username gt SPASSWORD lt password gt amp DATASET ID C2 CP RAP PAD L3DDGSTART DAT E 2010 01 01T00 00 00Z amp END DATE 2010 01 01123 59 59Z amp NON BROWSER fileName tempname gzFileName fileName gz gzFileName st urlwrite URL_CSA gzFileName gunzip gzFileName fileNames untar fileName for iFile 1 numel fileNames disp fileNames iFile end P
41. ecsececseesesseaesecsesecaecaessesecaesecseseeseeeeseeaesecaeeeaeeeeseeeeseeaeats 7 2 METADATA REQUESTS wisicscsasisscsecsssccscsssssocsscsscccascedsecsasescccssecssestececdsedvecsvecenecssvcedesssccscsossecsscosdonse 8 2 1 Metadata Request Parameters cccccsceccscssessesessesessescescsessesesscsseceseesesecseseeseeneseesecaeseeseeeeseesesessesecaeeeseeeeseeseaeeaeeeaeeeeseeeeas 8 2 2 Userof QUERY Parameter s ccccecasccacecesccecececcsesececesescsedecesesncacesssacacecesadasadesesesacsacedanenencdatensncteneacnebenedscedecedacensdecsnencnencs 10 3 INVENTORY REQUESTS oisscississccccscasccovecsvecnees decceackactactveedseetseecsoveveesseesseedsesswetecdccenseassctacsecceacceses s 13 4 DATA REQUEST 65 41 PPOGUCETCCUCSES NAE M 15 42 Hedderrequests 20 5 ASYNCHRONOUS DATA REQUESTS 21 5 1 Asynchronous ETERS Bere E 21 6 STREAMING DATA REQUESTS wii sccscsssscssssccsccensscscteesccesaesiocvcacvectscswescsectsesssesscesanccasecesccsbseassesessaesse 24 6 1 Streaming Product requests ccc c scccsscecssccesecescsesececesesesececedesececedesedacecedesesedesedesacsaceessdensncdstensscrsescsbebcons
42. er s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public PRODUCT REQUESTS Table 8 Streaming Product request parameters PARAMETER NAME S Unique identifier of the dataset whose data we went to retrieve Any of the Cluster DATASET_ID Please note that wild cards are NOT allowed datasets r ISO time format yyyy START_DATE 4 END_DATE NON_BROWSER Figure 5 HTTP streaming product request Page 26 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 7 CREDENTIALS REQUESTS Data requests and Asynchronous Data requests require the user to log in first The user details will be stored in a session cookie That means that the credential will be accessible as long as the user is using the web application The cookie will be deleted when the connection is closed for example when the browser exits All credentials requests have the following HTTP prefix http csa esac esa int csa aio login action There are two types of credentials requests e Login requests e Logout requests Please note that for credential requests e HTTP prefix and parameters are separated by a question mark e Requests are built using parameter value Sentences separated by an ampersand amp e Any value t
43. et content disposition http csaint esac esa int csa aio streaming action USERNAME lt username gt amp PASSWORD lt password gt amp DATASET ID C4 CP PEA LERL PSD amp START DAT E 2004 09 18T10 00 00Z amp END DATE 2004 10 20122 00 00Z amp GZIP 1 amp NON BROWSER Authentication cookie wget load cookies cookies file gt txt content disposition http csa esac esa int csa aio streaming action DATASET 4 CP PEA LERL PSD amp START DATE 2004 09 18T10 00 00Z amp END DATE 2004 10 20T22 00800Z amp NON BROWSER stores the retrieved data at a gzip file with a random name for example 1fee8280 0759 4d7b 9071 fd088ae49c37 cef gz If the parameter GZIP is not present or its value is 0 then this request stores the retrieved data at a cef file with a random name for example 4 47622 5 49 4 689 Page 40 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 27a55822da86 cef If a user cannot be authenticated wget will output an authentication log n file instead 10 2 Inventory request See list of parameters at section 3 There is no need to authenticate when requesting inventory information Please find below how to execute the first example presented in the Inventory tab wget content disposition http csa esac esa int csa aio metadata action SELECTED PFIELDS
44. etadata action SELECTED FIELDS DATASET INVENTORY amp RESOURCE CLASS DAT ASET INVENTORY amp QUERY DATASET INVENTORY DATASET ID 20like 20 C 25 EDI 9025 CDF 20AND 20DATASET INVENTORY START 20 lt 20 2004 12 07T01 00 00Z 20AND 20DATASET INVENTORY END TIME 20 gt 20 200 4 12 06123 50 00Z Note that as an Inventory request is a metadata request we can use at the QUERY parameter other restrictions too For example experiment name Below an example of how to retrieve info from ASPOC for a given time range Experiment ASPOC Time interval 2006 06 01T13 00 00Z 2006 06 02T01 00 00Z Format CSV http csa esac esa int csa 1 metadata action SELECTED FIELDS DATASET INVENTORY amp RESOURCE CLASS DAT ASET INVENTORY amp QUERY EXPERIMENT NAME9620 29620 ASPOC 9620AND 9e20DATASET INVENTORY START TIME9620 lt 9 20 2006 06 02T01 00 00Z2 20AND 20DATASET _INVENTORY END_TIME 20 gt 20 200 6 06 01T13 00 00Z amp RETURN TYPE CSV Page 14 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public 4 DATA REQUESTS Data requests are used to retrieve files These files can be either files that belong to a given dataset and that cover a given timespan or files that contain the metadata header information of datasets They are requests that the user will have to build using several param
45. eters Once this request is sent the CSA backend will return one HTML link to download one specific file All the data requests have the following HTTP prefix http csa esac esa int csa aio product action There are two types of data requests e Product requests e Header requests The kind of request you want to use is defined by the parameter RETRIEVALTYPE Depending on its value your request will be processed differently and different combination of parameters will be expected Please note that for data requests tis mandatory that the user is logged in see Chapter 6 e HTTP prefix and parameters are separated by a question mark Requests are built using parameter value sentences separated by an ampersand 62 e Any value that contains special chars or hyphens should be contained inside apostrophes 7 4 1 Product requests These requests retrieve files of data observed The maximum size of data to download in a single product request is 100MB For files the files delivered are direct copies of the files stored at the archive For CEF files new CEF file s is are created by extracting the data from the original files for the timespan given the file format CEF or CDF can be specified as the time interval that the delivered files will cover After the HTTP prefix the following parameters are used to build the product request Parameter Name s Description Mandatory Possible Values DAT
46. fix and parameters are separated by a question mark e Requests are build using parameter value Sentences separated by an ampersand amp Let s see the use of all the above through some examples e List Datasets 401 to 500 of PEACE experiment in VOTable format http csa esac esa int csa aio metadata action PAGE SIZE 100 amp PAGE A amp RESOURCE CLASS DATASET amp SELECTED FELDS ZDATASET amp EXPERIMENT NAME PEACE amp RETURN TYPE VOTABLE Bl x Ble Edit View History Bookmarks Tools Nen Tab esa inticf ACE SIZE 100SPACE aSRESOURCE 2 You have chosen to open 2 CFA Results xml Do this automatically for files like this from now on Figure 1 Metadata Request e List first 10 datasets for experiment CIS in CSV format selecting only ID Start and End Date and Title http csa esac esa int csa aio metadata action PAGE SIZE I0 amp PAGE I amp SELECTED FIELDS DATASET DATASET ID Page 9 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public DATASET START DATE DATASET END DATE DATASET TITLE amp RESOURCE LASSZDATASET amp EXPERIMENT NAME CIS amp RETURN TYPE CSV e ist all datasets for experiment ASPOC in VOTable format selecting ID Start and End Date Title and Description http csa esac esa
47. gin header JSESSIONID csa cred end strpos csa login header creds strmid csa login header csa cred start csa cred end cred start return creds endelse end csa product function csa product csa creds csa dataset id csa start date csa end date delivery format csa delivery format delivery interval csa delivery interval ref ref doc ingested since csa ingested since max file size csa max file size Function that Downloads data from the CSA AIO system Parameters CSA CREDS String containing the JSESSIONID cookie returned from csa login pro CSA DATASET ID String array containing one or more valid dataset ids one per element CSA START DATE String containing start date for data request in the format yyyy mm ddThh mi ssZ CSA END DATE String containing end date for data request in the same format as csa start date Keywords Optional parameters accepted by the CSA AIO product action with the exception of NON BROWSER and RETRIEVALTYPE are accepted as keywords Optional parameters MUST be passed as strings to the relevant keyword See the CSA AIO User Manual for accepted values Note START DATE and END DATE are mandatory and passed as parameters here Return Value If download is successful returns a string containing the downloaded file name If download is unsuccessful returns 0 or falls over gracelessly
48. hat contains special chars or hyphens should be contained inside apostrophes 7 7 1 Login requests After the HTTP prefix the following parameters are used to build the login request Parameter Name s Description Mandatory Possible Values CSA system EMEN cae system SUBMIT LOGIN as the request is for a login em or LOGOUT NON BROWSER Whether the request is performed NOT from a Browser If not specified the requests is done from a browser and proper HTML pages wil be displayed Table 8 Login request parameters If we are using a browser and we want to perform a Data or Asynchronous data request without being logged in a pop up window will be displayed requesting the username and password no need to make explicitly the call to the login request Page 27 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public cluster Archive Inter Operability subsystem AIO FOLLOW ESA SCIENCE COPYRIGHT 2000 2013 EUROPEAN SPACE AGENCY ALL RIGHTS RESERVED TERMS AND CONDITIONS Figure 6 Pop up window for login On the other hand if we are executing a Data or Asynchronous data request with the NON_BROWSER parameter and we are not logged in this will be the response of the system LOGIN REQUESTED lt http data or asynchronous data request gt Example Login of a user in
49. ing dir tcsa filename print Downloaded data to csa working dir csa filename outfile csa working dir csa filename return outfile endif else begin Page 48 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public print Something went wrong return 0 endelse end px Page 49 49 CAIO User s Manual Date 03 02 2015 Issue 1 Rev 8 European Space Agency Agence spatiale europ enne
50. int csa aio metadata action SELECTED FIELDS DATASET DATASET_ID DATASET START_DATE D ATASET END DATE DATASET TTILE DATASET DESCRIPTION amp RESOURCE LASS DATASET amp EXPERIMENT NAME ASPOC amp RETURN TYPE VOTABLE New Tab file C DOCUME 1 CFA Results 3 xml T 8 fie C 00CUME 1 omartinez Local Settings Temp CFA Results 3 xml B Google lt VOTABLE version 1 1 gt al lt type results gt lt DESCRIPTION gt CFA Archive Inter operability Service Total number of records 23 lt DESCRIPTION gt lt INFO name QUERY_STATUS value OK gt lt INFO gt lt TABLE gt lt FIELD ID DATASET DATASET_ID datatype char arraysize gt lt FIELD ID DATASET START datatype char arraysize unit iso 8601 gt lt FIELD ID DATASETEND_DATE datatype char arraysize unit iso 8601 gt FIELD ID DATASET TITLE datatype char arraysize gt lt FIELD ID DATASET DESCRIPTION datatype char arraysize gt lt FIELD ID DATA_LINK gt lt FIELD ID DETAILS_LINK gt lt DATA gt lt TABLEDATA gt lt gt lt gt 1 5 lt gt lt TD gt 2001 01 01 00 00 00 0 lt TD gt lt TD gt 2006 01 01 00 00 00 0 lt TD gt lt TD gt ASPOC instrument caveats lt TD gt lt TD gt This dataset contains the instrument caveats from the ASPOC experiment on the Cluster C1 spacecraft lt TD gt lt
51. iple datasets options will retrieve all the datasets that match the result of a query Internally they will make several requests one to execute the query and one for each of the dataset identifiers returned from it 9 1 Syntax for the J ava client Windows users Please be aware that the classpath declaration in Windows is different The syntax below will be java cp savotjar The syntax for the different options is e Dataretrieval from a single dataset Simply replace username and password by your CSA username and password in the example below java AioWEBClient S Server Name L DATASET ID value P lt PARAMETER NAME gt lt value gt AIOUSER username AIOPWD lt password gt O output dir Data retrieval from multiple datasets Simply replace username and password by your CSA username and password in the example below java cp savot jar AioWEBClient S Server Name gt L ALL complex query value gt P lt PARAMETER NAME gt lt value gt AIOUSER username AIOPWD lt password gt O output dir e Asynchronous Data retrieval from multiple datasets Simply replace username and password by your CSA username and password in the example below Page 33 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the P
52. l script aioclient with all the default values inside so you do not need to enter them in the command line after you set them inside Edit the script file and enter your values for the variables AIOUSER lt your CSA user name gt AIOPWD lt your CSA password jdkhome lt installation directory of your ava environment gt usr local java1 6 Example Jaiodient L GET ALL DATASET DATASET ID CP EDI EGD OR DATASET DATASET 10 START DATE 2001 02 010100 00 00Z END DATE 2001 02 01T12 00 00Z O FebruaryDirectory Where the files will be created in the folder FebruaryDirectory Page 34 49 CAIO User s Manual European SI Date 03 02 2015 Issue 1 Rev 8 Agence spatiale ESA UNCLASSIFIED Releasable to the Public 9 2 Examples of use Data retrieval from a single dataset Retrieve ASPOC Ion current data from C3 for time range 2004 06 18T11 35 00Z 2004 06 19T18 35 00Z java AioWEBClient S csa esac esa int csa L GET DATASET ID2C3 ASP IONC P START DATE 2004 06 18T11 35 00Z DATE 2004 06 19T18 35 00Z Data retrieval from multiple datasets Retrieve data from all Science datasets from FGM between the time range 2002 02 12123 30 007 2002 02 13T01 00 00Z and leave them at fgmDir java cp AioWEBClient S csa esac esa int csa L ALL EXPERIMENT NAME FGM AND DATASET MAIN
53. le below http csa esac esa int csa aio product action USERNAME lt username gt SPASSWORD lt pwd gt S amp RETRIEVALTYPE HEADER amp DATASET ID C _CIS CODIF RPA amp NON BROWSER 119 9 Include dataset reference documents This use case retrieves particular products with the additional criteria any documents referenced in the metadata are to be included in the delivery Type of request Data product request Example Simply replace username and lt pwd gt by your CSA username and password in the example below http csa esac esa int csa aio product action USERNAME username amp PASSWORD pwd amp DATASET ID2C3 CP RAP DE amp START DATE 2003 04 01T00 00 00Z amp END DATE 2003 04 02T00 00 00Z amp REF DOC 1 amp NON BROWSER 1110 UC10 Inventory information This use case retrieves the inventory information for a list of datasets that match a search criteria in JSON format By default inventory information includes the dataset ID start stop time for the interval the number of instances and the version Type of request Inventory request Example http csa esac esa int csa aio metadata action SELECTED FIELDS DATASET INVENTORY amp RESOURCE CLASS DATASET INVENTORY amp QU ERY DATASET INVENTORY DATASET ID9620like9620 C1 FULL 96200R9620DATASET INV ENTORY DATASET ID 20like 20 C1 SPIN 20AND 20DATASET INVENTORY START ME 20 lt 20 2008 02 04T00 00 00Z
54. m nx OW ma Wo moms qm gem cmm encoded m m m m m m now 5 25 2 23 24 7 26 7 27 290 20 T 2AT 2B 20 S2E S3A 3B 3p 940 S5B 5 1 for i 0 n_elements reserved 1 do begin startpos 0 foundpos 0 reserved positions 1 while foundpos ne 1 and startpos le strlen pass 1 do begin foundpos strpos pass reserved i startpos Page 45 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public if foundpos ne 1 then begin if startpos eq 0 then reserved_positions foundpos else reserved_positions reserved_positions foundpos startpos foundpos l endif endwhile if reserved_positions 0 ne 1 then begin reserved positions reverse reserved positions for j 0 n elements reserved positions 1 do begin if reserved positions j eq strlen pass 1 then begin pass before strmid pass 0 reserved_positions j new pass pass before tencoded i pass new pass ndif else if reserved positions j eq 0 then begin pass after strmid pass reserved positions jl 1 strlen pass reserved positions j 1 new pass encoded i pass after pass new pass ndif else begin pass before strmid pass 0 reserved positions jl
55. meters are separated by a question mark e Requests are built using parameter value Sentences separated by an ampersand amp e Any value that contains special chars or hyphens should be contained inside apostrophes 7 5 1 Asynchronous Product requests The maximum size of data to download in a single asynchronous product request is 50GB After the HTTP prefix the following parameters are used to build the product request Parameter Name s Description Mandatory Possible Values DATASET ID Unique identifier of the dataset whose YES Any of the Cluster data we want to retrieve It can appear datasets more than once and can contain wild cards RETRIEVALTYPE Type of retrieval For this request NOT PRODUCT or PRODUCT default value HEADER START DATE Start time of the time range that the NOT but ISO time format retrieved files should cover By default recommended yyyy MM Cluster Mission start time ddTHH mm ssZ END DATE End time of the time range that the NOT but ISO time format retrieved files should cover By default recommended yyyy MM Cluster Mission end time ddTHH mm ssZ DELIVERY FORMAT Delivery format applicable only to CEF CDFor CDF 2 7 Page 21 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public CEF files By default CEF Ee _ DELIVERY INTERVAL Specifies the delivery file interval
56. p RETURN TYPE CSV 114 UC4 Query by ingest date This use case retrieves a particular product that have been ingested since a particular date in daily files Type of request Data product request Example Simply replace username and lt pwd gt by your CSA username and password in the example below Page 5 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public http csa esac esa int csa aio product action USERNAME username amp PASSWORD pwd amp DATASET ID C1 CP EDI EGD amp SIART DAT E 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Daily amp INGESTED SINCE 2010 03 10T10 21 00Z amp NON_ BROWSER 115 UC5 Query specifying download filesize limit This use case retrieves a particular product with the additional criteria that the result cannot exceed a filesize limit Type of request Data product request Example Simply replace username and lt pwd gt by your CSA username and password in the example below http csa esac esa int csa aio product action USERNAME username amp PASSWORD pwd amp DATASET ID C1 CP EDI EGD amp SIART DAT E 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Daily amp MAX FILE SIZE 9216 amp NON BROWSER 116 UC6 List datasets without description This use case retrieves the list of datase
57. pass after strmid pass reserved positions j l 1 strlen pass reserved positions j 1 new pass pass beforetencoded i pass after pass new pass endelse endfor endif endfor construct host path and query parts of URL csa host csaint esac esa int csa login path csa aio login action csa login query USERNAME user amp PASSWORD pass amp SUBMIT LOGIN amp NON_BROWSER define successful login string for comparision with what s returned from CSA AIO csa login success usert has successfully logged create and set properties of an IDLnetURL object csa login obj obj new IDLnetUrl csa login obj SetProperty url scheme http csa login obj SetProperty url host csa host csa login obj SetProperty url path csa login path csa login obj SetProperty url query csa login query send login request to CSA AIO Page 46 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public csa login response csa_login_obj gt get string_array print response from CSA AIO print csa_login_response check for successful login and extract session cookie from CSA AIO response if csa login response 0 ne csa login success then begin return 0 ndif else begin csa login obj getproperty response header csa login header csa cred start strpos csa lo
58. rs 03 02 2015 Added example of Double Star data download 03 02 2015 Page 2 49 CAIO User s Manual Date 03 02 2015 Issue 1 Rev 8 ESA UNCLASSIFIED Releasable to the Public Table of contents 1 INFRODUCTION 4 11 Use CASES m S u asuy Su katu asha saka uuu 4 111 UC1 General query to be executed immediately 4 112 UC2 General query to be executed asynchronous y 5 113 SC PE ll EE 5 114 UCA Query 5 81110 266 5 115 UC5 Query specifying download filesize limit eene netten nete nnns 6 116 UCG6 List datasets without description n n snn 6 117 UC7 List datasets with 6 118 UCS8 Get dataset metadata 6 119 UC9 Include dataset reference documents U 7 1110 UC10 Inventory information eeeeseseecessesessesesseseseeseseese
59. rt for Content Disposition headers is enabled This can currently result in extra round trips to the server for a HEAD request and is known to suffer from a few bugs which is why it is not currently enabled by default This option is useful for some file downloading CGI programs that use Content Disposition headers to describe what the name of a downloaded file should be Page 41 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Thus maybe sometimes especially with very old versions of wget this may not work in which case you can always provide an output filename wget O myRequest tar gz load cookies cookies txt http 1 product ID2C3 CP ASP IONC amp START DATE 2004 06 18T11 35 00Z amp END 2004 06 19T18 35 00Z amp NON_ BROWSER Please have in mind that if you do a sequence of wgets the output filename must be changed otherwise it will be overwritten 10 5 How to create an authentication cookie In order to download data from Cluster Archive it is required to be able to authenticate the user For this reason it is strongly advised to create and store a session cookie that identifies the user The syntax to create and store the cookie is wget keep session cookies save cookies cookies file txt content dispo
60. sceasbeccendsaessoaasnenes 24 7 CREDENTIALS REQUESTS eer ee ere e bean ree d e Ye ES ea ERR SL RR prrs FREU S Ue YE REPRE EE KO BER a PES ES NEES 27 71 Eo ainiin o pol e uuu u 27 7 2 Logot regueton anaia y EE a EA E E EEE EO AE EEE E E lasa 29 8 LIST OF CSA ENTITIES AND ATTRIBUTES 30 9 APPENDIX HOW TO RUN THE DEFAULT CLIENT 33 91 Syntax for the Java client 33 9 11 Syntax for the shell script with the J ava client 34 9 2 Examples 156 26 35 93 Equivalence to data requests cccssescssssssseseesessesesceseseeseseesesecaeeeseeaeseeaesecaesecaesaesaesecsesecaeseeseeeeaeeaeaesaeseeaeeeeaeeaeas 36 10 APPENDIX HOW TO USE WITH WGET 38 10 1 Data download
61. sible to include the user credentials in the request Authentication Description Mandatory Parameters USERNAME CSA user name NOT recommended use of cookie instead PASSWORD CSA password NOT recommended use of cookie instead Examples e Request EDI EFW and FGM products for 4 days split into day files http csa esac esa int csa aio async product ID C1 CP EDI EGD amp DATASET ID C1 CP EFW L3 P amp DATA Page 22 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public SET ID2C1 FULLGSTART DATE 2001 02 01100 00 00Z amp END DATE 2001 02 05T00 00 00Z amp DELIVERY INTERVAL Dail cluster Archive Inter Operability subsystem Your request has been sent connecrwithus 8 9 FOLLOW ESA SCIENCE W COPYRIGHT 2000 n AGENCY ALL RIGHTS RESERVED TERMS AND CONDITIONS Figure 4 HTTP Asynchronous Data request Use of user credentials in the previous request Simply replace lt username gt and lt password gt by your CSA username and password in the example below http csa esac esa int aio async product action2USERNAME username amp PASSWORD password2 amp DATASET ID C1 EDI EGD amp DATASET CP EFW L3 P amp DATASET ID C1 CP FGM FULL amp START DATE 2001 02 01T00 00 00Z amp END DATE 2001 02 05T00 00 00
62. sition http csa esac esa int csa aio login action USERNAME lt username gt amp PASSWORD lt pwd gt SSUBMIT LOGIN amp NON_BROWSER Where e lt cookies file gt is the name of the file where the user wants to store the cookie If no path is supplied it will be created at the current directory so if you will be executing wget from different directories we recommend to provide an absolute path username is the CSA user identifier e password is the CSA password of the user It creates a cookies file with content similar to this one HTTP cookie file Generated by Wget on 2013 05 21 17 10 34 Edit at your own risk csa esac esaint FALSE FALSE 0 JSESSIONID 290552733 015064370 02677 0 02 6D7A1450672A015D75650D4060284656722A465A6F3F1451642A1C416831465345391C5A6F2E12737228015C7138465 6722A465A6F3F The login request will return a file named credentials log indicating whether the user has been successfully logged in or not The authentication cookies generated by CAIO are session cookies meaning that they are not stored and are deleted when the browser exits This is the reason for using keep Page 42 49 CAIO User s Manual European Space Agency Dete 03 02 2015 Issue 1 Rev 8 A gence spa tiale europ e nne ESA UNCLASSIFIED Releasable to the Public session cookies option at wget when saving the cookie As long as the cookies file is not modified it can be used in all
63. treaming mode Streaming Product requests Please note that for streaming data requests 6 1 It is mandatory that the user is logged in see Chapter 6 HTTP prefix and parameters are separated by a question mark Requests are built using parameter value Sentences separated by an ampersand amp Any value that contains special chars or hyphens should be contained inside apostrophes 7 Streaming Product requests After the HTTP prefix the following parameters are used to build the product request Parameter Name s Description Mandatory Possible Values DATASET ID Unique identifier of the dataset whose YES Any of the Cluster data we want to retrieve Please note datasets that wild cards are NOT allowed retrieved files should cover MM ddTHH mm ssZ MM MATH retrieved files should cover MM ddTHH mm ssZ GZIP Specifies if the retrieved file is gzip 1 compressed not default 0 NON_BROWSER Indicates that the request is not NOT performed from a Browser If not specified it means that the requests are done from a browser and proper HTML pages will be displayed in order to present information to the user Table 7 Streaming Product request parameters Page 24 49 User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public I Although we strongly recommend the
64. ts that match a search criteria The response in J SON format will contain only dataset ID start date end date and title Type of request Metadata request Example http csa esac esa int csa aio metadata SIZE IO amp PAGE I amp SELECTED FIELDS DATASET DATASET ID DATASET START DATE DATASET END DATE DATASET TITLE amp RESOURCE CLASS DATASET amp EXPERIMENT NAME CIS amp R ETURN TYPE j SON 117 UC7 List datasets with description This use case retrieves the list of datasets that match a search criteria The response in CSV format will contain dataset ID start date end date title and description Type of request Metadata request Example http csa esac esa int csa aio metadata action SELECTED FIELDS DATASET DATASET ID DATASET START DATEDATASET END DATE DA TASET TTILE DATASET DESCRIPTION amp RESOURCE CLASS DATASET amp QUERY DATASET DATASET ID 20like 20 C 25 EGD 96200R9620DATASET DATASET ID9620like9620 9625RAP 9625 amp RE TURN TYPE 2CSV 118 UC8 Get dataset metadata information This use case retrieves the dataset headers in XML format that match a search criteria Page 6 49 CAIO User s Manual European Space Agency Date 03 02 2015 Issue 1 Rev 8 Agence spatiale europ enne ESA UNCLASSIFIED Releasable to the Public Type of request Data Header request Example Simply replace lt username gt and lt pwd gt by your CSA username and password in the examp
65. ublic Where java cp savotjar AioWEBClient S Server Name gt L ASYNC complex query value gt P lt PARAMETER NAME gt lt value gt AIOUSER lt username gt AIOPWD lt password gt O output dir gt Server name is the http server location csa esac esa int csa for the operational archive value is the value of a parameter either of a DATASET ID or of any of the product request parameters see Table 4 NOTE if value contains white chars or hyphens as in CP CIS CODIF HS 1 CS then it must go between single quotes 9 complex query value equivalent to the value of a QUERY parameter within a Metadata request see QUERY parameter username and password are optional arguments to include the CSA login detail in the command when creating the query If these are not provided the user will be prompted to do so on the command line output dir is an optional argument for specifying where the user wants to store the files downloaded It is important to note that in the multiple datasets request we use pagination in order to prevent scalability problems in the client side If the number of records that fulfil the query is greater than the defined page size internally set to 1000 the system makes several different requests to retrieve them all 9 11 Syntax for the shell script with the J ava client If you work on UNIX we provide inside the package a shel
66. use of authentication session cookies it is also possible to include the user credentials in the request Authentication Description Mandatory Parameters USERNAME CSA user name NOT recommended use of cookie instead PASSWORD CSA password NOT recommended use of cookie instead Examples e Request DWP caveats for time range 2007 01 15 18 00 00 2007 01 16 00 00 00 with non browser option http csa esac esa int csa aio streaming action DATASET_ID C3_CQ DWP INST amp START DATE 2007 01 151T18 00 00Z amp END DATE 2007 01 16T18 00 00Z amp NON BROWSER Use of user credentials in the previous request Simply replace username and password by your CSA username and password in the example below http csa esac esa int csa aio streaming action PUSERNAME username amp PASSWORD password2 amp DATASET ID 2C3 _ CQ DWP INSI amp SIART DATE 2007 01 15T18 00 00Z amp END DATE 2007 01 16118 00 00Z amp NON BROWSER e Request ASPOC products for more than 1 day through the browser http csa esac esa int csa aio streaming action DATASET_ID C3 ASP IONS amp START DATE 2004 06 18T11 35 00Z amp END DATE 2004 06 19T18 35 00Z e Request PEACE products for more than 1 month in compressed format http csa esac esa int csa aio streaming action 2DATASET ID C4 CP PEA LERL PSD amp START DATE 2004 09 18T10 00 00Z amp END_DATE 2004 10 20T22 00 00Z amp GZIP 1 Page 25 49 CAIO Us

CAIO User`s Manual

Contents

Download Pdf Manuals

Related Search

Related Contents