Home

User manual - the LSA SAF

1. r 4 Settings Temporal compositing ET Evapotranspiration 30min ET X Day Advanced B rQuality Filtering Data value range Filter using error margin E Filter using bitwise quality fiag Lower Limit 0 0 w Upper Limit 0 0 Upper Limit 0 5 Data and error limits are expressed in physical unit mm B Remapping to Region of Interest Europe sub nadir Europe with sub nadir pixel size Restrict imported regions Euro NAfr SAfr SAme a to vi Composite files File formatting AVG MIN MAX SUM IDL ENVI SPIRITS img Daily composites vj o vj vj GeoTiff tiff vj Periodic composites ILWIS mpr v a Nbr observations V 5 ion Name ET Description ET example scenario a B Save Cancel Figure 31 Scenario settings window The scenario settings window itself can be split up into several sub panels as described in the following sections e Temporal compositing on top for selecting the layer and composite period e Quality filtering for optional filtering on error uncertainty values and bitwise quality flags e Remapping to Region of Interest for selecting previously created ROI in the remapping and fine tune the input regions NAfr etc to be considered e Composite files to define the compositing rules minimum maximum average sum per day or per period and optional number of observations e File formatting to select additional output file formats e Scenario info
2. Figure 24 Warning when recursively searching the input data folder The set of input products that the MSG Toolbox considers and searches for in the input data folder and possibly its sub folders is determined by several factors e The data layer being composited fAPAR LAI DSSF and thus the product type they are part of e g FWI layer is part of the Fire Risk product FRM and stored in files called HDF5 _FRM DSSF data layer is located in product files called HDF5 _DSSF and so on e The specified Region of Interest as this determines the relevant input regions e g HDF5 _Euro HDF5 NAfr e The relevant time period which is compared against the nominal date and time of the product e g HDF5 201201010000 The relevant time period is in turn determined through a combination of e The start and end date e The composite time period None Day Dekad Month Year selected in the scenario e The maximum consecutive gap which may be defined separately for daily and n day compositing e The hour UTC when the 24h period of the day starts 0 UTC by default The data layer the previously created Region of Interest the composite time period maximum consecutive gap and starting time of the day are all specified in the scenario see next section for more details VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 38 of 73 vision on technology Em LSA SAF
3. VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 35 of 73 LSA SAF 7 vito vision on technology r Information Task submitted Prepare ROI started Prepare ROI ended Build land sea mask started Build land sea mask ended Build inverted grid started Build inverted grid ended Task ready Grid saved name Europe sub nadir description Europe with sub nadir pixel size file D Workspace Eclipse MSGtoolbox MSGToolboxData grids Europe sub nadir xml Figure 19 Information dialog upon successful completion of the ROI flow The user is warned when trying to overwrite an already existing ROI as this may be used in the on going and or future scheduled tasks in the task queue When confirmed the overwritten ROI could disrupt these submitted tasks It is therefore recommended to first check the task queue cancel or pause relevant tasks before proceeding 1 Warning D ROI already exists and overwriting it may disrupt on going compositing Continue anyway Yes J No Figure 20 Confirmation to over write an existing ROI VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 36 of 73 LSA SAF 7 vito vision on technology 5 3 Producing composites Following the preparation of one or more Regions of Interest ROI for the study area s the Compositing tab figure below allows to produce different
4. 4 3 Products supported in the toolbox The MSG Toolbox aims to support all products from the LSA SAF that are operationally produced and thus routinely and openly available The first release of the toolbox will however not support products derived from MetOp polar orbiting satellite and is thus limited to products derived from Meteosat Second Generation MSG The LSA SAF is performing a number of changes to their products during the second Continuous Development and Operations CDOP phase such as the addition of new products and generating existing products at different time intervals As these products are not yet available they will only be considered in a future release of the toolbox VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 21 of 73 y m vito vision on technology The following table provides an overview of the support for the products that are currently operational in this first release of the toolbox Category Fire Fire Fire Vegetation Vegetation Vegetation Vegetation Vegetation Vegetation Snow Temperature Temperature Radiation fluxes Radiation fluxes Radiation fluxes Radiation fluxes Surface Albedo Surface Albedo Surface Albedo Evapo transpiration Evapo transpiration Product type Fire Radiative Power per fire pixel Fire Radiative Power grid Fire Risk Map Fraction of Cover Leaf Area Index Vegetation Fraction of Absorbed Pho
5. Figure 8 Welcome tab of the MSG Toolbox 5 2 Preparing Regions of Interest A separate workflow is to be run to prepare grids and land sea mask from static data as input to the remapping This is to be repeated for each study area and potentially also for different pixel sizes spatial resolutions levels of detail and with different advanced remapping configurations This remapping is needed to go from the native geostationary view projection to a regular geographic lat lon spatial reference system using the WGS84 datum Reprojection or warping to other spatial reference systems is not foreseen in the toolbox As the toolbox outputs are designed to fit into a variety of commonly used remote sensing and GIS or Remote Sensing software for further analysis those software packages can be used for the reprojection The preparation of Regions of Interest is done via the Regions of Interest tab shown in the below figure This tab is further divided into the following sub panels e For defining the rectangular bounding box coordinates Bounding box coordinates e For defining the pixel size spatial resolution Pixel size e For advanced remapping configuration Remapping e For additional information notably on the spatial reference system used Additional information e The Grid settings that contain in particular a name and short description that allow the computed ROI to be selected for use in the subsequent compositing workflows e And
6. LSA sar vito vision on technology Whereby var variable or product identifier code in the filename ALBEDO FARAR reg region one of Euro NAfr SAfr SAme dct daily composite type avg min max sum Examples sub daily product no composite flow grid using Euro NAfr and SAfr input files DSSF Euro 201202010130 img hdr DSSF_NAfr_201202010130 img hdr DSSF SAfr 201202010130 img hdr remapped file DSSF_201202010130 img hdr input files sub daily product daily composites flow grid using Euro NAfr and SAfr DSSF Euro 20120201 avg img hdr DSSF NAfr 20120201 avg img hdr DSSF SAfr 20120201 avg img hdr remapped file DSSF 20120201 avg img hdr Remarks e Inthe cases no composite flow and daily composite from sub daily products where no additional format conversion step is specified the remapped files will be placed in the output folder In all other cases the remapped files will be placed in the remap sub folder of the workspace folder 6 2 Viewing intermediate results The intermediate results are all stored in an ENVI file format with some extensions to the text header hdr to make it compatible with the Glimpse and SPIRITS software To visualize them it suffices to download the free SPIRITS software from http spirits jrc ec europa eu web site that also offers a tutorial and manual to get started Using the menu Analysis Maps it is possible to create a single quicklook
7. VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 23 of 73 amp isa sar vito vision on technology HINTS e The toolbox never mixes different types of products So when working with multiple types of products a separation of product types LAI FVC D10 should be done high up in the folder structure e g at first or second level e Keep only the relevant input regions EURO NAFR When in doubt on the regions needed prepare your ROI and do a small test run to see which regions are searched for Checkboxes in the Scenario configuration narrow these down further e When multiple input regions are used e g when looking at full MSG disc coverage the division by region should be low rather than high up in the folder structure The considered time period checked against the datetime in the filename is in turn determined by e The start and end dates specified via the text boxes or date pickers on the Compositing tab e The composite time period specified in the selected scenario e The maximum consecutive gap defined for the temporal compositing if any e The UTC starting time hour in the daily compositing if any And of course the folders where the toolbox will try to locate the relevant products are configured through e The top level input data folder e The checkbox to enable disable searching subdirectories HINTS e Make sure to read section 5 3 as it provides more informa
8. DT max hdr ERROR fun ERRmess 1 DTextremes i amp h gt bpp amp h gt Vlo amp h gt Vhi h gt data_type i optional Byte order If missing assume 0 as default h gt byte order atoi strHDR byte order str fp amp multi 1 0 h gt flip flip def h byte order h bpp if h gt flip amp amp flip ok Sprintf ERRmess Byteswap needed but not allowed in HDR s hdr ERROR fun ERRmess 1 Map Info amp Geo location optional strHDR map info str fp amp multi 1 0 if strlen str defaults for map info sprintf str Ss 9g 9g 9g 9g 9q 99 arbitrary CRmin CRmin 0 double h gt lines 1 0 1 0 Copy from header string to MAP INFO structure amp h gt mi within header structure h map find str amp h mi 0 h gt samples h gt lines 1 Optionally read projection info and coordinate system string For MSG Toolbox we only support Geographic lat lon in map info so don t really need this StrHDR projection info str fp amp multi 1 0 if strlen str gt sizeof h projection 1 str sizeof h gt projection 1 0 strcpy h projection str strHDR coordinate system string str fp amp multi 1 0 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 66 of 73 LSA SAF y m vito vision on technology if strlen str gt sizeof h gt coordsys 1 str sizeof h gt coordsys
9. if headerIn flip flip byte bufferIn headerIn samples BPPi byte swap process each pixel column in current line for column 0 column lt headerIn samples column some processing In Fortran90 programming language program img2ascci The main goal of this sample program is to clarify how to access img files data with Fortran using the directives present in the respective hdr file Compiling gfortran gfortran o img2ascci img2ascci f90 Usage Linux command line Input arguments filename The path to the img file lines The number of rows in the image samples The number of columns in the image bands The number of spectral bands in the image Optional if no header offset argument is given Default value is 1 header offset The number of bytes of data in the image file to skip in order to reach the start of the image data Optional The default 1 I 1 t I img2ascci filename lines samples lt bands gt lt header offset gt Y i I I j value is 0 bytes implicit none character 255 img filename input filename integer nrows ncols nbands skipbytes lines samples bands and header offset integer parameter data kind 4 number of bytes data type integer kind data kind dimension allocatable img data data handler integer nargs f unit io stat lin col band b
10. vito vision on technology 2 Installation and requirements 2 1 Requirements MSG Toolbox is written in the platform independent Java language and is designed as a shell that launches a few internal applications written in C The MSG Toolbox software is distributed in binary form without source code and only Microsoft Windows platforms Windows XP or more recent both 32 and 64 bit are currently supported In order to run MSG Toolbox properly a Java Virtual Machine JVM must be installed first A Java Virtual Machine is typically part of a larger Java Development Kit JDK intended for software developers or Java Runtime Environment JRE Though most Windows systems offer a Microsoft JVM the use of this JVM has not been tested It is therefore recommended to use Java from Oracle which can be downloaded and installed from http java com free of charge The toolbox works on Java version 7 JRE 1 7 or higher As it is possible to install multiple Java versions alongside each other and newer versions offer backward compatibility installing the latest Java should not cause any problems To check the version please type in java version at the command prompt or check Control Panel Java When processing MSG Toolbox will create files in temporary folders that may be located in its installation path For this reason it is required that there is sufficient free disk space 100MB or more on the hard disk where MSG Toolbox
11. 1 0 strcpy h gt coordsys str Glimpse SPIRITS header extensions ge Values strHDR values str fp amp multi 1 0 item char calloc strlen str 1 1 To hold 1 of the 8 elements if strsplit str 0 item 8 VALUES is missing or not properly defined 8 elements needed so fall back to defaults it empty variable name and unit strcpy h Vname strcpy h Vunit A the value range is set to full range of the data type FL set by DTextremes above h gt Vmin h Vlo h gt Vmax h gt Vhi No encoding scale 1 0 offset 0 0 h gt Vint 0 0 h gt Vslo 1 0 else VALUES has 8 elements as expected so parse them strsplit str 1 item if strlen item gt sizeof h gt Vname 1 item sizeof h gt Vname 1 0 element 1 variable name strcpy h Vname item if strlen h Vname 0 strcpy h Vname element 2 variable unit strsplit str 2 item if strlen item gt sizeof h Vunit 1 item sizeof h gt Vunit 1 0 strcpy h Vunit item if strlen h gt Vunit 0 strcpy h Vunit elements 3 amp 4 significant non flag digital value range strsplit str 3 item f atof item fl min f h gt Vlo f2 max f h Vhi if num diff fl h gt Vlo 0 000001 num diff f2 h Vhi 0 000001 sprintf ERRmess VALUE Vlo f beyond range f S
12. 1 momentaneous 0 unknown irrelevant Bg flags 254 out 255 sea Optional comment string Name and version number of creating program Sensor name type optional Projection info optional Coordinate System String optional ENVI classification not used in MSG Toolbox Nbr of classes consecutive starting at 0 Class names Per class RGB colors J F F K K R kk Sk kk ko kk kk kk kk kk ko ko ko ke ke IA K A k A K K A k A k K ko ke ke k AK k K k K K K k K k Kk k I kk ok k k k ke e e ee Routine to read relevant part of plain text header files into a ENVIHDR memory structure Parameters hdr and store the parsed information img pointer to the name of the image file for which the corresponding header is retrieved and parsed h pointer to header structure tis the header information pixel d3 ok allow multi layer file non zero checks the image s binary data size nbr of bytes with the expected value from e g number of lines x number of columns x number of bytes per e g 3 band RGB or require single layer DT min DT max range of ENVI data types that are supported offset ok require header offset of 0 or not flip ok allow byteswap or not kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkKkk f void envi hdr read char img struct ENVIHDR h short tis short d3 ok short DT min short DT max short offset ok short flip ok Char hdr MAX
13. ERROR str1 str2 exit value function prints an error message composed of str1 function name str2 actual error message and stops the program with exit value include stdio h include stdlib h include errno h include math h include float h include lt string h gt include lt ctype h gt include lt time h gt include lt io h gt include lt sys types h gt include lt sys stat h gt or include lt curses h gt lt sys types h gt and lt sys stat h gt for UNIX LINUX if defined _MAX PATH define MAX PATH 260 max length for windows path names endif for UNIX Linux equivalent string functions define stricmp strcasecmp unknown by UNIX define strnicmp strncasecmp define HDRlmax 10000 max length of ENVI header file lines Structures struct MAPINFO char name 101 Entry in ENVI file map proj txt short zone only for UTM char ns 11 only for UTM double mc mr magic pixel column record double mx my magic pixel X Y double dx dy pixel X Y size resolution double xmin xmax X X of absolute edges of image double ymin ymax char datum 101 optional char units 21 optional degrees or meters struct ENVIHDR char description 257 char file type 101 ENVI standard or ENVI classification long offset Byte offset in data file long samples Nbr pixels columns per line long lines Nbr
14. LSA SAF y vito vision on technology For the masking of the coarsest resolution pixels near the edge of the MSG disc a threshold is used on the fraction of the nominal and the effective resolution using the following formula Ratio 100 Nominal Resolution Effective Resolution Threshold Whereby e The nominal resolution is hereby defined as 3km e The Effective Resolution is computed as the angular distance between each input cell and its most removed neighbour in its 3x3 environment This threshold is set to 596 by default 096 disables it completely and 3096 is the most severe masking The below figures show the effect of changing this parameter from its default value of 596 to 3096 with green colour highlighting the pixel values that were masked Figure 14 Difference between 596 default and 3096 most severe threshold on nominal effective resolution fraction The accuracy of the grid inversion can be specified as a fraction of a pixel The 1 1 or full pixel precision should be limited to values that may not be interpolated When using sub pixel precision from half pixel 1 2 to one fifth of a pixel 1 5 the higher the accuracy the longer the grid preparation will take One fifth 1 5 pixel is the most accurate but also the slowest As most applications require only infrequent updates to the grids a high accuracy is recommended To get an idea of the effect of changing the accuracy setting here are extracts from th
15. NULL continue strmid p 1 1 HDRlmax dest strtrim dest 2 if strnicmp dest 1 found scalar parameter multi 0 return dest look for list of values between maybe multi line multi 1 for i 0 i strlen dest i dest i dest i 1 strtrim dest 0 l strlen dest while feof fptr amp amp dest strlen dest 1 record 0 0 fgets record HDRlmax fptr strtrim record 2 l 1 1 strlen record if 1 gt HDRlmax sprintf ERRmess ITEM s longer than allowed d bytes src HDRlmax ERROR LIBfunc strHDR ERRmess 1 strcat dest strcat dest record multi 4t if dest strlen dest 1 if fatal sprintf ERRmess ITEM s should end with src ERROR LIBfunc strHDR ERRmess 1 else multi 0 break dest strlen dest 1 0 strtrim dest 2 return dest if fatal sprintf ERRmess HDR item s not found src ERROR LIBfunc strHDR ERRmess 1 strcpy dest return dest VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 69 of 73 vision on technology e LSA SAF y vito 8 I IK KK kk AAAI ke ke kk ARR ke ke kk IRR ko ke ke ko ko kk kk eee A ke map find parses map info spatial information into MAP INFO structure Parameters s string containing the map info string in ENVI header file gr resulting structure samples lines values retrieved fr
16. Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 60 of 73 LSA SAF NV Ito vision on technology Group Name LSA SAF GNC Europe Description LSA SAF Euro region on SAF Europe channel Date position 46 File id position 36 Destination folder myServer LSA SAF EURO Dated folders no issing data log myServer LSA SAF missing missing euro log Duration of storage ONE WEEK Item Name MSG ET Euro Pattern S LSA HDF5 LSASAF MSG ET Euro Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG DSLF Euro Pattern S LSA HDF5 LSASAF MSG DSLF Euro Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG LST Euro Pattern S LSA HDF5 LSASAF MSG LST Euro Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Group Name Description LSA SAF GNC Americas LSA SAF SAme region on SAF Americas channel Date position 46 File id position 36 Destination folder myServer LSA SAF SAME Dated folders no issing data log myServer LSA SAF missing missing same log Duration of storage ONE WEEK Item Name MSG ET SAme Pattern S LSA HDF5 LSASAF MSG ET SAme Process yes Times per day 8 Times to store all Expected segm
17. cee eceesessecececesseseeteceeeeeeeseeseaaes 16 Figure 2 Flows for producing n day composites from sub daily top or daily bottom input products 16 Figure 3 Processing flow without temporal compositing eese nnne nnne nein 17 Figure 4 Secondary ROI preparation flow cccceessssececceecesseseesecececssesseauaeceeecssssseauaeeeeeeuseeesaaaeeeeeeeseeseaaeas 17 Figure 5 Product download page of the LSA SAF web Sit c cccccscsssessssececececsesesssaeeeeeceseesessaeeeeeeeseeseaaees 20 Figure 6 Warning messages informing users that the compositing task may take long to complete 25 Figure 7 Questionmark icon for help left and Advanced button for experienced users right 27 Figure 8 Welcome tab of the MSG Toolbox sese ne nnn nnne nnne ni inna enses inns 28 Figure 9 Regions of Interest tab eese seen eennn sns nns sisi th nass si th assa ssa sa ss seda assa seda sa sss asas 29 Figure 10 Bounding box coordinates sub panel on the Regions of Interest tab c ccccccccssssssssseceeecessesenees 29 Figure 11 Bounding box selection via a map with part of Europe selected eese 30 Figure 12 Pixel size sub panel in the Regions of Interest tab ccccccccccssscceessscecessesceceessssececsssaeceesssaeesesseaees 31 Figure 13 Advanced remapping configuration dialog WINKOW cccceessssecececeesesecnececececeseeseasaeceeecesses
18. more information on available options can be found through e Typing java basic options and java X extended options at the Command Prompt e Searching the web for instance http www oracle com technetwork java javase documentation index html for Oracle JVM When the MSG Toolbox starts up for the first time it will open up the Welcome tab figure below Other main tabs include Regions of Interest see section 5 2 Compositing see section 5 3 and Processing see section 5 4 On the Welcome tab s left hand side the About sub panel provides a short description of the toolbox acknowledgements of the developers and in particular the version number The version number is particularly relevant when communicating on software issues or questions VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 27 of 73 vision on technology amp LSA SAF y vito On the right hand side the Getting started sub panel describes the regular steps in using the MSG Toolbox notably some preparatory reading the definition of ROIs the compositing and monitoring of the processing tasks E Tes Welcome Regions of Interest Compositing Processing i About Getting started Meteosat Click on the question mark icons below to open up the related section of the software user manual LSA SAF Second Main processing flows of the toolbox Generation Introduction t
19. 2400 Mol Belgium Page 44 of 73 amp isa sar 7 vito vision on technology 5 3 2 3 Quality Filtering Quality Filtering Data value range Filter using bitwise quality flag Lower Limit Upper Limit 0 0 Upper Limit 1 0 Data and error limits are expressed in physical unit a Figure 35 Quality filtering sub panel of the Scenario settings window Early on in the processing workflows before the main temporal compositing and remapping steps the main data layer can be optionally filtered on quality This is threefold e The rescaling of the data values themselves imposes a range lower and upper thresholds e A filtering using an upper limit threshold on the error margin uncertainty values e A filtering using the bitwise quality flag information The re scaling is always performed and the corresponding data value range is shown on the left hand side aiming to maintain the data range of the original input products as much as possible The other filters are optional and subject to the availability of such layers in the input product The error margin filtering currently requires an absolute error value to be provided For example the fAPAR product contains the following three layers e FAPAR as main data layer e FAPAR err as layer providing the error margin uncertainty e FAPAR QF as the bitwise quality information layer whereby individual bits indicate if the pixel is land or see af
20. Composite of daily max for Dekad 2013 07 21 2015 06 16 13 43 35 START INFO Composite start date 201307210000 INFO Composite end date 201307310000 INFO Max missing values 3 INFO Max allowed gap size 2 INFO AVG output file D XDataMMSGADemoXOutputXDSSF Dekad avg 20130721 max INFO MAX output file D XDataMMSGXADemoXOutputXDSSF Dekad max 20130721 max INFO SUM output file D XDataMMSGXDemoXOutputXDSSF Dekad sum 20130721 max Figure 42 Example of logging messages showing the output files produced by dekad compositing VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 49 of 73 vision on technology LSA SAF y vito For the further interpretation of the files produced for instance reading the ENVI hdr file to discover the characteristics data range flagged values etc of the produced files please see Chapter 7 5 4 Monitoring processing tasks and progress r MSG Toolbox Welcome Regions of Interest I Compositing Processing 71 9 Id 41 fapar 1 from 2013 07 22 to 2013 07 22 73 Task Id 4 S Id 42 Period 2013 07 22 73 Task name Import product of period 2013 07 22 for region NAfr D Id 43 Import region Euro 100 Id 44 Import region NAfr 100 2015 06 16 14 12 13 START Id 45 Import region SAfr 100 INFO Importing LSA SAF product HDF5 LSASAF MSG FAPAR NAfr 2013072 9 Id 46 Import region SAme 100 INFO Decompressing D
21. PATH str HDRlmax 1 s 501 fun 25 item m 2 short YYYY yy mm dd dd st ttm tty dpm doy short Ly De Ky multi double Ro ib E25 FILE fp strcpy fun ENVI HDR READ Retrieve name of header file from related image file s name img name img hdr 2 Open header file for text read print error if file open fails if fp fopen hdr rt NULL sprintf ERRmess Opening HDR s hdr ERROR fun ERRmess 1 Check first line to see if it is an ENVI header file str 0 0 fgets str HDRlmax fp sprintf ERRmess Line 1 in HDR s should be ENVI hdr if str NULL ERROR fun ERRmess 1 strtrim str 2 if stricmp str ENVI ERROR fun ERRmess 1 strcpy ERRmess mn Read standard IDL ENVI parameters into structure amp trHDR description str fp amp multi 1 0 if strlen str gt sizeof h description 1 str sizeof h gt description 1 0 strcpy h description str only support ENVI standard file type not e g ENVI Classification StrHDR file type str fp amp multi 1 0 strcpy h file type str VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 65 of 73 vision on technology e LSA SAF y vito if strlen h file type 0 strcpy h file type ENVI standard h gt offset atol strHDR header offset str fp amp multi 1 0 if h gt offset gt 0 amp amp
22. When updates of the grids are made available suitable versions will be posted on the LSA SAF website so that users may update them without having to re install the toolbox software completely A land sea mask is derived from GLC2000 resolution of 112 pixels degree beforehand as well and and is similarly provided 3 1 5 Limitations in the flows Since all the proposed processing flows include either remapping temporal compositing or both they cannot provide layers with discrete values such as classified snow cover as output Such flags are thus used only for optional quality filtering of the input products The remapping currently assumes that the input products are provided in the four typical regions NAfr SAfr Euro and SAme and in geostationary view GEOS projection thus preventing the preventing the processing of in particular Fire Radiative Power FRP products There is currently no workflow to produce monthly or yearly composites from ten daily D10 input products These products can however by remapped by the fourth workflow that omits temporal compositing While LSA SAF aims to back process products in order to provide a consistent time series same version of algorithm users may still have products from different versions which often should not be combined into the same temporal composite A check on the product version is in the roadmap for the next version of the MSG Toolbox software 3 1 6 Intermediate data outputs for qu
23. allows to output an additional file with the number of observations used per pixel For daily composites this is the number of time slots with usable observation For periodic composites the number of days in the period with usable observation HINT When calculating periodic composites from input products with sub daily frequency the daily composites are intermediate results Therefore they are stored in the Workspace folder and automatically removed unless Keep Intermediate data checkbox is enabled see below The number of observations used in each daily composite is not provided 5 3 2 6 File formatting File formatting IDL ENVI SPIRITS img GeoTiff tiff ILWIS mpr Figure 38 File formatting sub panel in the Scenario settings window In the File formatting sub panel the user can choose to produce output files in GeoTIFF and ILWIS formats in addition to the standard IDL ENVI format For more information on these formats please see section 7 5 3 2 7 Scenario info Scenario info Name ET daily sum Description Sum of daily ET Figure 39 Scenario Information sub panel in the Scenario settings window VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 47 of 73 y vito vision on technology Using the text field s in the Scenario info sub panel the user can specify a short name and easy to read description for the scenario that is being created or edited When editing an
24. applicable the error margin and bitwise quality flags ENVI files are combined into a single imported file These files will be placed in the mport sub folder of the workspace folder and use the following naming convention input file imported filtered and rescaled files var reg YYYYMMDDhhmm prod img hdr var reg YYYYMMDDhhmm img hdr and if applicable var reg YYYYMMDDhhmm err img hdr and if applicable var reg YYYYMMDDhhmm qflags img hdr Whereby var variable or product identifier code in the filename ALBEDO FARAR reg region one of Euro NAfr SAfr SAme prod HDF5 dataset name for the main data layer err HDF5 dataset name for the error margin layer qflags HDF5 dataset name for the bitwise quality information layer Example input files workspace folder import FAPAR Euro 201201020000 FAPAR img hdr workspace folder import FAPAR Euro 201201020000 FAPAR QF img hdr imported file workspace folder import FAPAR Euro 201201020000 img hdr Remarks e The data type scaling and the flag values of the imported files is determined by layer specified in the selected scenario 6 1 2 Daily compositing step In case of sub daily input products in the daily or n day composites processing flow the imported sub daily files are composited into daily files These files will be placed in the Daily sub folder of the workspace folder Following naming convention will be used input f
25. be started until the user chooses to either Resume this task or Resume all tasks via the respective buttons on the Processing tab see section 5 4 It is however recommended to try running similar tasks via the Add task button first until the scenario is fine tuned to provide satisfactory results Addtask Add paused task Figure 22 Buttons for submitting compositing workflow tasks VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 37 of 73 isa sar y vito vision on technology 5 3 1 Input data folder After defining the necessary scenario s these can thus be re used on input data located in different folders or different time periods specified as days via the nput data folder sub panel Input data folder D Data MSG MET EURO som Search subdirectories 20130402 Ca 20130402 3 Figure 23 Input data folder sub panel on the Compositing tab The selection of the top level folder can be done by either typing in the path into the text field or via the folder browse L button The user can opt to recursively search through sub folders via the checkbox When this sub folder searching is enabled the user will however receive a warning message on potential slow performance and be asked to confirm upon submission of a task 1 Warning N E Recursive folder searching can take a long time to complete or exhaust Java application memory Continue anyway l Yes No
26. daily input products with monthly compositing windows days 01 to end of month Year for n day compositing from sub daily or daily input products with yearly compositing windows 01 January to 31 December None to select the workflow without temporal compositing available to all sub daily daily 10 daily inputs VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 43 of 73 LSA SAF y vito vision on technology 5 3 2 2 Advanced temporal compositing settings Advanced composite settings im r Within same day Day starts from D UTC Max missing 25 2 Max consecutive gap 10 96 rBetween days Max missing 25 Max consecutive gap 2 days Defaults OK f Cancel Figure 34 Advanced temporal compositing dialog window If the selected input layer has sub daily frequency and depending on the selected composite period the dialog will allow to specify the daily compositing settings Within same day sub panel at the top the n day compositing settings Between days sub panel bottom or both For instance for Dekad composites of the sub daily input layer DSSF all the text fields will be available for updates as shown in Figure 34 For Day composites or a daily frequency input layer such as LAI the text fields in the Within same day part will be disabled For the daily temporal compositing the user can specify e The start time within the d
27. erebi Site Search Personal FTP server Password eeEwrs Messages Server Workshops FTP address IP or hostname ex landsaf ipma pt Login Forum Your FTP server login goes here i UU Your password goes here Download Path Description The remote path with final slash ex tmp Static Data amp Tools Special Note Development Status o Dissemination Ordering Method Random slot ordering It allows the user to pick individual random slots for any available Produts LICCENENEAAN Long series ordering Easy way for ordering long time series of Products The user only has to define the date time range and the given products Note List that this ordering method requires human intervention and may take up to a week to fulfil depending on the amount of data ordered Publications Order Delivery Destination LandSAF server user account The order will be placed in the LandSAF server and link will be provided Nevertheless this method has limitations User Support depending on the current server load FAQs Personal FTP server The order will be placed in the configured FTP server user s own FTP server The only limitation is the one imposed by the user s FTP server Helpdesk Contact Proceed User Proto Logout LSA SAF System and EUMETSAT LandSAF UTC Time 17 26 38 Web site developed by Disclaimer Optimized for Mozilla Firefox 3 1024x768 EDI SOFT AAA Figure 5 Product download page
28. finally the button Start Task to check the configuration and launch the secondary ROI preparation workflow that actually produces the grid and mask files VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 28 of 73 LSA SAF y vito vision on technology r n MSG Toolbox 5 Welcome Regions of Interest Compositing Processing Bounding box coordinates Remapping x Advanced Custom ROI 65 5 2 5 B Additional information Spatial Reference System Geographic lat lon WGS 84 Grid settings Name Europe 4km Description Europe at 4 km pixel size B Pixel size Sub nadir pixel size 0 0275 degrees per pixel a Degrees per pixel 0 035714285714 a Add task d Figure 9 Regions of Interest tab 5 2 1 Bounding box In the Bounding box coordinates sub panel the rectangular bounding box coordinates can be defined in decimal degrees with positive values for North and East hemispheres as illustrated below Bounding box coordinates Custom ROI 12 5 Figure 10 Bounding box coordinates sub panel on the Regions of Interest tab The coordinates can be defined in three ways e By selecting a pre defined bounding box from the drop down list e By selecting Custom ROI and entering the coordinates manually in the text fields e By selecting Custom ROI and the Select on Map button to
29. instance cloud or snow occurrence that obscure the measurement and hence decrease the quality of the data value or may even prevent the value from being computed retrieved at all LSA SAF products are typically made available for four specific regions called NAFR northern Africa SAFR southern Africa EURO Europe and SAME South America with fixed sizes in terms of numbers of lines and columns and in the native geostationary projection The default output format is IDL ENVI with some extensions to the text header hdr file to allow exploitation in SPIRITS Additional outputs in GeoTIFF and ILWIS format are available as well These output files all contain a single data layer resulting from the processing of the main variable layer of the inputs with e g optional filtering on quality remapping temporal compositing 3 1 Processing flow schematics The flows below are all illustrated using the following legend i _ Processing step Processing step Script or logical F outside toolbox Separate to be added group 1 scope flow bed rs GE ee ae Directinput Data source Static data Processing step VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 15 of 73 i LSA SAF A vito vision on technology The arrows indicate the logical flow of data from one processing step to the next Configuration files optional input data e g la
30. interactively draw the bounding box on a map see next section The pre defined bounding boxes in the drop down list represent the coordinates for all the countries partly or entirely in the MSG disc hence all of Europe Middle East Africa and parts of South America VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 29 of 73 LsA sar vito vision on technology These bounding boxes are rounded to integer degrees so they may be larger than expected in particular for small islands and city states e g Vatican City Upon selection from the drop down list the corresponding coordinates are copied to the text fields below Switching to Custom ROI via the radio button then allows the user to modify them 5 2 2 Selection of bounding box on a map The Select on Map button on the Bounding box coordinates sub panel opens up a simplified low resolution map that is limited to approximately the coverage of the MSG disc and uses different colours for the various countries to facilitate selection see figure below To select any area on the map simply draw a red rectangle by clicking and dragging the mouse cursor with left mouse button The mouse scroll button can be used to zoom in and out To pan the map drag the mouse while pressing the right mouse button Double clicking resets the map to the central position In case the window is resized the map itself will be scaled while retaining the aspect heigh
31. lines long bands Nbr of layers typically 1 short interleave For multi layer images 1 BSQ 2 BIL 3 BIP short data type l byte 2 short 3 long 4 single prec float short bpp Bytes per pixel respectively 1 2 4 4 for data type 1 2 3 4 short byte order O0 Big Endian 1 Little Endian short flip Flip byte order needed for actual system int64 pixels Total pixels in image samples lines struct MAPINFO mi Regular Map info struct MAPINFO mit Map info for Pseudo Images not used in MSG Toolbox char Vname 201 Meaning of physical value Y VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 64 of 73 LSA SAF Lond Surface Analysis char Vunit 51 double Vlo double Vhi double Vmin double Vmax double Vint double Vslo long date Short days char flags 257 char comment 257 char program 101 char sensor 51 char projection 257 char coordsys 1001 only for file type short classes char cnames unsigned char hi eolorsr 7 vision on technology Units of Y Nlo hi range of significant digital values Nmin max actual limits as observed in image Note Vlo lt Vmin lt Vmax lt Vhi Linear relation Y Vint Vslo V where V digital value Y physical value YYYYMMDD Image registration date or startdate for composites syntheses Periodicity in days 1 7 10 30 60 90 180 360
32. to give a name and a description to the scenario e Save and Cancel buttons in the bottom right corner to save the changes or exit the window without saving 5 3 2 1 Temporal compositing Temporal compositing ET Evapotranspiration 30min ET X Day X Advanced ith Figure 32 Temporal compositing sub panel of the Scenario settings window VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 42 of 73 amp LSA SAF y vito vision on technology On the Temporal compositing sub panel the user can from left to right Select the layer to be processed Define the composite period type depending on the selected layer Use the Advanced button to open the advanced temporal compositing settings for experienced users only The layer selection is done via a drop down list Each entry in the list corresponds to a currently configured and supported layer in a specific input product The entries are formatted as lt product identifier gt lt layer description gt layer identifier Whereby lt product identifier gt is a code identifying a specific type of input LSA SAF product as it is reflected in the file names e g HDF5_ FAPAR_ See the column code in the filename in Table 1 of section 4 3 lt layer description gt is a short description of the main data layer in that product e g for LAI product the main layer contains the Leaf Area Index values layer identifier is the ide
33. types of composites using one of the main workflows that are described in chapter 3 n M EENE s n G SSUES ae ae 008 P Welcome Regions of Interest Compositing processing Input data folder Workspace D Data MSG Jan2014 inns D Workspace MSGToolbox V Search subdirectories V Keep intermediate data 20140128 E 20140128 e B 8 Scenario Output folder Select Scenario X D Data MSG Results_MSGToolbox J a New B Add task Add paused task _ Figure 21 Compositing tab The main configuration to the workflows is done through the definition of processing scenarios These are managed via the Scenario sub panel on the bottom left The other sub panels define the input data folder and date range Input data folder sub panel top left the workspace for storing intermediate results Workspace sub panel top right and the output folder Output folder sub panel bottom right In the bottom right corner two buttons allow the user to add the task at the end of the processing queue The Add task button foresees that the task is scheduled to run when previous tasks that were already on the queue are complete For longer processing tasks in particular for execution outside business hours the task can be added and directly paused via the Add paused task button The task is then queued like a normal task but it will not
34. 011 5 89575 97 56943 6 2 2 238126 1 34992 98 91935 7 3 3 91647 0 51954 99 43889 8 4 4 39651 0 22478 99 66367 9 5 5 19061 0 10806 99 771713 10 6 6 11417 0 06472 99 83645 11 7 7 7697 0 04363 99 88009 12 8 8 5667 0 03213 99 91221 13 9 9 4243 0 02405 99 93626 14 10 10 3144 0 01782 99 95409 15 11 ENTE 2281 0 01293 99 96702 16 12 12 1755 0 00995 99 977697 17 13 13 1302 0 00738 99 98435 18 14 14 936 0 00531 99 98965 C 1 5 pixel the most accurate BIN V Y Npix Npix CUMS 2 2 FLAG 10013429 56 76547 56 76547 4 0 0 6955261 39 42892 96 19439 5 1 1 452741 2 56656 98 76095 6 2 2 113979 0 64614 99 40709 7 3 3 40944 0 23211 99 63920 8 4 4 20313 0 11515 99 75435 9 5 5 12192 0 06912 99 82346 10 6 6 8204 0 04651 99 86997 11 7 7 5781 0 03277 99 90274 12 8 8 4303 0 02439 99 92714 13 9 9 3239 0 01836 99 94550 14 10 10 2468 0 01399 99 95949 15 11 11 1885 0 01069 99 97018 16 12 12 1484 0 00841 99 97859 17 13 13 1096 0 00621 99 98480 18 14 14 861 0 00488 99 98968 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 33 of 73 LSA SAF gt m vito The interpolation resampling creates a mask spatially congruent to the desired output region from a static land sea mask derived from GLC2000 1km resolution The parameter defines the threshold for regarding a pixel as land by default at least 50 of the pixels in the GLC2000 based land sea mask that overlap with the output pixel need to be indicated as a land class To illustr
35. 6 GDAL Geospatial Data Abstraction Library Location in the installation to be included in future Source http www gdal org GDAL is a translator library for raster geospatial data formats GDAL is distributed under an X MIT license reproduced below The OGR Simple Features Library is an C open source library and command line tools providing read and sometimes write access to a variety of vector file formats including ESRI Shapefiles S 57 SDTS PostGIS Oracle Spatial and Mapinfo mid mif and TAB formats OGR and GDAL libraries are provided as an integrated package The GDAL licensing terms are intended to give you permission to do whatever you want with the GDAL source code download modify redistribute as you please including building proprietary commercial software no permission from Frank Warmerdam OSGeo Foundation or anyone else is required Copyright c 2000 Frank Warmerdam Permission is hereby granted free of charge to any person obtaining a copy of this software and associated documentation files the Software to deal in the Software without restriction including without limitation the rights to use copy modify merge publish distribute sublicense and or sell copies of the Software and to permit persons to whom the Software is furnished to do so subject to the following conditions The above copyright notice and this permission notice shall be include in all copies or substantial portions of the Softwa
36. Data MSG Demo Input FAPAR HDFS_LSASAF_MSG_Fi Id 47 Remap to grid Full disc sub nadir size resolution thr INFO Extracting HDFS dataset FAPAR Q Id 48 Conversion to GEOTIff 0 INFO Extracting HDF5 dataset FAPAR OF INFO Converting to FAPAR_NAfr_201307220000 INFO Filter on error margin None INFO Filter on quality Mask 8 Flagged values 8 4 m INFO Converting from n Digital range 0 10000 Scaling 10000 0 Pause Pause All Offset 0 0 RemmeA i INFO Converting to e Resume Digital range 0 10000 Cancel Id 41 Slope 1 0E 4 Intercept 0 0 s INFO Re scaling Id 1 fapar 1 from 2013 07 22 to 2013 07 22 digital output 0 0 1 0 digital input Id 2 Period 2013 07 22 INFO Pixels read 2544861 Id 3 Import region Euro INFO Pixels converted 1855869 etr INFO Pixels filtered Id 5 Import region SAfr INFO on data range 682678 1d 6 Import region SAme INFO on quality 6314 Id 7 Remap to grid Full disc sub nadir size resolution thre INFO Cleaning up Id 8 Conversion to GEOTiff INFO Done cleaning up 37 Id 9 fapar 1 from 2013 07 22 to 2013 07 22 2015 06 16 14 12 14 END 5 8 Id 17 fapar 1 from 2013 07 22 to 2013 07 22 t Id 25 fapar 1 from 2013 07 22 to 2013 07 22 8 Ql Id 33 fapar 1 from 2013 07 22 to 2013 07 22 lt D 7 Remove Remove All Refresh Figure 43 Queue and
37. ERR EOS ECEE FERRE RENE ER e FE PREX RERO Fe Ei FR EE 31 5 2 5 Grid settiligs reete eee eoe REL at Ye Pe EREER FERE Ee e dass ARN RET EY E FER ERA GOV ERREUR FAR FEAT UT E NER ERE PEE FO GET ra RE PRU Ee e a ER ERR EO 35 5 2 6 Launching the ROI preparation Workflow ccccccccccccesssssscecccecsessseesecececesseeseseeecesseseeseseeecessesneaeeeeseseseneea 35 5 3 Producing COMPOSITES siscscsessecsssestesssseseessssesteesssesscssssessesssnesecssesessesssecsecdssesseecssecsecessesdeessecsdecssseessecsseeseces 37 5 3 1 np tdatatoldet uiii ter rette Retorno ER eene ER en ER Fb doves ensveatectieaveasKawcdieed Enana aa a nth 38 JE MEESSDIIgloge tcI ERES 41 5 23 37 WVORKSD ACC hats oisi tesi etae exce atic tacerayeansceh sat ius saipenaseede hav epi inen Ra a UR ARRA RR TERR RBN ORE UU 48 534 Output Folder 5 reatu eo eaae de t ess ede ed ee eg en rasa ea oed ek Re ches dod nex enean eeu ete pelle ehesacdenv ve che t esema Leo be re ER 48 5 4 Monitoring processing tasks and progress c eeeeee ecce ecce eene eene eene eene ettet ennt tenente tenente nennen 50 6 idi 52 6 1 File organization in the workspace eee eee e eere eene eene eene eene enne ennt nennen nnne nennen nennen 52 6 11 Imiportsub flOW sire teet niente ptr innen tarea et tne en vensdobendhddescanenneedrdbasevacuvtandegnestecescedseddedvendead eotederdeas 53 6 1 2 Daily compositin
38. F product 2 111 ece1 eei eee Ier L ee eI eee ee I ep eoo 0 ET eo EE eS EN ERO FOCUS o9 Fee EE EC O0 eS Ee sea EE e eo EUN EE EE 20 4 2 File NAMING CONVENTION eese 21 4 3 Products supported in the toolbOxX 2 1 0 1 Lee De Lees c EIU Ines Heo Un es eeu e ees ees PP CS ov ro Ue eS oe oU SC So rU aee EET 21 4 4 Working with a large number of products eeeeeeeee eee eee ee eee nennen nennen nenne senes ese s essa essa essa esas esas esas ene 23 4 5 Data management SOMW ANG 5 50 ec 00e 0 ooo eo eo 1E 0000 OD HE eI eS EE ER C598 CR EET eS EY ESO 000 10 FEE eS EYE CR C2 FEES es ee e eoe eo rods 25 5 BASICUSAGE s eE E A E E A 27 5 1 cg n 27 5 2 Preparing Regions of Interest 2 cec oic oo rede reco 11 o ee esve e ee YU Ee Co neo ee NEEN o 08 o i NESEN NEEN oo aao eoo ENEN soos 28 5 2 1 Bounding DOX nr a Ue Rd e ERR RREERNER RENE a URN ERU KE REN RRN MER CREAR NEL RN EAE e Ea ENS OR 29 5 2 2 Selection of bounding box on a map ececccessessscecccecseseeeececececeeseseesecececeesesessesececeesesesseseeeceseesesaeseeeeeeeesees 30 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page4 of 73 LSA SAF y vito vision on technology cA MD ECIDI T H 30 5 2 4 Advanced remapplhg tette rin reiecta reb cn s EE ee E E ERE Gee Ren Ed E Fe de
39. MSG DSLF SAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG LST NAfr Pattern S LSA HDF5 LSASAF MSG LST NAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG LST SAfr Pattern S LSA HDF5 LSASAF MSG LST SAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Group Name LSA SAF GNC Global Description LSA SAF Full disc on SAF Global channel Date position 46 File id position 36 Destination folder myServer LSA SAF Global Dated folders no issing data log myServer LSA SAF missing missing global log Duration of storage ONE WEEK VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 62 of 73 vision on technology LSA SAF vito Item Name MSG FRP GRID Pattern S LSA HDF5 LSASAF MSG FTA FRP GRID Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 63 of 73 vision on technology amp LSA SAF y vito Annex 2 Sample source code for reading outputs in IDL ENVI format These below source code samples are provided for illustrative purposes only without guarantee or even compilation or testing In regular C programming language In the below code
40. NGEMENT IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM DAMAGES OR OTHER LIABILITY WHETHER IN AN ACTION OF CONTRACT TORT OR OTHERWISE ARISING FROM OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE 2 4 4 Hdf java Location in the installation MSGToolboxData libs HDF5 Source http www hdfgroup org downloads It is Java library for accessing reading and writing files in the Hierarchical Data Format HDF notably version 5 HDF5 It is provided with a BSD style open source license as specified on http www hdfgroup org products licenses html 2 4 5 Glimpse Location in the installation MSGToolboxData libs Glimpse Source in house developed by VITO NV the developer of the MSG Toolbox The GLobal Image Processing SoftwarE Glimpse is a set of command line driven image processing routines developed since 1990 notably as part of several contracts for the European Commission DG Joint Research Centre s unit on monitoring of agriculture via remote sensing MARS http mars jrc ec europa eu The VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 13 of 73 amp isa sar 7 vito vision on technology MSG Toolbox contains Glimpse programs for grid inversion GRIDinvM exe remapping GRIDmapM exe and mask interpolation INTERPOL exe in particular The GRIDinvM and GRIlDmapM were modified rewritten to serve the toolbox purpose 2 4
41. P vito The start and end day can be entered as days via the text fields or date pickers figure below in the Input data folder sub panel so that the same processing scenario can be easily repeated for different time periods of input data e g per month The specified dates are rounded to the start or end of the day dekad month or year depending on the composite period selected in the scenario Please note that this also implies that for composite period type None the fourth workflow without temporal compositing always a full day of input data is processed It is not possible to limit the processing to a single 15 30min or hourly time slot 20140128 2014 5 Sun Mon Tue Wed Thu Fri Sat 01 121125054 E 5 6 7 amp 9 0 1 03 12 13 14 15 16 17 18 04 19 20 21 22 23 24 25 05 26 27 ZEN 29 30 31 Figure 25 Date picker for entering the start or end date For instance when specifying dates 20130101 and 20130201 with dekad composite periods selected in the scenario four composites will be created one for each dekad in January days 01 10 11 20 and 21 31 and one for the first dekad 01 10 of February When the user specifies a date range of more than one month with sub daily frequency input images the toolbox will warn about potential effect on performance and ask for confirmation upon submission of the processing task as illustrated in the below figure Fr 1 Warning Processing more than one mont
42. SF Euro 20120201 avg img hdr workspace folder daily DSSF Euro 20120201 min img hdr workspace folder daily DSSF Euro 20120201 min img hdr workspace folder daily DSSF Euro 20120201 max img hdr workspace folder daily DSSF Euro 20120201 max img hdr e The first file of a day hhmm and the heads and tails range is determined by the start of the day and the maximum consecutive gap settings specified in the selected scenario 6 1 3 Remapping step The imported files for the different regions Euro NAfr SAfr SAme are remapped to geographic lat lon and combined according to the grid specified in the selected scenario Following naming convention will be used input files remapped files var Euro YYYYMMDDhhmm img hdr var YYYYMMDD img hdr daily products and or var Nafr YYYYMMDDhhmm img hdr and or var SAfr YYYYMMDDhhmm img hdr and or var SAme YYYYMMDDhhmm img hdr sub daily products in var Euro YYYYMMDDhhmm img hdr the no composite and or flow var Nafr YYYYMMDDhhmm img hdr and or var SAfr YYYYMMDDhhmm img hdr and or var SAme YYYYMMDDhhmm img hdr var YYYYMMDDhhmm img hdr composited and or sub daily products var NAfr YYYYMMDD dct img hdr and or var SAfr YYYYMMDD dct img hdr and or var Euro YYYYMMDD dct img hdr var SAme YYYYMMDD dct img hdr var YYYYMMDD dct img hdr VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 56 of 73
43. UMENE eee 10 1 3 CO Na Cts eee M S 10 2 INSTALLATION AND REQUIREMENTS jasssssessssssscssscscscauavavacsesessseseueececesaenensueueueseueuenavavavavaraeaeaeaeaesesevecesesaeaeaeasasaeses 11 2 1 hieu e 11 2 2 DPRTIBrRENnRNnOCIEDgEML 11 2 3 Upgrading from a previous version coc eoo eor eoo eoo eoo eoo ee eee esee eo eee eee eoe o eee e eee osea so eo eee esee eese e seus 12 2 4 Software packages included in the installation eeeeeeee esee eee eene eene eene nennen nennen nennen nennen nenne 12 2 4 1 PDF Renderer ireira ra eren Dae reor iter redeo ae ae eren Doer Pee egy edi Pe Tad re eo RH YER Deere doe RR ce E Y 12 2 4 2 jCalendar and part of jGOOGIES ccccccesccceessscecsssseeecseseceessececssssececsssseceesseseceesaeescesseeeesaesecseaaeeeeseneeees 13 pP CENT O 13 DAA MAIVA e a seen RE ERRATERNIA edis el arbi VERRE TAREA E EUR BADEN UTERINE NB RN ORARE MER 13 PS EEE CI METTI AERE NE 13 2 4 6 GDAL Geospatial Data Abstraction Library esesssseeseseeesee eene nnne ennt entran nnne enn 14 3 MAIN PROCESSING WORKFLOWS sscscesesssecsssssnscesecscnentesstuasacccdeeanstesnassecuecsens
44. aeeecsesaeeseseaaess 46 Figure 37 Composite files sub panel in the Scenario settings window essere 47 Figure 38 File formatting sub panel in the Scenario settings window cesses nnns 47 Figure 39 Scenario Information sub panel in the Scenario settings window cesses 47 Figure 40 Warning when storing intermediate data files esses nnns 48 Figure 41 Output folder sub panel on the Compositing tab eese ennemis 48 Figure 42 Example of logging messages showing the output files produced by dekad compositing 49 Figure 43 Queue and progress of submitted tasks on the Processing tab ccccesssssececeeeesssesssseeeeecesseseaees 50 Figure 44 Example of the workspace file and folder organization cessere 53 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 7 of 73 Em LSA SAF y m vito vision on technology List of tables Table 1 Overview of supported LSA SAF input ProdUcts ccccccccccecsssssssececececesseeneaeceeeceseesecnsaeeeeseeseeseaaees 22 Table 2 Bitwise g ality filtering sess coire rrt teet rese Enn ev eoe eve ehe venne eee Neve en veh ev een ud 46 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 8 of 73 LSA sar 7 vito vision on technology 1 Background and scope In late June 2011 the EUMETSAT Council gave the green light for t
45. ager LSASAF txt and select it when running the Data Manager either interactively or by adding it as command line parameter automatic start Note This file will be automatically overwritten when you run the Data Manager You can make edits to this file when the program is not running but it is no use to change the layout of this file or add your own comments Lines starting with are comments and default values reset by the software Uncomment them delete to make them tak ffect m itle Data Manager sample configuration for LSA SAF Source folder pc received Unmatched files folder po received unmatched Copy files no Autostart delay 10 Columns 3 Group Name LSA SAF FTP Description LSA SAF via FTP Date position 46 File id position 36 Destination folder myServer LSA SAF Dated folders no issing data log myServer LSA SAF missing missing euro log Duration of storage ONE WEEK Item Name MSG HDF5 Pattern HDF5 LSASAF MSG Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name EPS MetOp Pattern HDF5 LSASAF EPS Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG ASCII Pattern ASCII LSASAF MSG Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all VITO NV
46. ality control As the de compression incl file copy and HDF5 to ENVI conversion steps are preparatory as they do not affect the data values their intermediate outputs are not made available to the user In other words the de compression HDF5 to ENVI conversion and Quality filter amp rescale steps can be considered together as an atomic import sub flow repeated for each relevant input file And first intermediate output files are only available at the end of this sub flow effectively the output of the Quality filter amp rescale step To allow users to verify and visualize the processing performed the outputs of this import sub flow daily compositing Remapping and n day compositing steps may be optionally retained on disk in simple flat binary ENVI with text header extensions format See chapters 5 3 3 and 6 for more information 3 1 7 Optional processing steps The de compression un zipping of input LSA SAF products bz2 is only performed when the inputs files were not already decompressed prior to their ingestion in the MSG Toolbox When not filtering on the error margin or the bitwise quality flag layers the Quality Filter amp Rescale step falls back to re scaling the ENVI input file implying a filter on the range of pixel values for the main variable layer and adjusting the intermediate ENVI header file The temporal compositing steps both daily and n day are optional When the output for
47. and offset row offset character 16 str buffer integer parameter nshow 5 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 71 of 73 V Ito vision on technology Get Input arguments from command line The given input arguments 2 3 4 and 5 should be present in the hdr respective file of the given img filename nargs iargc call getarg 1 img filename call getarg 2 str buffer read str buffer nrows call getarg 3 str buffer read str buffer ncols if nargs 3 then call getarg 2 str buffer read str buffer nbands else nbands 1 end if if nargs gt 4 then call getarg 5 str_ buffer read str buffer skipbytes else skipbytes 0 end if write new_line a Input arguments new line a amp amp filename trim img filename new line a amp amp lines nrows new line a amp amp samples ncols new line a amp amp bands nbands new line a amp amp header offset skipbytes new line a new line a amp amp The following arguments are immutable for this program new line a amp amp data type 3 Long 32 bit signed integer new line a amp amp interleave bsq new line a amp amp byte order 0 new line a allocate img data 0 ncols i 0 nrows 1 0 nbands 1 if skipbytes 0 then open newunit zf unit file trim img filename status old action rea
48. at 6 UTC When the daily composite starts at 6 UTC keeping the three hours maximum gap and thus also the length of the lead and trail data of the above example the input data considered effectively ranges from the 3 UTC time slot on day D included to the last time slot before 9 UTC e g 08 00 for hourly 08 30 for 30 minutes 08 45 for 15 minutes on day D 1 Setting the starting time delayed start or time shift is particular useful e For composites of specific products such as Land Surface Temperature e To better capture the diurnal cycles for users in the South American area or with significant time zone difference with UTC VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 40 of 73 vision on technology c Example of time period considered for n day dekad compositing 0 UTC first day 23 59 UTC last of dekad N day of dekad N Lead up to Composite of Dekad N E 1 gap 2 days max cons gap in dekad 2 gap e cays Dekad N 1 days Dekad N 1 Figure 29 Example relevant time period for dekad compositing For n day compositing such as dekad 10 daily monthly or yearly compositing a maximum consecutive gap can also be specified as a number of missing days not hours Again the lead and trail data used for interpolating the missing first and last time slots is limited in amount to one gap In the above example all input products are considered within the effective time range from the la
49. ate the effect of this threshold setting let s first examine the original land sea mask derived from GLC2000 figure below Figure 15 Land sea mask from GLC2000 for entire MSG disc left and zoomed in to Italy Sicily right The following figures illustrate what happens when the setting is changed to 1096 as extreme low value figure below left kept at its default of 5096 middle or set to an extreme high value of 9096 right A low setting will of course increase the number of pixels identified as land in the created land sea mask and thus also the number of pixels considered for the remapping in the main workflows PPP Figure 16 Examples of mask interpolation threshold 1096 land left 5096 default mid and 9096 right with overview of Italy top and zoom to Southern Italy Sicily with sub nadir pixel size VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 34 of 73 vision on technology 5 2 5 Grid settings The Grid settings sub panel of the Regions of Interest tab allows to specify a unique name and short description that identify the ROI that is computed After the ROI computation task is submitted via the Add Task button the newly defined ROI will appear in the drop down list of ROIs that appears in the compositing scenario definition window see section 5 3 2 and can then be used for starting compositing tasks Note however that the ROI computation task albeit very short in duration i
50. ay which defaults to O UTC to 23 59 UTC see also section 5 3 1 e The maximum number of allowed missing data values for any given pixel expressed as of available time slots within that day These data gaps may be spread throughout the day e The maximum consecutive gap also expressed as of time slots within the day for which the pixel does not have a valid value For the n day composites the Between days sub panel likewise allows to define a of maximum missing days spread across the n day period or the number of consecutive days not expressed as percentage Whenever the 96 missing or the maximum consecutive gap are exceeded the corresponding pixel is flagged as missing data Smaller gaps are interpolated where needed Products that are constructed by the provider as daily integrations of other products notably the daily integrated DSLF DSSF DIDSLF and DIDSSF and daily ET DMET products can contain the following data layers e Max_nslots_missing maximum consecutive gap encountered during production e Missing values percent total percentage of missing values encountered during production e and Weight missing values percent sum of cosine of the solar zenith angle thereby giving higher weight to daytime missing values These layers are at present not considered in the MSG Toolbox and therefore unrelated to the above Advanced temporal compositing settings VITO NV Remote Sensing Dept Boeretang 200
51. constrained to the simple Geographic Lat Lon spatial reference system Key information in the header file includes e The data type e The number of lines and samples bands is always 1 e The mapping information magic point pixel size e The Values tag that details the variable physical unit value ranges and slope intercept for the conversion from digital to physical values e The Flags tag that indicates pixels which contain specific values for e g missing data Sample code written in ANSI C and Fortran programming languages for reading the images is provided in Annex 2 The sample code in C in addition parses the textual header information hdr file Sample code in the object oriented Java programming language can be provided on request 7 2 Additional GeoTIFF and ILWIS formats In addition to the default IDL ENVI output format the MSG Toolbox can provide copies of the output files converted into the GeoTIFF and or ILWIS formats using the standard gdal_translate utility of GDAL GeoTIFF is an extension of the standard TIFF image format adding in the geo information geolocation spatial reference system etc in the form of metadata tags or attributes GeoTIFF format is widely used in GIS and remote sensing software The toolbox adds the filename extension tiff and filename prefix GTIFF_ to the GeoTIFF formatted output files it produces For example the output file GTIFF DSSF 20130408 avg tiff is a daily composite of the shortwave f
52. d amp recl data kind nrows ncols access direct iostat io stat do band 0 nbands 1 band_offset band ncols nrows read f unit rec i band offset iostat io stat img data band end do else open newunit f unit file trim img filename status old action read amp recl data kind access direct iostat io stat do band 0 nbands 1 band_offset band ncols nrows do lin 0 nrows 1 row offset lin ncols do col 0 ncols 1 read f unit rec i skipbytestband offsettrow offsettcol img data col lin band end do end do end do endif amp amp iostat io stat VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 72 of 73 LSA SAF y vito vision on technology Print corner pixels Output do band 0 nbands 1 write Band band new_line a do lin 0 nshow 1 write img data 0 nshow 1 lin band img data ncols 5 ncols 1 1lin band end do write new line a new line a d do linznrows nshow nrows 1 write img data 0 nshow 1 lin band img data ncols 5 ncols 1 1lin band end do end do deallocate img data end Program img2ascci VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 73 of 73
53. e histograms up to 99 99 of pixels for different values each time combining the four regions NAFR SAFR SAME and EURO into one grid 75 with pixel size of around 4km 0 035714285714 degrees per pixels to be exact The Y column hereby represents the error in distance between estimated and true value expressed in kilometre Pixels in space or over ocean are flagged with digital value V 2 As you can see the accuracy is in any case high with between 98 5 and 99 5 of pixels with an error smaller than or equal to the nominal resolution 3km VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 32 of 73 vision on technology LSA SAF 7 vito a Half pixel 1 2 the fastest sub pixel precision BIN V Y Npix Npix CUM 2 2 FLAG 10010255 56 74748 56 74748 4 0 0 4530018 25 68037 82 42785 5 1 1 2205364 12 50206 94 92991 6 2 2 468849 2 65787 97 587719 7 3 3 198835 1 12718 98 71497 8 4 4 97989 0 55549 99 27046 9 5 5 51680 0 29297 99 56344 10 6 6 27671 0 15687 99 72030 11 7 7 14666 0 08314 99 80344 12 8 8 8987 0 05095 99 85439 13 9 9 6382 0 03618 99 89057 14 10 10 4848 0 02748 99 91805 15 11 TL 3761 0 02132 99 93937 16 12 12 2965 0 01681 99 95618 17 13 13 2183 0 01238 99 96855 18 14 14 1598 0 00906 99 97761 19 15 15 1204 0 00683 99 98444 20 16 16 954 0 00541 99 98985 b 1 3 pixel the default BIN V Y Npix Npix CUM 2 2 FLAG 10011914 56 75688 56 75688 4 0 0 6159323 34 91680 91 67368 5 1 1 1040
54. e section 5 3 3 e Starting compositing tasks preferably with a small set of input data e g a short time period so they can be easily repeated with minor changes e Visualizing the intermediate data to troubleshoot the workflow step by step For this visualization it is of course important to first have a basic understanding of the workflows the various processing steps involved and the organization of the intermediate data in the Workspace 6 1 File organization in the workspace Intermediate results of each processing step except the last one are temporarily written to the specific folders called the workspace They are cleaned up automatically unless the user specifically opts to keep them To this end the user configures a single top level workspace folder Within this top level folder the toolbox determines the names of the intermediate files and organizes them into subfolders per variable and processing step that yields intermediate output data in order to e limit the number of files stored in each folder for performance reasons e and to facilitate experienced users in quickly locate the intermediate files for inspection When working with sub daily frequency input LSA SAF products the import sub flow in particular works with high amounts of files up to hundreds per day For this reason the related import folder in the workspace is further sub divided per day The daily images that result from import of daily LSA SAF prod
55. ed in the software installation It describes the software components how they work together and how users can configure and use the toolbox 1 1 References and acknowledgements Please see the web site of the LSA SAF http landsaf ipma pt for more details on LSA SAF work and products or the EUMETSAT web site for more details on the related SAF networks From its inception the MSG Toolbox was inspired by and re uses parts of other existing software executables graphical user interface code design concepts etc in particular from e The LSA SAF product processing chains developed and operated by VITO on behalf of and delivering its output to the European Commission DG Joint Research Centre JRC Unit Monitoring of Agriculture through Remote Sensing MARS http mars jrc ec europa eu http www marsop info These chains include a number of modules executables from the Global Image Processing Environment or Glimpse library developed by VITO e The Time Series analysis software called Software for Processing and Interpretation of Remotely sensed Image Time Series or SPIRITS in short that VITO developed for the MARS unit at the JRC e VITO s VGTExtract utility for integrating VEGETATION products into various commonly used GIS and remote sensing software And to less extent the Georeferencer utility developed by Ricardo Da Silva and LSA SAF and the free ILWIS software with its GEONETCast toolbox extension avai
56. electing the layer to be composited e YYYYMMDD HHmm the nominal date and or time as copied from the input file e Extension file name extension depending on the format For ENVI format a binary data file img and a text header file hdr are provided Example FAPAR 20130722 img For the workflow that produces daily composites from sub daily frequency input products var YYYYMMDD dct extension Whereby the additional dct signifies the type of daily composite and is one of avg min max sum or cnt for number of observations Example DSLF 20130401 avg img For the workflows producing periodic n day composites the file names are composed as follows var per pct YYYYMMDD for daily frequency input files var per pct YYYYMMDD dct for sub daily frequency input files Whereby e per composite period type which is one of Dekad Month or Year e pct periodic n day composite type one of avg min max sum or cnt for number of observations Example the file DSSF Dekad avg 20130721 max img is the 10 daily average of the daily maximum DSSF values Output files in GeoTIFF format are additionally prefixed with GTIFF and have the tiff as filename extension ILWIS output files are all prefixed with LWIS and have their standard extensions mpr grf csy mp When in doubt the user can check the task logging messages for the output file naming as in the below example for a dekad processing step Task Id 103 Task name
57. else strcpy gr datum str b 1 gr gt dx a gr ome b gr omx gr xmin CRmin a b gr xmax CRmin samples a b b 1 gr dy a gr mr b gr my gr ymax CRmin a b gr ymin CRmin lines a b VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 70 of 73 cuum y vito vision on technology FH IK KK kk RRA ke ke IAAI II RR ko ke ke ke kk kk kk Sk AAAI ke ko kk ko kk eee kekekekekeke Extract from main routine EOKCKCKCRCKCKCKkCKCkCKCk Ck k Ck k k k k Ck CkCkCk Ck ck ck ck k ck ck ck ck ck kCKCKCKCkCkCk Ck ck k ck ck ck ck ck ck ck ck k K K Ck Ck Ck Ck ck ck ck ck ck ck ok ck ck ko ko k kc k ck ck ck sk sk ke ke ke ke e ke eek void main int argc char argv char fileIn MAX PATH short BPPi long line column struct ENVIHDR headerIn hm ho FILE filePointerIn unsigned char bufferin Read header file for 1 layer accepting any datatype byte offset or endian ness envi hdr read fileIn amp headerIn 1 0 1 4 1 1 Open binary image file if filePointerIn fopen fileIn rb NULL sprintf ERRmess Opening input image ss fileIn ERROR PRE PROCESSING ERRmess 1 Skip the byte offset indicated in the header fseek filePointerIn headerIn offset SEEK SET read the image file line by line for line 0 line headerIn lines line fread bufferIn BPPi headerIn samples filePointerIn
58. enaaes 31 Figure 14 Difference between 596 default and 3096 most severe threshold on nominal effective eco Ui eTeToie suot T 32 Figure 15 Land sea mask from GLC2000 for entire MSG disc left and zoomed in to Italy Sicily right 34 Figure 16 Examples of mask interpolation threshold 1096 land left 5096 default mid and 9096 right with overview of Italy top and zoom to Southern Italy Sicily with sub nadir pixel size 34 Figure 17 Grid settings sub panel of the Regions of Interest tab sessesssssesee eene nnns 35 Figure 18 Build grid dialog showing progress of the ROI preparation workflow eeeeeeeneneenene 35 Figure 19 Information dialog upon successful completion of the ROI flow sees 36 Figure 20 Confirmation to over write an existing ROI cccscssscccccessssesseaececececeesesaaeseceeeseeseasaeeeeeceseesenaeas 36 Figure 21 Compositing tab ssssssssessseseeeee eene nennen nenne nass enne ne tinae sas sessi setis a assa ases sss si ga assa sss sse serias 37 Figure 22 Buttons for submitting compositing workflow tasks nennen 37 Figure 23 Input data folder sub panel on the Compositing tab eeessesssssseeee eene enne 38 Figure 24 Warning when recursively searching the input data folder 38 Figure 25 Date picker for entering the start or end date ccccccccccscssssssseceeeceseeseses
59. ents 1 Segments to store Item Name all MSG DSLF SAme Pattern S LSA HDF5 LSASAF MSG DSLF SAme Process yes Times per day 8 Times to store all Expected segments 1 Segments to store Item Name all MSG LST SAme Pattern S LSA HDF5 LSASAF MSG LST SAme Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgiu m Page 61 of 73 vision on technology LSA SAF 7 vito Group Name LSA SAF GNC Africa Description LSA SAF NAfr and SAfr regions on SAF Africa channel Date position 46 File id position 36 Destination folder myServer LSA SAF NAfr Dated folders no issing data log myServer LSA SAF missing missing nafr log Duration of storage ONE WEEK Item Name MSG ET NAfr Pattern S LSA HDF5 LSASAF MSG ET NAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG ET SAfr Pattern S LSA HDF5 LSASAF MSG ET SAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG DSLF NAfr Pattern S LSA HDF5 LSASAF MSG DSLF NAfr Process yes Times per day 8 Times to store all Expected segments 1 Segments to store all Item Name MSG DSLF SAfr Pattern S LSA HDF5 LSASAF
60. existing scenario the Name cannot be changed The provided name will be translated e g removing whitespace to a name suitable for a file on the hard disk as the scenario is saved as an XML file The saved XML scenario files are automatically loaded when the MSG Toolbox starts up 5 3 3 Workspace Workspace D Workspace MSGToolbox J Keep intermediate data The Workspace is specified as a top level folder via the text field or the folder browse button It represents the folder where all the intermediate results are stored Those are automatically deleted during the workflow execution unless the Keep intermediate data checkbox is checked As keeping the intermediate data around is only suitable for troubleshooting with typically short processing time and limited amount of input data the user will receive a warning in case this checkbox is checked upon submission of the task The files and folders in the workspace are organized by the toolbox typically in sub folders per processing step and then further per day For more details on the organization of the workspace please refer to Chapter 6 on Troubleshooting O Warning t Keeping intermediate results can store a scc ee volume of data on disk Continue anyway l Yes Figure 40 Warning when storing intermediate data files 5 3 4 Output Folder Output folder D Data MSG Results_MSGToolbox mun Figure 41 Output folder sub panel on the Compositing
61. f in HDR s f h gt Vlo h gt Vhi hdr ERROR fun ERRmess 1 h gt Vlo f strsplit str 4 item f atof item fl min f h gt Vlo f2 max f h gt Vhi if num diff fl h gt Vlo 0 000001 num diff f2 h gt Vhi 0 000001 sprintf ERRmess VALUE Vhi f beyond range f f in HDR s f h 2SVlo h Vhi hdr ERROR fun ERRmess 1 h Vhi f elements 5 amp 6 value range actually used or fall back to significant range strsplit str 5 item h gt Vmin atof item strsplit str 6 item h gt Vmax atof item if h gt Vlo gt h gt Vmin h gt Vmin gt h gt Vmax h gt Vmax gt h gt Vhi printf An WARNING Bad sequence of VALUES in s Vlo g Vmin g Vmax g Vhi g n n hdr h gt Vlo h Vmin h gt Vmax h gt Vhi VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 67 of 73 LSA SAF V Ito vision on technology element 7 amp 8 Strsplit str 7 strsplit str 8 free item H Date optional h gt date atol strHDR date Days optional e g h gt days atoi strHDR days if h gt date date test h date 1 4 Flags optional StrHDR flags Sur fp str sizeof h gt flags strcpy h flags strHDR sensor str str fp if strlen str gt str sizeof h gt sensor fi Sensor optional strcpy h gt sensor s
62. fected by clouds or snow etc As indicated by a small text at the bottom the data value range and the threshold for the error margin filtering are to be specified in physical values not digital numbers expressed in the given unit The above figure provides the example of FAPAR which is dimensionless as indicated with the unit As marked in the Product User Manual for the LAI FAPAR and FVC products VEGA suite the uncertainty should be limited to 0 2 for FVC and FAPAR and 1 5 for LAI beyond which use should be restricted Regarding the filtering on the bitwise quality flag the currently implemented filtering is provided in the following table Please note that unprocessed pixels e g marked as Ocean or Space Algorithm failed are always filtered either by the data range filter or the quality flag filter When troubleshooting via the logging panel see 5 4 itis thus important to consider the number of pixels filtered by either filtered as well as the number of pixels converted remaining to assess the impact of the filtering VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 45 of 73 won s LSA SAF y m VIto vision on technology Product Layer Filter removes MET ET Poor quality decimal values 581 645 709 800 DMET ET None ALBEDO any daily amp 10 Snow daily broadband and spectral DSSF DSSF Cloud filled cloud contaminated snow DIDSSF DSSF None DSLF DSLF Cl
63. flows e g daily 10 daily or monthly different data layers e Try out the scenarios on a limited time period and verify results For such try out it can be useful to keep the intermediate results on disk e Now you are ready to start producing composites regularly or try out larger time periods larger sets of input products A command line batch mode will be added in a later release to further facilitate and automate the production of similar composites at regular time intervals e g every day in near real time HINT Throughout the entire toolbox e the questionmark icon figure below left opens up the relevant section or chapter in this manual e and Advanced buttons figure below right are intended only for experienced users only m Advanced Figure 7 Questionmark icon for help left and Advanced button for experienced users right 5 1 Startup To start up the MSG Toolbox double click on the main Java archive jar file in the top level folder of the installation called MSGToolbox jar This starts the application with default Java Virtual Machine JVM settings Alternatively a simple Batch file bat file can be created as per the provided example to act as an application launcher This offers the advantage of setting specific JVM settings such the minimum and maximum available memory respectively through the Xms and Xmx options While the available options depend on the provider of the JVM
64. g Step ise uote ee eR ae da eee E bake Ead e chicane EAR Rae tie en ken Ee Rd aA ee a Y EA RE VER CIR Tea R an EE MAR ER PERLE eR 55 6 1 3 Remapping Step prre er pe na rn vo sa putain a deut pe dn Vo Mun pear RR jiu v sais Va ea Eae oe dU gene n 56 6 2 Viewing intermediate results eene ere nen nenne ne nnn na annua naa nen nana nana nega ana na gan aaa aga ana eaa NESEN SEBNO ESEON 57 EMUVUEI0i v T 58 7 1 3cpnablBibiieme H m 58 7 2 Additional GeoTIFF and ILWIS formats 02 cciscesecesessscatscersvesnnsccesieresscesenedcdnsessrevensoqscesdesesevetensessessvereseces 58 ANNEXES c 60 Annex 1 Example configuration for the GEONETCast Data Manager software eeeeeeeeeeeeeeeee 60 Annex 2 Sample source code for reading outputs in IDL ENVI format eeeeeeeee eene eene eene nennen nnne 64 In regular C programming language ccccccccccesssssssececccscsesesaeeeeececeesesaeseeecsceeeaeseeeescsesseeesesececeesesaeeeeeesseeseaeaeees 64 In Fortran90 programming langllage onei cer e trennen a etuer ann gg ee e R e eee an EEEE ENE e aiK ie 71 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 5 of 73 LSA sar vito vision on technology List of figures Figure 1 Main processing flow from sub daily input to daily composite
65. g step called N day compositing to compute the n day values from daily ones immediately following the remapping e the ingestion of daily frequency input LSA SAF products e g Leaf Area Index therefore omitting the daily compositing step bottom figure The resulting composites over dekads periods of 10 days months or years called n day or periodic composites comprise the minimum maximum average and or sum again optionally accompanied by the number of observations By convention the dekads are defined as days 1 10 11 20 and 21 end of month 3 1 3 Processing flow without temporal compositing Repeat per input file Sub daily or x Sub daily or Daily LSA SAF lp Download amp Decompress HDF5 to ENVI EN Quality filter EIN Remap MEN Format gt daily remapped Data Management conversion amp rescale conversion bd products data ROI preparation Figure 3 Processing flow without temporal compositing In this fourth flow the temporal compositing steps both daily and n day are removed altogether This flow works on a file per file basis and can thus ingest sub daily daily and ten daily D10 frequency input products 3 1 4 Secondary ROI preparation flow In all of the above main processing flows the remapping step uses grids and masks prepared by the same secondary flow called ROI preparation depicted here It is
66. h of sub daily frequency input data requires substantial memory disk storage and processing time Continue anyway Yes Figure 26 Warning when processing more than one month of sub daily input products in one go The below timelines help to illustrate how the relevant time period is determined for a regular daily compositing b daily compositing with starting time and c n day dekad in this example compositing a Example of time period considered for regular daily compositing 0 UTC 23 59 UTC Lead up to Composite of Day D up 1 gap 3h max consecutive gap in day gap Day D 1 3h Day D 1 Figure 27 Example relevant time period for regular daily compositing VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 39 of 73 EB ah LSA SAF VIto vision on technology By default the compositing for day D is defined from 0 UTC to 23 59 UTC and thus considers the time slots from 0 UTC included up to and including e The 23 00 UTC time slot for hourly frequency products e The 23 30 UTC time slot for products with 30 minutes frequency e Orthe 23 45 UTC time slot for products with 15 minutes frequency The subsequent O UTC slot is considered as part of the next day D 1 The maximum consecutive gap setting is used to determine how many consecutive data values time slots may be missing for a given pixel 3 hours in the above diagram It is configured as a percentage of a
67. he SAF Europe SAF Americas SAF Africa and SAF Global channels on EUMETCast Depending on the reception beam e g C band reception in Africa or Americas Ku band in Europe Northern Africa users may or may not be able to receive all of these channels The sample configuration further defines one input folder for retrieved LSA SAF products and one output folder per group with one week of data being stored archived in each group It handles only a few types of LSA SAF products ET LST etc but can be easily extended by replicating the Items in the respective groups For more details on the Data Manager configuration please see the installation manual provided on the above web site The latest version of the Data Manager 2 0 1 is updated to allow an automatic restart rather than an interactive one which is convenient to restart after e g computer reboot VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 26 of 73 isa sar 7 vito vision on technology 5 Basic usage The expected usage of the toolbox includes the following steps with further details in the sections below e First prepare the Regions of Interest ROI This is to be repeated for each study area It can be useful to try different pixel sizes spatial resolutions or levels of detail and for advanced users remapping settings e Once satisfied with the ROI that have been created define one or more scenarios for the compositing work
68. he Satellite Application Facilities SAF network to receive funding for and thus enter the second part of the Continuous Development and Operations Phase CDOP 2 This phase brings continuity to the existing and operational SAF products as well as a wide range of further developments One of those SAFs is the Land Surface Analysis SAF LSA SAF or L SAF headed by the Portuguese Institute for Sea and Atmosphere IPMA formerly known as the Portuguese Meteorological Institute Soon after VITO signed the contract to participate in and contribute to the LSA SAF in its CDOP 2 phase This effectively marked the first time VITO joined EUMETSAT s SAF network VITO s contributions to LSA SAF are two fold operational production and delivery of 10 daily vegetation indicators based on MetOp AVHRR and a user tool to aid exploitation of LSA SAF products called the MSG Toolbox For the preparatory work on the MSG Toolbox VITO received further funding from the Belgian Science Policy Office BELSPO through its PRODEX Programme with the European Space Agency ESA This preparatory work the first step in the toolbox development includes the analysis of requirements comparison of existing software solutions that may contribute to the toolbox and the initial software design i e chosen concept structure To ensure later take up in the LSA SAF context this initial work was reviewed by IPMA This document is the user manual of the MSG Toolbox and is includ
69. iles daily composite files var reg YYYYMMDDhhmm img hdr var reg YYYYMMDD dct img hdr collection of all files DDhhmm relevant for the day to be composited Whereby var variable or product identifier code in the filename ALBEDO FARAR reg region one of Euro NAfr SAfr SAme dct daily composite type avg min max sum Note that the number of observations cnt is not provided Example input files workspace folder import DSSF Euro 201201312200 img hdr workspace folder import DSSF Euro 201201312230 img hdr workspace folder import DSSF Euro 201201312300 img hdr heads from workspace folder import DSSF Euro 201201312330 img hdr previous day workspace folder import DSSF Euro 201202010000 img hdr workspace folder import DSSF Euro 201202010030 img hdr VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 55 of 73 y mm Vito vision on technology tails from next day daily composites Remarks workspace folder import DSSF Euro 201202010130 img hdr workspace folder import DSSF Euro 201202012330 img hdr workspace folder import DSSF Euro 201202020000 img hdr workspace folder import DSSF Euro 201202020030 img hdr workspace folder import DSSF Euro 201202020100 img hdr workspace folder import DSSF Euro 201202020130 img hdr workspace folder daily DSSF Euro 20120201 avg img hdr workspace folder daily DS
70. image visualization with colours legends titles logos and so on When satisfied the quicklook template can be saved and re used to create maps for larger time series of files e g all intermediate results of different days VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 57 of 73 isa sar vito vision on technology 7 Output formats This chapter briefly describes the various output formats that are provided extended IDL ENVI format GeoTIFF ILWIS and their usage 7 1 Default IDL ENVI format The default output format is compatible to IDL ENVI s standard file format that consists of a flat binary data file img and a plain text header file hdr For further exploitation in Glimpse and SPIRITS software developed by VITO for the European Commission s Joint Research Centre some additional information has been added to the header For more details please see the User Manual extensive description and or the Tutorial summary description of the SPIRITS software and ENVI header format available on http spirits jrc ec europa eu The IDL ENVI web site http www exelisvis com docs ENVIHeaderFiles html describes the standard IDL ENVI format without the Glimpse SPIRITS extensions Note that the MSG Toolbox and hence this manual does not use all aspects of the format For instance the classification keywords class names etc are beyond the scope of the MSG Toolbox and the map info is
71. img hdr or flags var reg YYYYMMDDhhmm h5 Whereby var variable or product identifier code in the filename ALBEDO FARAR reg region one of Euro NAfr SAfr SAme prod HDF5 dataset name for the main data layer err HDF5 dataset name for the error margin layer qflags HDF5 dataset name for the bitwise quality information layer Example input file workspace folder import ALBEDO Euro 201310010000 h5 extracted workspace folder import ALBEDO Euro 201310010000 AL BB BH img hdr files workspace folder import ALBEDO Euro 201310010000 AL BB BH ERR img hdr workspace folder import ALBEDO Euro 201310010000 Q Flag img hdr Remarks e The data type of the ENVI files will be selected automatically to match that of the extracted dataset as close as possible e ENVI does not support all HDF5 data types Example HDF5 supports a signed byte type ENVI does not a 16 bit integer will be used instead ENVI data type 2 e Not all HDF5 and or ENVI types are implemented Following ENVI types are supported o 1 8 bit byte 2 16 bit signed integer 3 32 bit signed long integer 4 32 bit floating point 5 64 bit double precision floating point 12 16 bit unsigned integer 13 32 bit unsigned long integer 14 64 bit signed long integer OO0o00000 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 54 of 73 isa sar vito vision on technology The extracted main data layer and if
72. intended to be run first for each study area and thus independently of the available input LSA SAF products Irregular geolocation Grid inversion grids Inverted geolocation grids amp mask Land sea Nak mask from i GLC2000 interpolation Figure 4 Secondary ROI preparation flow Through respectively the grid inversion and mask interpolation resampling steps static input grids and land sea mask are processed into inverted grids and a mask that are spatially congruent to the user defined region of interest ROI in terms of spatial extent bounding box pixel size spatial resolution and spatial reference system currently fixed to geographic lat lon with WGS84 This ROI preparation flow is started independently from the main flows and only executed when the user defines another target ROI i e study area or resolution It thus allows the prepared ROI specific grids and masks to be re used for subsequent runs of the main processing workflow s For example the same grids can be re used in subsequent processing of composites of different product types Leaf Area Index Fire Risk Map and or temporal intervals days dekads months over the same study area VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 17 of 73 vision on technology The irregular geo location grids were downloaded from LSA SAF website and prepared for use in the toolbox in advance see section 2 2
73. is installed And if it is installed in a protected system path for instance in a subfolder of c program files or c program files x86 administrator privileges may be required for both the installation and for running the software To avoid this install MSG Toolbox in an un protected path such as c MMSGToolbox making sure to avoid spaces in folder names Beyond the set of input products potentially large in volume and or number of files the MSG Toolbox can write a very large number of temporary files to the user configurable workspace folder By default these are automatically removed unless the Keep intermediate results checkbox is checked on the Compositing window see Chapter 3 for details It is therefore recommended to foresee significant free hard disk space e g 1GB for the workspace 2 2 Installation The current version of MSG Toolbox can be installed via self extracting archive MSGToolbox exe The following files and folders are included in the main folder e The main program in the form of a Java archive MSGToolbox jar e Arelated example batch routine bat file for launching the application e The LSA SAF program icon iconLSASAF ico e This user manual e The release notes e The legal terms and conditions VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 11 of 73 isa sar vito vision on technology The sub folder MSGToolboxData contains e Pre defined bounding boxes fo
74. ites based on the 30 day synthesis products ALBEDO D10 LAI D10 FAPAR D10 FVC D10 that are delivered with 10 daily sliding window intervals 4 4 Working with a large number of products The relatively course spatial resolution of MSG derived products means that the file size per product is modest The high temporal frequency the main advantage of using a geostationary satellite as source can however make for a considerable number of files that need to be stored ingested and processed and a considerable total disk space volume occupied Of course this depends on the type of application with for instance near real time monitoring typically requiring limited amounts of data e g the most recent plus long term statistics and other applications requiring longer time series The amount of required LSA SAF products can easily amount to hundreds or even thousands files Just imagine an application that requires spatial coverage of the full MSG disc four regions using a product with high temporal frequency e g every 15 minutes while producing composites over large time spans e g yearly composite but for instance also daily composites for several months in one go As the MSG Toolbox needs to produce per input product several intermediate files for each of the processing steps in particular when the user asks to keep those files around for troubleshooting using the Keep Intermediate data checkbox on the Compositing tab it is clear
75. ixel in the output region of interest regular grid Resamples the land sea mask making it spatially congruent with the output region of interest VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 19 of 73 vision on technology LSA SAF y vito 4 Retrieval and management of input products 4 1 Retrieval of L A SAF products LSA SAF products can be retrieved both via the web FTP and via the EUMETCast satellite broadcast For web access navigate to the LSA SAF web site http landsaf ipma pt register and login The Download option should then appear in the menu on the left and allow ordering of specific time slots periods of the various types of products Bulk orders of large quantities are possible as well however only with manual intervention by IPMA staff as indicated on the web site An automatic dissemination can be provided as well The data can then be accessed in a routinely updated FTP folder FTP client software such as the free FileZilla with graphic user interface or GNU wget see here and here and LFTP for automatic command line based downloading are freely available IRFACE ANALYSIS es PPLICATIONS FACILITY 2 LSA sar do rinm Dissemination Method Home Random slot ordering Overview Long series ordering Links Order Delivery Destination Dissemination Method Automatic Dissemination Contacts Address t Site Map LandSAF Server user account
76. ke a long time to complete 1 Warning T Keeping intermediate results can store a big volume of data on disk T Recursive folder searching can take a long time to complete or exhaust Java application memory E Processing more than one month of sub daily frequency input data requires substantial memory disk storage and processing time Continue anyway Yes No Figure 6 Warning messages informing users that the compositing task may take long to complete 4 5 Data management software The ILWIS and its various toolbox extensions for instance for dealing with EUMETCast data so called GEONETCast toolbox includes a software that helps to automate data management This software the GEONETCast Data Manager can be downloaded and installed separately from the 52North Earth Observation Community web site which also provides more a manual for its installation http 52north org communities earth observation reception stations data manager software VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 25 of 73 LSA sar 7 vito vision on technology A sample configuration to be saved in plain text file hence easy to edit provided in Annex 1 defines e One group LSA SAF FTP for managing LSA SAF products downloaded via the website FTP i e without the S LSA filename prefix e Different groups e g LSA SAF GNC Europe for managing the LSA SAF products received via t
77. lable from University Twente faculty ITC and 52North VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 9 of 73 LSA sar 7 vito vision on technology 1 2 Structure of this document Following this first introductory chapter chapter 2 focuses on the requirements and procedure for installing the MSG Toolbox software Chapter 3 details the main processing capabilities and workflows essential to the understanding of how the toolbox works Chapter 4 provides information on the products from LSA SAF that are used as inputs to the MSG Toolbox Chapter 5 describes how the toolbox can be configured and used to process LSA SAF products Chapter 6 provides tips and techniques for troubleshooting and improving toolbox results Finally Chapter 7 provides more details on the formats of the files produced by the toolbox and how these can be further read or analysed in GIS and Remote Sensing software Some information is provided in Annexes for convenience 1 3 Contact For technical assistance in using MSG Toolbox software please contact VITO s helpdesk via email to helpdeskticket 2 vgt vito be preferably or by calling 32 0 14 336855 For questions related to the characteristics and retrieval of LSA SAF products please contact the LSA SAF Helpdesk service at helpdesk landsaf ipma pt or call 35 1218447098 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 10 of 73 isa sar 7
78. loffset ok sprintf ERRmess OFFSET 1d should be 0 in HDR s h offset hdr ERROR fun ERRmess 1 h gt samples atol strHDR samples r Str fp multi 1 1 if h gt samples lt 1 sprintf ERRmess Samples 1d in HDR s h gt samples hdr ERROR fun ERRmess 1 h gt lines atol strHDR lines Str fp amp multi 1 1 if h gt lines lt 1 sprintf ERRmess Lines ld in HDR s h gt lines hdr ERROR fun ERRmess 1 h gt bands atol strHDR bands Str fp emulti 1 1 if h gt bands lt 1 sprintf ERRmess Bands ld in HDR s h gt bands hdr ERROR fun ERRmess 1 h gt pixels int64 h samples h gt lines if h bands gt 1 amp amp d3 ok sprintf ERRmess Bands ld should be 1 in HDR s h bands hdr ERROR fun ERRmess 1 strHDR interleave str fp amp multi 1 h gt bands gt 1 if strlen str 0 strcpy str bsq default band sequential bsq switch sgn stricmp str bip case 1 h interleave 2 break jy GEL case 0 h gt interleave 3 break bip case 1 h interleave 1 break bsq i atoi strHDR data type str fp amp multi 1 1 if i 1 i 4 sprintf ERRmess Data type d beyond allowed range 1 4 in HDR s i hor ERROR fun ERRmess 1 if i DT min i DT max sprintf ERRmess Data type d beyond range td d in HDR s i DT min
79. lux DSSF containing the average value for the 8 of April 2013 It is the GeoTIFF equivalent of the DSSF 20130408 avg img and hdr files in IDL ENVI format VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 58 of 73 S isa sar 7 vito vision on technology The Integrated Land and Water Information System or ILWIS is a GIS and remote sensing software that was initially developed by ITC currently a faculty of the University of Twente Since a few years it became ILWIS Open an open source software driven by 52North s ILWIS community with popular extensions on data access incl LSA SAF product support developed in the EO community Output files in ILWIS format are actually composed of a group of files varying only by the filename extension For example the file ILWIS DSSF 20130401 max mpr is the main file This small text file contains references to the other related files ILWIS DSSF 20130401 max mp the binary file containing the actual pixel values and ILWIS DSSF 20130401 max grf describing the geographic mapping which in turn further refers to the file IIWIS DSSF 20130401 max csy describing the spatial reference system VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 59 of 73 vision on technology z LSA SAF y m vito Annexes Annex1 Example configuration for the GEONETCast Data Manager software Save the below configuration in a plain text file e g DataMan
80. mat is ENVI format with extensions for use in SPIRITS software the format conversion step does not perform any operation except moving the resulting files from the workspace to the final output folder VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 18 of 73 Lond Surface Analysis LSA SAF y m vito vision on technology 3 2 Overview ofthe processing steps Processing step Decompress HDF5 to ENVI conversion Quality filter amp rescale Daily compositing Remap Format conversion Grid inversion Mask interpolation Main function s Removes the bzip2 compression from input bz2 files Converts the files to ENVI as internal working format of the MSG Toolbox Also extracts the HDF5 file attributes Optionally filters the data layer based on the associated bitwise quality flag layer or error margin layer Re scales the main layer of input data which implies the use of a pre configured data value range Combines time slots within a day into a daily minimum maximum or average Remaps to geographic lat lon and spatially assembles the NAFR SAFR SAME and or EURO regions with optional filtering on spatial resolution Masking with land sea mask is also part of this step Translates the default ENVI format outputs to additional output format s for subsequent file read import in a specific Remote Sensing GIS software Inverts the fixed geolocation grids to provide column record lookups for each p
81. mp doy 1 164d bytes VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 68 of 73 LSA SAF y vito vision on technology J F F K K R KK I kk AAA ke ke ke ko A K KK K K k K K K A k A K K ke k A k IIA II A ko kk II k k Kk k k k ke StrHDR Search and retrieve the value dest of an item src in an ENVI header file Values can be either scalar multi 0 typically on a single line Header file syntax src dest Or multi element multi 1 possibly multi line Header file syntax src vl v2 v3 Parameters Fptr pointer to the open header file src keyword to search for case insensitive dest retrieved value string rew first seek to start of file 0 1 fatal 0 for optional keyword returns empty string if not found 1 for mandatory error if missing multi indicates if the retrieved value is single line 0 or multi line 1 KKK KKK kCkCkCkCkck ck ck ok ck ck kk kc kokck ok ck kk ke ke ke ko kk RAI ke ko kk KAA I I I A A I x f char strHDR char srco char dest FILE fptr short multi short rew short fatal char record HDRlmax 1 ss 501 p short qu d if rew rewind fptr strcpy ss src strtrim ss 2 ss copy of source but trimmed 1 strlen ss scan file while feof fptr record 0 0 fgets record HDRlmax fptr strtrim record 2 if strnicmp record ss 1 0 line with keyword src found p strstr record if p
82. nd sea mask for quality filter and optional e g intermediate output data are not shown for simplicity 3 1 1 Main processing flow producing daily composites from sub daily input products Repeat per day Repeat per input region NAFR SAFR Repeat per input file Sub daily LSA Download amp y P Daily SAF products Data Management l compositing Format Output daily u conversion composite HDF5 to ENVI Quality filter Decompress i ampas conversion amp rescale v gt Remap ROI preparation Figure 1 Main processing flow from sub daily input to daily composite After optional decompression and conversion to the internal ENVI like file format of each relevant file the main data layer is re scaled implying a filter on data value range and optionally filtered using thresholds on the error margin layer and the bitwise quality flag information where these are available in the input products These resulting imported filtered values still per time slot 15 30 minutes or hours are fed into the daily compositing to derive the daily minimum maximum average and or sum optionally accompanied by the number of observations used Each daily composite image is then remapped to Geographic lat lon using a separately prepared region of interest ROI And finally the forma
83. nescaransesssecuasscerneseesessnesseteasssteeceuestevearie 15 3 1 Processing flow SCHEMATICS cisccccsscssccssscssccssesccsssscsccsssscscssscccessaescsesscesccessscsecessssecsessscsssessssecsescssscsessess 15 3 1 1 Main processing flow producing daily composites from sub daily input products ss 16 3 1 2 n day composites from sub daily and daily input ProdUcts ccccccccccesssssscececececsesesaeceeececsestsaeeeeeeseseaees 16 3 1 3 Processing flow without temporal COMPOSITING cccesccccecsessssscecececeesesessesececeeseessesececessesesaeseeseseeeeees 17 3 1 4 Secondary ROI preparation flOW cccccccccccccssssssecccecsesessesecececseseseeseeececeesesesseseeeceesesesaeseeeceesesesaeseeseseeegeea 17 SUS Lim itatiONs in the tlOWS y 18 3 1 6 Intermediate data outputs for quality CONtIOL cece cessssecececsesesseseceeeceeseessesecececeesesaesecscesseeeaeseeeeeseeeea 18 3 1 7 Optional processing Steps o roni enero nra sa peak Pao en eee ERR RR eo sduueseaedeseaenaguunaesSaasuuobasaundsnecastwouncaencenserseaes 18 3 2 Overview of the processing Steps iciecciecccsdcdedcdedccacdcadcdecccadsdedccedcledscedsdesscedsledscsdsderccedsdedscedstesscedesedscsdvess 19 4 RETRIEVAL AND MANAGEMENT OF INPUT PRODUCTS eee en nnn natn anna nnn nata uana sana saam Ra IER RA RE RR RE Ra sR sm Rasa uS nna 20 4 1 RetrieValiof ESA SA
84. ntifier code of the dataset within the HDF5 file that contains the data Depending on the selected layer the scenario settings window will be updated as follows The Quality Filtering sub panel will reflect the proper data value range and unit as well as the presence absence of an error margin uncertainty value layer and bitwise quality layer in the same product For daily frequency input products the Day compositing is disabled see Figure 33 which in turn affects the content of the Advanced temporal compositing dialog and the possibility to calculate daily minima maximum average and sum values in the Composite files sub panel For the 30 day synthesis products delivered with 10 daily sliding window intervals ALBEDO D10 LAI D10 FAPAR D10 FVC D10 the temporal compositing drop down selection is fixed to None and unusable the Advanced temporal compositing dialog is inaccessible and the selections on the Composite files sub panel are disabled Figure 33 Drop down selection of composite period with Day option disabled The second drop down list selects one of the compositing periods and thus one of the corresponding workflows It contains the following entries Day for the workflow that produces daily composites from sub daily inputs Dekad for n day compositing from sub daily or daily input products with 10 day dekadal compositing windows days 01 10 11 20 and 21 to end of month Month for n day compositing from sub daily or
85. o extension at all are considered to be HDF5 data files that were already decompressed prior to ingestion in the MSG Toolbox VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 53 of 73 isa sar vito vision on technology In case the user selected the option to keep the intermediate data these files will be copied from the input data folder to the mport sub folder of the workspace folder with the following naming convention input file copied file HDF5 LSAF MSG var reg YYYYMMDDhhmm var reg YYYYMMDDhhmm h5 HDF5 LSAF MSG var reg YYYYMMDDhhmm h5 var reg YYYYMMDDhhmm h5 Var reg same as above Example input file data folder HDF5 LSASAF MSG FAPAR SAfr 201206240000 copied file workspace folder import FAPAR SAfr 201206240000 h5 Following the above optional decompression and file copy the the main data layer product layer and if applicable the error margin and bitwise quality flags layers HDF5 datasets are extracted from the HDF5 file and converted to the internal ENVI formatted files These files will be placed in the mport sub folder of the workspace folder and use the following naming convention input file dataset extracted dataset files HDF5 LSAF MSG var reg YYYYMMDDhhmm product var reg YYYYMMDDhhmm prod img hdr or error var reg YYYYMMDDhhmm err img hdr HDF5 LSAF MSG var reg YYYYMMDDhhmm h5 quality var reg YYYYMMDDhhmm qdflags
86. o how the MSG toolbox processes LSA SAF a i Toolbox gesdocis Version 1 0 Input products An overview of the currently supported products and how B The MSG Toolbox brings together a set of software utilities to help to retrieve and manage them with the further processing and analysis of Earth Observation products that are operationally produced and disseminated by the Land Surface Step 1 Preparing Regions of Interest Analysis Satellite Application Facility LSA SAF as part of the Prepare the grids and land sea mask for remapping This is 8 EUMETSAT SAF network to be repeated for each studv area Its main functions include daily and n day temporal compositing Step 2 Producing composites spatial subsetting and remapping while combining the four LSA SAF Define a scenario and start producing the desired 8 product regions NAFR SAFR EURO and SAME and filtering on composites via one of the workflows provided product quality layers BEN i Step 3 Monitoring processing tasks and progress a Follow the progress of tasks that were submitted to the queue andtheir logging output The MSG Toolbox is developed by VITO NV Belgium as contribution to the LSA SAF The preparatory work for this first version of the Troubleshooting toolbox received further funding from the Belgian Science Policy Office through its PRODEX programme with the European Space Agency a Output formats vito E d
87. of the LSA SAF web site For access via the EUMETCast satellite broadcast and after setup of the receiving station antenna etc it suffices to register for LSA SAF products via the EUMETSAT EO Portal and enabling the proper channels on your receiving station European Ku band receivers can receive all three channels SAF Africa SAF Europe or SAF Americas EUMETCast users in Africa and Americas C band beams can only receive the respective SAF Africa and SAF Americas channels offering fewer products For the latest updates on the dissemination of LSA SAF products you can consult the EUMETSAT Product Navigator This also provides sample file names typical file sizes and delivery frequencies Simply use the Extended search option and select LSA SAF as provider VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 20 of 73 isa sar vito vision on technology 4 2 File naming convention LSA SAF products use the following file naming convention For online delivered products HDF5 LSASAF SENSOR TYPE REGION YYYYMMDDhhmnmY extension Whereby e The literal characters HDF5_LSASAF indicate the data format HDF5 and source LSA SAF e SENSOR is an identifier representing the sensor e g MSG for Meteosat Second Generation or MO1 MO 2 for MetOp 1 and 2 e TYPE is an identifier for the type of product which references the purpose or the main data variable e g Leaf Area Index product provides LAI data val
88. olated values in between actually measured values The Advanced remapping configuration however allows to filter pixels on their size hence avoiding the coarsest resolution pixels near the edge of the MSG disc For example a pixel size of approximately 4km for most of Africa and 5km for most of Europe is fine 5 2 4 Advanced remapping HINT This window is intended for experienced users only The default configuration should suffice for common applications The Advanced button in the Remapping sub panel on the ROI tab opens up the Advanced remapping settings dialog illustrated in Figure 13 4 Advanced remapping settings To mask the coarsest resolution pixels near the edge of disc Allow pixel when fraction nominal effective resolution gt 5 Accuracy of grid inversion 1 5 pixel is most accurate Predsionis 1 3 pixel 1 to 5 Land sea mask interpolation Allow pixel when atleast 50 of area is land Figure 13 Advanced remapping configuration dialog window This dialog is further divided into three 3 sub panels The first two configuring the masking of coarse resolution pixels and the accuracy of the grid inversion are related to the grid inversion step in the ROI preparation workflow see section 3 1 4 The latter has to do with the land sea mask interpolation step in the same workflow VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 31 of 73
89. om samples lines in header file grid size kkkkkkkkkk kkkkkkkkkkk kkkkk kkkkk kk kkkkk kkkkk kkkkk kkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkkk void map find char s struct MAPINFO gr long samples long lines char str HDRlmaxt1 short i NO N double a b strcpy gr name if strlen s 1 return N strsplit s 0 str if N lt 7 sprintf ERRmess Incomplete MAP INFO s s ERROR LIBfunc MAP FIND ERRmess 1 SEESPLIE S7 9 Lr Str if strlen str gt sizeof gr name 1 str sizeof gr name 1 0 strcpy gr name str strsplit s 2 Str gr mc atof str strsplit s 3 str gr gt mr atof str strsplit s 4 str gr gt mx atof str strsplit s 5 str gr gt my atof str Strsplit s 6 Str gr dx atof str strsplit s 7 str gr dy atof str NO 7 gr gt zone 0 strcpy gr gt ns if stricmp gr gt name UTM NO 9 strsplit s 8 str gr gt zone atoi str strsplit s 9 Str strcpy gr 2ns str if qr gt zone lt 1 strlen gr ns sprintf ERRmess Incomplete MAP INFO for UTM zone d NS s gr zone gr gt ns ERROR LIBfunc MAP FIND ERRmess 1 strcpy gr datum strcpy gr units for i 0 i lt N NO i strsplit s NOLL str if strnicmp str units 5 0 strsplit str 2 gr units
90. oud filled and cloud undefined uncertain Algorithm quality below nominal DIDSLF DSLF None LST LST Corrupted satellite image suspect inputs Snow ice contaminated Algorithm quality below nominal FAPAR FAPAR Traces of inland water LAI LAI Traces of inland water FVC FVC Traces of inland water FRM FWI Mode 0 Initialization FRM DSR Mode 0 Initialization Table 2 Bitwise quality filtering For the ALBEDO D10 FAPAR D10 LAI D10 and FVC D10 products the same filtering configuration is used as for their daily equivalents HINT e The configuration of the bitwise quality filtering proved particularly tricky and is thus likely to change in future versions of the software Make sure to check results carefully 5 3 2 4 Remapping to Region of Interest Remapping to Region of Interest Europe sub nadir Europe with sub nadir pixel size Restrict imported regions Euro NAfr SAfr SAme B to vj Figure 36 Remapping to Region of Interest sub panel In the Remapping to Region of Interest sub panel a simple drop down list allows to select the Region of Interest ROI that was previously prepared via the ROI preparation workflow The entries on the list are formatted as name description using the name and description provided on the Regions of Interest ROI tab When the ROI is created it stores the required input regions NAfr SAfr Euro and or SAme that could be used to make up the selected bounding box When a
91. progress of submitted tasks on the Processing tab After clicking on the Add task or Add paused task buttons on the Compositing tab a task is added to a waiting list called the task queue This queue is depicted as a hierarchy on the top left of the Processing tab with usual tree navigation buttons Lower levels in the tree are for sub tasks child tasks Green circles for successful tasks orange ones for tasks with warnings red for tasks with errors yellow for tasks waiting to be started and white for paused tasks Warnings and errors are escalated to higher level parent or ancestor tasks up to the top level Even when the majority of sub tasks was successful a single warning or error will hence make the parent task get warning or error status orange or red circle Error status takes precedence over warning status when both occur The buttons just below allow the user to remove tasks pause them and resume then when they are paused The Remove Pause and Resume buttons each affect only a single selected top level task or the currently active one if none are selected The Remove All Pause All and Resume All affect all top level tasks in the queue The Cancel button sometimes with a reference to the Task Id number of the currently selected task will cancel this task These buttons are followed by a blue progress bar indicating task completion in of the currently selected top level task VITO NV Remote Sensing Dept Boe
92. r countries and economic regions covered by MSG Africa Europe part of South America Middle East in the roi folder e Pre defined list of data types in the outputType folder e Pre defined list of output formats in the outputFormat folder e Various images and logos used by the toolbox in the images folder e External software libraries and programs in the ibs folder e Static input grids in the ibs GLIMPSE REFGRIDS The MSGToolboxData sub folder furthermore contains three empty folders e Temp to be used as default workspace can be overridden by the user e Scenarios for storing processing scenarios configuration files e Grids for storing the outputs of the ROI preparations inverted grids and land sea masks for use in the remapping A set of ancillary static data files is to be downloaded separately see above link so that users can update them independently of the software in case of infrequent changes These data files include e SA SAF s input latitude and longitude data with 4 byte precision for the centre of each pixel and filenames following the pattern RRRR_XXX img where RRRR LSA SAF region NAFR SAFR SAME and EURO and XXX being LON for longitude and LAT for latitude e Static IGBP mask named RRRR MSK img with RRRR again referring to the region NAFR e GLC2000 derived land sea mask for the entire MSG disc with the filename GLC2K LS img These ancillary data files are available in ENVI format ready for
93. re THE SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM DAMAGES OR OTHER LIABILITY WHETHER IN AN ACTION OF CONTRACT TORT OR OTHERWISE ARISING FROM OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE A more complete collection of license terms for GDAL OGR and subcomponents will be included within the software distribution in the LICENSE TXT file VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 14 of 73 vision on technology 3 Main processing workflows The MSG Toolbox operates the following four main image processing workflows using operationally provided LSA SAF products as input e A flow for the production of daily composites from sub daily i e 15 min 30 min hourly frequency input products This is the primary objective of the toolbox hence the most important workflow e Two workflows to produce n daily or periodic 10 daily or dekadal monthly and yearly composites from respectively sub daily and daily frequency input products e A fourth flow to remap input products on file per file basis without any temporal compositing operation The above four processing flows all share a remapping step that uses grids and a mask as input These are prepa
94. red separately in a specific workflow called ROI preparation flow see section 3 1 4 The grids and mask prepared by this separate flow can be re used in any subsequent run of four main workflows The processing flows do not mix different types of input LSA SAF products radiation flux DSSF vegetation cover FVC etc This chapter provides an overview of these workflows and the individual processing steps that they are composed of Knowledge of these flows and components is vital to understanding how the MSG Toolbox works and what it aims to accomplish The initial step in all of the flows being the retrieval and management of LSA SAF products is not included in the toolbox software As these are important pre requisites to exploiting the toolbox properly good practices and examples of this are provided in the next chapter For the proper understanding some prior knowledge of the LSA SAF products and their formatting is recommended While this is described on the LSA SAF website http landsaf ipma pt and the product documentation provided there it is good to know that LSA SAF products are organized as Bzip2 compressed archives of data files in HDF5 format Each data file in turn includes one or more likely several data layers For example one layer with the main variable one layer with uncertainty estimates and one layer with bitwise encoded quality information often called quality flags The latter uses specific bits to indicate for
95. retang 200 2400 Mol Belgium Page 50 of 73 LSA SAF y vito vision on technology HINT The toolbox will wait for a running task or sub task to finish before cancelling removing or otherwise disrupting it When the task is completed it moves to the bottom left sub panel that provides a similar view The attached Remove and Remove All buttons can then remove those finished tasks that are no longer of interest The colours of the circles indicate the status black means finished successfully red means failed orange is for warnings A cross over the circle indicates a cancelled task Both for the on going tasks selected from the queue top left as well as for the finished ones bottom left the right panel will provide detailed messages on progress These are particularly useful for troubleshooting and can be easily copy pasted via the Clipboard to a text file or emailed when asking for support VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 51 of 73 isa sar vito vision on technology 6 Troubleshooting Beyond the regular monitoring of the progress and status of processing tasks see section 5 4 whereby the detailed information messages may be copied and emailed for support the more experienced user can also investigate the problem in more detail This is done by e Enabling the storage of intermediate results via the Keep intermediate data checkbox on the Compositing tab se
96. roducts from the first days of 2014 also up to the number of days specified as maximum gap VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 24 of 73 vision on technology Keeping in mind that this type of application will likely want to produce composites for several years which makes it tempting to store the full multi year archive in one spot it is not recommended to have the toolbox search through so many files A better approach could be to work with the following structure product type year 2010 year 2011 year 2012 year 2013 month January day 1 month February Duplicate of 1 gap at end of 2012 e g full month of December Duplicate of 1 gap at start 2014 This allows you to use the year 2013 folder as starting point for the search top level input data folder thereby avoiding searching the multi year archive irrelevant years 2010 2011 As such studies are not regularly repeated the lead trail data from previous and subsequent year can be duplicated manually once off and the archive only contains the clean folder structure i e without the duplicates The toolbox will warn users when they enable the search of sub directories when the intermediate results are kept and when they start up the compositing of large periods of sub daily frequency input products Users can still continue the processing after this warning however with the risk that it can ta
97. s added to the task queue It needs to have completed successfully before the new grids and land sea mask are ready for use in compositing tasks so be careful when disrupting the task order in the queue via e g removal of tasks Grid settings Name Europe 4km Description Europe at 4 km pixel size Figure 17 Grid settings sub panel of the Regions of Interest tab The specified Name is translated into a machine usable file name e g allowing alphanumeric characters hyphens and underscores but no whitespace 5 2 6 Launching the ROI preparation workflow After clicking the Start Task button on the ROI tab the ROI preparation workflow see 3 1 4 is launched It is run in its own processing thread not added to the task queue section 5 4 While this ROI preparation is performed a simple dialog window shows the progress amp Build grid MSG Toolbox Preparing Region of Interest Please Wait This can take some time 72 ly Figure 18 Build grid dialog showing progress of the ROI preparation workflow Upon completion of the ROI preparation flow an information dialog is shown summarizing the start end of the different steps involved as well as the name description and location of an XML file This XML file contains the grid settings name description etc and in turn refers to the location of the grid and mask files Users should not edit the XML file directly it is better to re launch the workflow
98. seceeeessseseasseeeeseeseesenaeas 39 Figure 26 Warning when processing more than one month of sub daily input products in one go 39 Figure 27 Example relevant time period for regular daily compositing ccseseessscecececeesesesneaeeeeecesseseaees 39 Figure 28 Example relevant time period for daily compositing with start at 6 UTC eseeeeeee 40 Figure 29 Example relevant time period for dekad compositing sess 41 Figure 30 Scenario sub panel on Compositing tab The previously saved scenario with the name ET daily sum and description Sum of daily ET is selected sessi 41 Figure 31 Scenario settings WINKOW cessccccccecessessssececececsssesnsaeceeesscsscesaeaecesecscsseaaaaeeeseesseesasaeeeeseessesenaaas 42 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 6 of 73 LSA sar 7 vito vision on technology Figure 32 Temporal compositing sub panel of the Scenario settings window csse 42 Figure 33 Drop down selection of composite period with Day option disabled usse 43 Figure 34 Advanced temporal compositing dialog window cessere nns 44 Figure 35 Quality filtering sub panel of the Scenario settings window cesses 45 Figure 36 Remapping to Region of Interest sub panel cccccccccesssececssscesecsescececseseeceeseseeeecees
99. selection is then made in the drop down the checkboxes on the right reflect these required regions The user can then de select the checkboxes for the regions that were not downloaded to reduce the number of warnings errors produced This is particularly useful for study areas near the borders between regions considering the remapping VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 46 of 73 vision on technology 5 3 2 5 Composite files Composite files Daily composites E 7 Periodic composites E IE Nbr observations 7 Figure 37 Composite files sub panel in the Scenario settings window In this sub panel it can be specified if the toolbox needs to calculate minimum maximum average or sum values per day in the row of checkboxes labelled daily composites or in the n day compositing period periodic composites row The checkboxes for the daily composites are disabled in case the selected input layer is not provided with sub daily 15 30 min hourly frequency All these checkboxes are disabled in case of input products delivered at 10 daily D10 intervals or when the compositing periodicity is set to None workflow without temporal compositing When checkboxes on both rows are enabled the toolbox will calculate all possible combinations For instance it is possible to compute minimum and maximum daily values and average those daily minima and maxima across a month The Nbr observations checkbox
100. st two days of dekad N 1 included up to the first two days of dekad N 1 HINTS e When producing yearly composites with a one year maximum consecutive gap the toolbox will effectively search through three 3 years of input data and may thus produce a lot of missing input file warnings when those are absent e nnear real time applications it is normal for the trail data to be absent The toolbox will nevertheless search for it 5 3 2 Scenario Using the Scenario sub panel on the Compositing tab the user can define a new processing scenario New button and edit Edit button or delete Delete button a previously saved scenario Via the drop down list any previously saved scenarios can be selected through their name and description Scenario ET daily sum Sum of daily ET New Edit Delete Figure 30 Scenario sub panel on Compositing tab The previously saved scenario with the name ET daily sum and description Sum of daily ET is selected When adding a new scenario or editing an existing one the scenario settings or scenario definition window will appear When adding a new scenario the name will be changeable When editing an existing one the name is greyed out to indicate it cannot be changed as illustrated in Figure 31 VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 41 of 73 y m Vito vision on technology
101. t width ratio a gt Select ROI West 14 81 Figure 11 Bounding box selection via a map with part of Europe selected The information bar at the bottom informs the user of the currently selected coordinates Selected North West South East and the coordinate the mouse cursor is currently point to Position X and Y The OK button confirms the selection and returns to the Regions of Interest tab with the coordinates copied to the text fields for Custom ROI The Cancel button returns to the same tab however without storing the coordinates that were selected on the map 5 2 3 Pixelsize In the Pixel size sub panel the user can choose between using the sub nadir pixel size 0 0275 degrees per pixel which is the finest level of spatial detail highest resolution achievable or entering a custom pixel size expressed as decimal degrees per pixel VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 30 of 73 ELSA SAF 7 vito vision on technology Pixel size Sub nadir pixel size 0 0275 degrees per pixel Q9 Degrees per pixel 0 035714285714 Figure 12 Pixel size sub panel in the Regions of Interest tab Due to the curvature of the Earth the pixel size increases resolution becomes coarser fewer spatial detail towards the edges of the MSG disc Fixing the pixel size to the sub nadir one for those areas is thus not recommended as it will introduce a lot of interp
102. t conversion step adds optional outputs in other formats e g GeoTIFF 3 1 2 n day composites from sub daily and daily input products Repeat per n day compositing period Repeat per day in n day compositing period Repeat per input region NAFR SAFR Repeat per input file d N 3 Sub dailyLSA Download amp HDF5 to ENVI Quality filter Daily Lj N day Format Outputn day SAF products Data Management Decompress conversion amp rescale compositing Remap compositing conversion gt composite VJ i j X V ERR ERES i Y V ROI preparation Repeat per n day compositing period Repeat per day in n day compositing period Repeat input region NAFR SAFR MOI MELDE i n AE i Daily LSA SAF Download amp v HDF5 to ENVI Quality filter N day Format Outputn day products Data Management Decompress conversion amp rescale Remap compositing conversion gt composite b 1 t ROI preparation Figure 2 Flows for producing n day composites from sub daily top or daily bottom input products VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 16 of 73 i LSA SAF A vito vision on technology Compared to the above first workflow there are two additions e a processin
103. tab The Output folder sub panel is used to define the output folder via the text field or folder browse button In this folder the output files from the main compositing workflows are stored The file names vary depending on the workflow performed and its configuration such as the option to include the format conversion step Each produced image holds a single data layer This corresponds to the main data layer selected in the compositing scenario configuration s drop down list as output by the processing quality filtered remapped etc Moreover the value range is largely kept the same as the input products as much as possible VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 48 of 73 vision on technology EB ah LSA SAF y m VIto Exceptions are e The additional image for the number of observations used which contains the number of time slots in the day up to 96 for products with 15min frequency or number of days in the compositing window up to 366 for yearly e The daily or periodic sum composites which have a larger data range or even a change in data type e g conversion from integer to floating point pixel values For the flow performing only remapping and no temporal compositing the files are named var YYYYMMDDHHmm extension for sub daily inputs var YYYYMMDD extension for daily inputs Whereby e var the product identifier code variable as listed in Table 1 or in the drop down list s
104. that the software needs to organize those files carefully See section 6 1 for more information on how this is accomplished But before the input LSA SAF products can be ingested and processed the toolbox needs to locate the relevant files on the hard disk first Proper management of the input data including organization in folders and archiving of un used data is an clear MUST HAVE to keep the toolbox performance reasonable in terms of time duration hard disk space and memory consumption Computers organize files on a hard disk using a so called filesystem e g NTFS FAT32 With most common filesystems file searching takes considerably longer when a large number of files and or sub folders e g thousands are located in the same folder HINTS e Make sure to avoid putting unnecessary files in the same folders as input products keep the folders tidy e Recstructure the folders whenever you see or expect large numbers of files or sub folders in the same folder The set of input products that the MSG Toolbox considers and searches for is determined by e The data layer being composited fAPAR LAI DSSF and thus the product type they are part of e g FWI layer is part of the Fire Risk product FRM e The Region of Interest specified as this determines which of the four input regions are relevant e The relevant time period which is compared against the nominal date and time of the product e g HDF5 201201010000
105. tion MSGToolboxData libs jcalendar Source http www toedter com en jcalendar JCalendar is a Java date chooser bean for graphically picking a date It is provided as free software under the terms of the GNU Lesser Public License LGPL See http opensource org license Igpl html for details 2 4 3 Jbzip2 Location in the installation MSGToolboxData libs jbzip2 Source http code google com p jbzip2 Jbzip2 is a Java based compression decompression library for bzip2 archives It is provided with the following open source license according to the MIT model see http www opensource org licenses mit license php Copyright c 2010 Matthew J Francis and Contributors of the jbzip2 Project Permission is hereby granted free of charge to any person obtaining a copy of this software and associated documentation files the Software to deal in the Software without restriction including without limitation the rights to use copy modify merge publish distribute sublicense and or sell copies of the Software and to permit persons to whom the Software is furnished to do so subject to the following conditions The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software THE SOFTWARE IS PROVIDED AS IS WITHOUT WARRANTY OF ANY KIND EXPRESS OR IMPLIED INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE AND NONINFRI
106. tion on how the toolbox determines if a product is relevant for the requested composite e For management keep in mind in particular the lead trail gap in the compositing as well as the correct specification of the start end date Example 1 a near real time production of dekadal composites for monitoring The production of the first dekad of February requires the following input data to be present e The products for those 10 days as many time slots as possible e The products for the preceding days last dekad of January at least up to the number of days specified as maximum gap e The products for the next 10 days 28 dekad of February are not available due to near real time application so the trail will be empty anyhow For this type of application it can be envisioned to set up automatic data retrieval via FTP or EUMETCast in combination with data management that archives input data over e g one month old With this archiving in place the number of days is limited to approximately 20 The folder structure can thus be simplified to two levels product type day 1 day 2 Example 2 producing monthly composites covering a full year for a scientific study To produce monthly composites for the year 2013 the following input products are required e Allthe time slots of products for 2013 e Products from the last days of 2012 up to the number of days specified as maximum gap note that this can be up to a full year extra e P
107. tosynthetic Active Radiation Fraction of Vegetation Cover Leaf Area Index Fraction of Absorbed Photosynthetic Active Radiation Snow cover Land Surface temperature Land Surface temperature Down welling Surface Short wave Radiation Flux Down welling Surface Long wave Radiation Flux Down welling Surface Short wave Radiation Flux Down welling Surface Long wave Radiation Flux Surface Albedo broadband Surface Albedo broadband Surface Albedo spectral Evapo transpiration Evapo transpiration Frequency Code 15 min Hour Daily Daily Daily Daily 10 daily 10 daily 10 daily Daily 15 min Daily 30 min 30 min Daily Daily Daily 10 daily Daily 30 min Daily in the Comments filename FRP PIXEL ListProduct and FRP PIXEL QualityProduct Not supported yet FRP GRID Not supported yet FRM F024 All layers except the risk classes and reference temperature TRef FVC LAI FAPAR FVC D10 LAI D10 FAPAR D10 SC2 Not supported yet ISI Not available yet DSSF DSLF DIDSSF DIDSLF ALBEDO ALBEDO D10 AL C1 AL C2 Supported except for BRDF AL C3 model parameters K012 and covariance matrix CK ET DMET Table 1 Overview of supported LSA SAF input products VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 22 of 73 isa sar vito vision on technology Note the MSG Toolbox does not include a workflow that produces monthly yearly compos
108. tr Comment optional strHDR comment str fp if strlen str gt str sizeof h gt comment e g strcpy h gt comment str Program optional StrHDR program str fp if strlen str str sizeof h gt program strcpy h program str Close HDR file fclose fp flag name amp multi 1 amp multi sizeof h gt sensor multi sizeof h gt comment 1 multi sizeof h gt program 1 intercept and slope 30 for month YYYYMMDD default 0 item h gt Vint atof item item h gt Vslo atof item str fp amp multl 1 Oyy 10 for dekad composite Str fp amp multi 1 0 amp yyyy amp yy amp mm m amp dd 1 0 0 r 1 0 1 0 1 MSG 1 0 1 0 1 0 i OF value pairs if strlen str amp dd st amp ttm gt Check file size of image data file against header information if tis __into4 byt exp byt tru ELL amp fil struct _statio4 if stati64 img sprintf ERRmess ERRmess ERROR fun byt_tru byt_exp h gt offset if byt tru lt byt exp sprintf ERRmess byt tru byt exp ERROR fun ERRmess estat estat Cannot open IMG s 1 filestat st size h gt pixels h gt bands IMG s smaller img 1 164d img h gt bpp than expected amp tty sizeof h gt flags 365 for year amp dpm a
109. ucts daily compositing of sub daily inputs and the remapping are organized per year The 10 day and monthly composites are organized per year whereas the yearly ones are simply stored in one folder This is illustrated in the below figure VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 52 of 73 isa sar vito vision on technology Workspace Daily FVC after import m gt FVC per year E x 2012 P NS f Ss 30 min DSSF after import L DSSF per day D 20130101 gt daily remap a M gt 10daily k N FVC 10 day composites of 2013 FVC per year Figure 44 Example of the workspace file and folder organization 6 1 1 Import sub flow Input files with a bz2 file name extension are considered to be bzip 2 compressed product archives containing an hdf5 file These files will be decompressed and the resulting files will be placed in the Import sub folder of the Workspace folder The following naming convention will be used input file decompressed file HDF5 LSAF MSG var reg YYYYMMDDhhmm bz2 var reg YYYYMMDDhhmm h5 Whereby var variable or product identifier code in the filename ALBEDO FARAR reg region one of Euro NAfr SAfr SAme Example input file HDF5 LSASAF MSG FAPAR NAfr 201206240000 bz2 decompressed file FAPAR NAfr 201206240000 h5 Input files with an h5 extension or n
110. ues e REGION represents one of the four regions NAFR SAFR EURO or SAME e YYYYMMDDhhmm is the nominal date and time for the product It is usually the start of the time period time slot covered by the product The time slot can cover 15 minutes mm 00 15 30 45 30 minutes mm 00 or 30 one hour mm 00 a full day Rhhmmz0000 or a 10 day period Products disseminated via the EUMETCast satellite broadcast are in addition prefixed with S LSA S LSA HDF5 LSASAF MSG var region YYYYMMDDhhmm The filename extension is typically bz2 for the bzip2 compressed archives products and omitted or set to h5 for the HDF5 data files contained in those archives The toolbox automatically decompresses the bz2 files as needed but can also work with the HDF5 files with or without the h5 extension and supports also the S LSA prefixed file names It can also search through sub folders recursively with a speed optimization In particular the MSG Toolbox searches for input files in following order of precedence e bz2 compressed products without S LSA prefix e bz2 compressed products with S LSA prefix e h5 de compressed products without S LSA prefix e h5 de compressed products with S LSA prefix e HDF5 files without filename extension and without S LSA prefix e HDF5 files without filename extension and with S LSA prefix Other formats of LSA SAF products or related information e g files starting with ASCII are not supported
111. use in the toolbox in an archive It suffices to decompress this archive into the MSGToolboxData libs GLIMPSE REFGRIDS folder 2 3 Upgrading from a previous version As this is the first official release version 1 0 there are no previous versions For test users who were using the beta release 1 0rc1 it should suffice to update the MSGToolbox jar file and MSGToolboxData libs gdal sub folder 2 4 Software packages included in the installation The MSG Toolbox relies on several software components libraries programs provided by third parties i e beyond LSA SAF or VITO These are all free of charge and publicly available but may be subject to specific licensing terms and conditions such as open source license For informational purposes only and without intending to provide an exhaustive list here is a list of these third party software components 2 4 1 PDF Renderer Location in the installation MSGToolboxData libs PDFRenderer Source http java net projects pdf renderer PDF Renderer is an all Java library that renders PDF documents to the screen using Java2D PDF Renderer is licensed under the GNU Lesser Public License version 2 1 LGPL 2 1 More information on LGPL 2 1 can be found at http opensource org license lgpl 2 1 php VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 12 of 73 isa sar 7 vito vision on technology 2 4 2 jCalendar and part of jGoodies Location in the installa
112. vailable time slots which basically comes down to a 96 of the day which themselves can have durations of 15min 30min or hourly depending on the layer used in the compositing A configuration of 2596 would thus allow up to six consecutive hours of missing data which can thus be 6 time slots of one hour 12 slots of 30min or 24 slots of 15min When the first or last slot O UTC and the slot just before 23 59 UTC are missing then these are interpolated not extrapolated using the lead and trail data of respectively the preceding and the next day The amount of lead and trail data is limited to the maximum consecutive gap length In the above example timeline If the O UTC slot has no valid data for the given pixel the MSG Toolbox will search backwards in the data of the preceding day D 1 for the same pixel and not going beyond the last three hours of that day In case these last three hours also do not contain a valid value then the pixel is flagged as missing data The effective time period considered is thus from the 21 00 UTC time slot on day D 1 up to 02 59 UTC time slot on day D 1 b Example of time period considered for daily compositing with start at 6 UTC 06 00 UTC OUTC 05 59 UTC Trail up to Lead up to ite of Day D p Composite of Day D 1 gap 3h 1 gap 3h max consecutive gap in day 3h Starting at 6 UTC Y Calendar day D Figure 28 Example relevant time period for daily compositing with start
113. vision on technology MSG Toolbox Software User Guide Date 2015 06 16 Version 1 0 Author s T Jacobs D Haesen B Smets H Eerens VITO Dissemination level Public vision on technology e LSA SAF vito All rights amongst which the copyright on the materials described in this document rest with the VITO NV VITO Boeretang 200 2400 Mol Belgium Register of Legal Entities VAT BE 0244 195 916 The information provided in this document is confidential information of VITO This document may not be reproduced or brought into circulation without the prior written consent of VITO Without prior permission in writing from VITO this document may not be used in whole or in part for the lodging of claims for conducting proceedings for publicity and or for the benefit or acquisition in a more general sense isa sar 7 vito vision on technology Change Record 2014 02 03 Initial draft for test version rc1 2015 06 16 First release of software and user manual VITO NV Remote Sensing Dept Boeretang 200 2400 Mol Belgium Page 3 of 73 LSA SAF vito vision on technology Table of Contents 1 BACKGROUND AND SCOPE mec 9 1 1 References and acknowledgements sssssessseseessessseseeessessssseseeeesesesessessseesssesesssesssssssssessssssssssesseees 9 1 2 Structure of this COC

User manual - the LSA SAF

Contents

Download Pdf Manuals

Related Search

Related Contents