EnviHamManager USER GUIDE 1. Introduction
Contents
1. This applies particularly to a 2 PC setup 4 4 Data Cleanup Once a day at a specified time old files may be deleted according to the age of each file Different ages may be specified for each image type and each file type see section 5 Section 4 8 1 gives details of how to specify this Note that the age of file calculated when cleaning up refers to the time the image was taken not the time it was received by you Sometimes this can be a day later suggest e Derived JPG thumbnail DIMAT uncompressed 1 files Since they occupy a lot of space and because the original compressed N1 gz data files will still be available it s probably a good idea to tidy up these files frequently after you have finished looking at the most recent Normally there won t be many uncompressed 1 files lying around but some programs can leave them e Raw compressed N1 gz data files as received Page 6 of 25 How frequently you clean these files up is really up to you as to how long you wish to keep things However EnviHamBrowse allows you to browse images and archive those which you wish to keep which you may find useful Cleanup only occurs in the currently specified data directory If you have changed this see section 4 8 1 3 cleanup will not occur in the previous one Page 7 of 25 4 5 Status Panel As EnviHamManager is intended for continuous operation there are no controls as such only a status panel directory L EnviHa2. be downloaded from 6 Additionally if the receiving and data directories are different see 3 3 information will be logged about the transfer of files from receiving to data not shown in the figure Cleanup log stats Monthly stats Weekly stats Daily stats Hourly Files received Processing Processing status og Page 13 of 25 4 6 5 Cleanup log Shows the results of a cleanup session Files deleted successfully are shown in black any which failed to delete are shown in red Cleanup log Rix stats Monthly stats Weekly stats Daily Rix stats Hourly Files received Processing status Processing log Page 14 of 25 4 7 Thumbnails Display Also under the status panel is a set of tabbed displays showing thumbnails of the latest file of each type to be received Note that thumbnails will only be shown if automatic conversion to JPG format see section 4 8 1 2 is enabled The thumbnail for file type ATS is not very useful because the image is so long and thin however it is included for completeness Latest MER RR Latest ATS Latest MER FRS 71 4 Y MER FRS 1 20110609 083412 000001633103 00208 48438 8375 1 42 Received 2011 06 03 23 15 03 Page 15 of 25 Latest Latest Latest ATS MER RR MER FRS R
3. by DDS as well as certain file reception errors e Automatically generate DIMAT and JPG files both full size and thumbnails from received files DIMAT is the native file format used by VISAT and will therefore be loaded more quickly by VISAT JPG is the familiar image file format which is used EnviHamBrowse for previewing Show thumbnails of the latest received files e Monitor the DDS file reception log Provide graphs of counts of files received successfully and failed for various periods to see how you are doing e Delete old or unwanted files to keep disk space under control e Report status of its functions talk about directories in this guide Microsoft users may prefer the term folders However used directories most of my working life so will stick to that Please email me at vi0123 btinternet com with any bugs comments or suggestions Page 1 of 25 2 What s New The major new features in v 0 2 7 are Catchup mode added Previously EnviHamManager only processed new files as they came in Files which had been received earlier but not processed were ignored You can now choose to process existing files which are newer than a specified age Because this may result in a long queue for processing a status page has been added to monitor how things are going Option to delete raw files without processing Useful for unwanted files such as ATS and MER_RR Finer control over cleanup amp process
4. of DIMAT and JPG files speeds up browsing and VISAT usage in EnviHamBrowse The original raw Nigz data files as received are not affected When generating JPG files a thumbnail image file will also be created for use both by EnviHamManager and EnviHamBrowse Therefore generation of JPGs should not normally be inhibited since thumbnails will not be available Conversion uses the peonvert program which you will have installed as part of VISAT One thing which JPG conversion requires is the specification of a profile which tells the converter how to use the raw multi spectral images to produce a false colour result Profiles are provided as part of the VISAT package or for enthusiasts can be generated manually using VISAT Which one is used in the conversion is selected for each file type in Settings see section 4 8 1 2 4 3 Catchup Mode In versions of EnviHamManager prior to 0 2 7 only newly received files were converted If you had files present which had not been converted when EnviHamManager started they were ignored Although manual conversion can be done using EnviHamBrowse a catchup mode has now been added so that files newer than a specified age will also be processed see section 4 8 1 1 as to how to set this up recommend that you leave catchup mode on with an age of at least one day This means that if you have EnviHamManager shut down for a short time any files received during that period will still be processed
5. received from DDS They are compressed versions of the Envisat N1 data files e Uncompressed N1 file extension N1 N1 files are the native format developed by ESA for storing and disseminating data from Envisat Not generally found since EnviHamManager will normally delete them automatically but sometimes VISAT can leave them behind e DIMAT file extension dim plus an associated directory with extension data Partially processed N1 files This is the native format used by VISAT It is quickest for VISAT to load and therefore EnviHamManager automatically generates them e JPG file extension JPG Full size JPG image file automatically generated from the Compressed N1 file by EnviHamManager Much easier to view than using VISAT The companion program EnviHamBrowse uses JPG and thumbnails for easy browsing e Thumbnail file extension tn JPG Small scale JPG image file 5 2 Image Types At the moment EnviHam transmits 3 types of image e ATS AATSR Advanced Along Track Scanning Radiometer instrument level 1 gridded brightness temperature and reflectance MER RR MERIS Medium Resolution Imaging Spectrometer instrument level 2 reduced resolution geophysical product MER FRS MERIS instrument level 1B full resolution European coverage A lot more detail about these topics can be found in 4 Page 22 of 25 6 Command Line Options When EnviHamManager exe is started it is possible to add options to the comman
6. EnviHamManager USER GUIDE Author Francis Breame G8ISI vf0123 btinternet com Version 0 2 7 111117 1 Introduction To quote from the EnviHam introduction document 2 Some categories of non professional users e g Education Radio Listening E O Enthusiast have shown interest to receive near real time Envisat 3 4 data via the satellite Data Dissemination System DDS 7 The major objective of the Envi Ham project shall be to widen the knowledge of ESA E O missions among non professionals to retrieve possible observations done by users and to coordinate in a single non commercial project a whole groups of users which otherwise would be applicant for individuals ESA approved scientific CAT 1 projects When you are registered for EnviHam you will be sent the DDS reception software and licence key and can download the very comprehensive VISAT viewer software 5 VISAT is an application built on the BEAM toolbox Details about how to set up the reception hardware and software can be found in GEC Quarterlies 25 amp 26 and 8 After that prepare for the data EnviHamManager iS a companion to EnviHamBrowse which displays on a world map the areas covered by the Envisat amongst other things by automating some of the tasks Both programs are available from my www site 1 EnviHamManager is intended to be run all the time DDS reception is running Its main functions are to e Detect new files as they are received
7. RTICULAR PURPOSE See the GNU General Public License for more details For details of the GNU General Public License see http perldoc perl org perlgpl html or write to the Free Software Foundation Inc 59 Temple Place Suite 330 Boston MA 02111 1307 USA 9 Links and References 1 My www site htto www elnath org uk please note new URL 2 EnviHam http dwlinkdvb esrin esa it DDS docs Public_ Docs EnviHam abstract pdf 3 Envisat Operations http www esa int SPECIALS Operations SEMOZY8L6VE_0 html 4 Envisat Documents http earth esa int pub ESA DOC ENVISAT 5 VISAT BEAM http www brockmann consult de cms web beam 6 VISAT BEAM software download http www brockmann consult de cms web beam software 7 DDS Operations http dwlinkdvb esrin esa it DDS welcome html 8 David Taylor s notes on setting up EnviHam reception http www satsignal eu wxsat EnvisatDDS Page 25 of 25
8. SS amp Fh H 2 eee ec 1 _ __2 20110608_152800_000001973103_00198_48488_7070 1 92 Received 2011 06 09 17 20 44 Page 16 of 25 Latest Latest Latest ATS MER RR MER FRS Page 17 of 25 4 8 Menus Not a lot to say really Files Exit wonder Settings Brings up the Settings popup see section 4 8 1 Help About List version information For actual usage help you ll have to look at the file you re currently reading Page 18 of 25 4 8 1 Settings Popup Brought up by the Settings menu SETTINGS SSC e JEATock bear 430 Ntinipcorvertbat 9 C Page 19 of 25 4 8 1 1 GENERAL section Receiving directory Data directory pconvert exec DDS log file Colour scheme Catchup mode Location of your received files see section 3 3 Location to put received and processed files see section 3 3 Location of the pconvert executable file see section 3 3 Location of the DDS file reception log file see section 3 3 This will always be named iles 1log but its location may vary depending on where you installed the DDS software Select either black or white background for the log tabs White uses the original scheme Catchup mode see 4 3 is enabled if ticked Only files newer than the specified ago in days based on the time the image was taken not the time it was received by you wi
9. cerned with deleting old compressed N1 raw data files so that you don t lose data which you wanted to keep JPG conversion profiles When generating JPG images these tell the converter how to use the raw multi spectral images to produce a false colour result See section 4 8 1 for details on how to change these settings Also please review the other default settings to ensure that they are suitable If any settings are wrong you will be warned when you start the program Page 4 of 25 4 Usage 4 1 Getting Started EnviHamManager is very simple to use On startup you will see the following File Settings Help Rx directory LAEnviHamVeceived Data directory SAEnviHamvdata2 m 48 Files failed 0 Processing 0 oday today queue 41 X used 67 used Data Catchup mE 1747 GB free 6172 GB i Receiving Processing eta Uu Latest Processing fle Files Rxstats Rxstats Rxstats Processing Processing Cleanup Latest received Hourly Daily Weekly Monthy status log log ATS MER RR MER FRS Latest Latest MER FRS 1PNPDK20111117 114230 000001213108 00368 50813 5500 Received 2011 11 17 13 04 55 The picture on the right just shows the last picture you received will be blank if you haven t received anything before Aft
10. d line They re probably not generally useful However if you have a need you can always edit the desktop shortcut so that they are always specified nodel Prevents cleanup actually deleting files although those it would have deleted will be shown in green in the log Useful for testing if you want to check your criteria before actually trashing anything Additionally if the receiving and data directories are different see section 3 3 received files will be copied to the data directory but not deleted from the receiving They will be show in yellow in the conversion log Page 23 of 25 7 Change History 2 N N o 0 2 5 0 2 4 0 2 3 0 2 2 0 2 1 0 2 0 0 1 0 111117 Finer control over cleanup amp processing settings Option to delete raw files without processing Log pages now have choice of colour scheme Settings validation improved Settings popup layout improved Add catchup mode Add processing file display queue size and status window Fix bug which would prevent startup if an initial directory was dud 110810 Trap pconvert JPG conversion failure to prevent crash happens with overlong ATS files Trap missing file in fileAge 110721 Add disk utilisation scales 110623 Allow for separate receiving directory cf EUMETCast software JPG files could sometime be held open thereby preventing their deletion Update VISAT executable default locations Increase converter timeo
11. desktop The first time you run EnviHamManager or a new version of it it will cache some files which will make it take longer to start say 30 seconds so be patient Thereafter it will start normally 3 2 Upgrading Follow the same procedure if you are upgrading an existing installation Providing that you unzip it to the same directory your existing settings will be used Defaults will be utilised where new settings have been added however it is worthwhile looking at the settings to see if you wish to customise any new ones 3 3 Customisation You will first need to do a little customisation to adapt to your setup see section 4 8 1 for details on how to change these settings This involves four things e Receiving directory This will need to be altered to wherever you setup the DDS reception software to put the files The default is set to my location yours will be different e Data directory This will need to be altered to wherever you want EnviHamManager both to store the received files and the converted files it generates The default is set to my location yours will be different If you are receiving the files on one PC and running EnviHamManager on another obviously the receiving and data directories will be different Page 3 of 25 If you are receiving and processing on the same PC you can if you wish set both directories the same although recommend that you keep them different as it keeps the function
12. er it has been running for a bit assuming that DDS is working you should see _Envitiam File Manager 0 2 7 111117 101 File Settings Help Rx directory LAEnviHamVeceived Data directory S EnviHlam data2 Files rx 45 Files failed 0 Processing 0 today today queue 43 used 652 used m Data Catchup ees 1704 GB free 633 7 GB d Receiving B Proceso mode Latest fle LAEnviHam received tmp MER 20111118 074516 000001213108 00390 50825 5747 N1 92 Processing fle LNEnvilamiveceivedWEF 2PNPDK20111118 063225 000009773108 00379 50824 5729 1 Files Ri stats Rx stats Px stats Rix stats Processing Processing Cleanup Latest Latest Latest received Daily Weekly Monthly status log log ATS MERRR MER FRS MER_FRS_1PNPDE20111117_070055_000001883108_00365_50810_0458 N1 gz Received 2011 11 18 09 52 53 Leave it running minimised if you like as long as DDS is running Page 5 of 25 4 2 Data Processing Once a file has been received various processing is performed on it principally by moving it from the receiving to the data directory and then generating DIMAT and JPG files It is possible to select just what processing is performed for each image type see section 4 8 1 2 for details In particular unwanted raw data files can just be deleted The presence
13. failed today count is maintained from the data in the DDS files received log and will therefore always be correct Page 9 of 25 4 6 2 Statistics Reception statistics are provided showing counts of files received successfully green and failed red summed over various periods Hourly for the last two days Daily for the last month Weekly for the last year Monthly for the last year stats Monthly stats Weekly Rix stats stats Files received Processing Cleanup Processing log log status Daily Hourly 1u 59114 143743 SIMI SIYE 141543 1543543 215415433 12437243 L 3243243 45432431 ITITI 143441 4561543 45432431 143 45432431 42437243 42432431 15215431 21561543 15435433 215415433 143543 215635433 1541541 1501543 21541543 Hour MM Files failed WB Files ok Page 10 of 25 a m 5 9 o c pu 2o 2 a e 3 m 8a a 0 LE ES c 10 Xx 5 x c m m 2 D da 8 MUS mm da Files received 19099 800 MM Files failed WB Files ok Save Cleanup log Processing log Process
14. ing status Rix stats Monthly stats Weekly stats Daily stats Hourly Files received Syuy 4000 3000 59114 2000 1000 90211 50211 50211 50211 21201 11201 01401 Files failed Files ok Save Page 11 of 25 4 6 3 Processing status Shows the status of the processing queue see section 4 5 Normally there won t be much in it unless heavy use if being made of catchup mode Rx stats Monthly Fix stats Weekly Rix stats Daily stats Hourly Files received Processing status Cleanup log Processing log Page 12 of 25 4 6 4 Processing log Shows the logs generated by pconvert when doing DIMAT or JPG conversions They may be of interest if something goes wrong Don t worry too much the program seems very pessimistic and a lot of the errors can be ignored In particular always get Error Could not load mediaLib accelerator wrapper classes Continuing in pure Java mode which may be generic to Windows It makes no difference Unfortunately a bug in the version of pconvert supplied with BEAM VISAT 4 8 causes it to fail JPG conversions with some profiles see http www brockmann consult de beam jira browse BEAM 1200 Additionally it fails with ATS files Therefore please upgrade to BEAM VISAT 4 9 0 1 or later which seems to fix these problems It can
15. ing settings File cleanup ages can be set individually for each image type and file type Each stage of file processing can be selected again individually for each image type and file type Log pages now have choice of colour scheme prefer a black background but you can set it back to the original white if you prefer Settings validation and popup layout improved Hopefully more user friendly Page 2 of 25 3 Installation EnviHamManager is written in Perl and Perl Tk However it is distributed as a self contained exe file which includes all the necessary Perl runtime files You do not therefore need to have Perl installed It runs under versions of Windows from at least XP onwards and quite probably 2000 As it has a lot of interactions with the operating system it will not run under Linux although could produce a version which does if there is interest it s much easier using the OS under Linux than Windows If you are not familiar with the different image and file types which are handled by EnviHamManager have a look at section 5 which gives details 3 1 Installing EnviHamManager isn t installed as such just unzip the downloaded file EnviHamManager zip to somewhere convenient say C Program Files EnviHamManager for Windows or 15 for Windows Vista amp 7 From wherever you unzipped the files just click on EnviHamManager exe to run it or setup a shortcut on your
16. ll be processed Setting the age to zero will disable catchup mode thus the age must be set gt 1 before the tick can be set which will happen automatically anyway recommend leaving it set to at least 1 day 4 8 1 2 PROCESSING section Allows you to select what processing is to be done on each image type and file type see section 5 Some combinations of processing are contradictory and will be blocked e g if you select None then no other processing can be selected or are mandatory e g if you select Convert then Copy rx is automatically selected Processing operations are None Delete rx Copy rx Convert DIMAT Obvious The original received compressed N1 file will be deleted after copying from the receiving to the data directory if Copy rx is selected Can be used on its own to get rid of unwanted files without further processing Has no effect if the receiving and data directories are the same and Copy rx is selected Copy files from the receiving to the data directory Has no effect if the receiving and data directories are the same Automatically selected if either of the Convert operations are selected Convert the file to DIMAT format suitable for quicker input to VISAT Page 20 of 25 Convert JPG Convert the file to full size JPG and a thumbnail JPG thumbnail Recommended that you do not disable this as EnviHamBrowse Uses both for easy browsing Convert profile JPG conversion profile
17. m received Data directory S EnviHam data2 Arcu oS e G 170 4 GB free 633 7 GB free Latest file L EnviHam received tmp MER_FRS_1PNPDK20111118_074516_000001213108_00380_50825_5747 N1 g2 Processing file L EnviHam received MER_RR__2PNPDK20111118_063225_000009773108_00379_50824_5729 N1 92 Files Files failed Processing 48 n 5 today today queue Receivin Processin Data Catchup 9 9 oO cleanup mode The disk utilisation scales show green for 0 74 yellow for 75 89 and red for 90 100 Latest file shows that most recent file found in the receiving directory normally the latest to be received Processing file is the file currently being processed if any Once reception starts a file prefixed tmp should appear in the reception directory When detected the Receiving indicator will light When reception has finished assuming that it didn t fail Receiving will go out and processing of the file will commence as shown by the Processing indicator To save computer load only one file is processed at a time so that other files may be received simultaneously Because a queue of files to be processed can build up if catchup mode see section 4 3 is used with several older files the number of files in the queue is displayed in not surprisingly Processing queue Further details of the queue are displayed in the Processing status tab see section 4 6 3 Data cleanup shows that deletion of unwa
18. nted files is in progress see section 4 4 Catchup mode shows if it has been selected see section 4 3 Note that the counts Files rx today and Files failed today are maintained from the data in the DDS files received log and are only updated once an hour on the hour Thus these counts may appear to lag a little Page 8 of 25 4 6 Logs and Statistics Display Under the status panel is a set of tabbed displays containing either logs or statistics graphs Note that each log display is limited to the last 1000 lines The Save button allows you to save the contents of the currently displayed log as a text file or statistics graph as a jpg file 4 6 1 Files received This shows er a list of the files received similar to the display on the DDS software Timestamps refer to the time reception of the file finished at your station stats Monthly stats Weekly stats Daily Fix stats Hourly Files received Processing status Processing log Cleanup log EnviHamBrowse can detect certain files for which reception has failed and these are shown in red This is done but detecting a file prefixed tmp which is not immediately followed by the same file missing the tmp prefix meaning that reception has failed Some failures may be missed because files are only scanned every minute and therefore a tmp file which lasts for less than one minute may be missed The Files
19. s applied see section 4 2 User generated profiles will be suffixed user 4 8 1 3 DATA CLEANUP section Schedule at Time in the form hhmm when data cleanup will be run It will only be run once every 24 hours sometime overnight is probably a good idea Cleanup will be performed on files which have ages in days greater than those specified based on the time the image was taken not the time it was received by you Setting the age to zero will disable cleanup thus the age must be set 1 before the tick can be set which will happen automatically anyway Different ages may be specified for each image type and file type see section 5 Other deletion criteria may be added in the future suggestions are welcomed 4 8 1 4 Action buttons Save Save the settings in the ini file and quit the popup Apply Apply the settings to the current program but don t save them Quit the popup Cancel Give up without doing anything Values will be checked for validity and you will not be allowed to Save or Apply them if they are incorrect You will be told what is wrong unlike some software Page 21 of 25 5 File and Image Types EnviHamManager deals with a number of different file types each of which may hold different types of image They are listed here as a number of the processing options of EnviHamManager apply to them individually 5 1 File Types e Compressed N1 file extension N1 gz These are the raw Envisat files
20. s separate Location of the pconvert program This is a data conversion program provided with VISAT and is located in the same directory aS VISAT The default setting is adequate for many versions of Windows Windows 7 64 bit users will however probably need to replace C Program Files with C Program Files x86 Location of the DDS files log This is the log file files log kept by the DDS reception software on the receiving PC If EnviHamManager is running on the receiving PC then the default C Program Files Envisat DDS files log may work although Windows 7 64 bit users will probably need to replace C Program Files with C Program Files x86 If however the PCs are different a share will need to be set up This is a little more difficult if you have installed DDS in C Program Files since Windows from XP onwards will not allow you to share C Program Files so you can t just set up a share to the c drive It will allow you to share subdirectories however so you will have to setup a share to C Program Files Envisat DDS and use that This is a very brief description If you are using separate PCs anyway you will probably know about sharing drives however let me know if you need help with getting shares set up Additionally please check that the settings for the following meet your requirements Data cleanup Check that the settings for automatic cleanup of older data meet your needs especially those con
21. ut from 600 to 1200 s Conversion log shows conversion run time 1106068 incorrectly titled 0 2 2 although the date is correct Increase thumbnail size to 600 Add last received thumbnail display Reenable file failure reporting not exhaustive as a short lived tmp may be missed in a scan cycle o Check to see if a tmp file is still current but it or its permanent version no longer exists in which case exit rx mode Probably rx has failed and it has been deleted without anything coming in behind it o Check that file following a tmp is its permanent version 110518 Add command line to conversions log Correct stats file counts if nothing received today Add Save button to all log stats pages Create thumbnail at same time as jpg Remove disabled area field in Data Cleanup Settings and associated ini settings dataArea 110505 Fix problem in converter whereby dirname returns E from Efile 110503 Add reception statistics graphs Remove original failure detection 110430 Initial issue Page 24 of 25 8 Legal Stuff This program is free software you can redistribute it and or modify it under the terms of the GNU General Public License as published by the Free Software Foundation either version 2 of the License or at your option any later version This program is distributed in the hope that it will be useful but WITHOUT ANY WARRANTY without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PA
Download Pdf Manuals
Related Search