Fieldextra
Contents
1. 5 3 private release e Bug correction internal code improvements Ves Support input files mixing GRIB1 and GRIB 2 records one e Consolidate GRIB2 support e Implement NetCDF output e Code optimization e Implement shared memory parallelism OpenMP e Implement MOS corrections e Implement EPS based standard deviation and quantiles difference e And many more refinements e New FAQ Frequently asked questions e Tutorial at ARPA SIMC Tutorial for Capacity Building event Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch Shared memory parallelism e Shared memory multitasking is available and implemented with OpenMP directives e Two levels of parallelism are implemented and can be simultaneously used e parallel production of output outer loop parallelism e parallelization of some of the algorithms used during the production of each output inner loop parallelism e Two exclusive types of algorithm parallelization are available e Parallel computation when the same operator e g regridding is applied on many records within the current iteration e Grid points partitioning computation of derived field only e No distributed memory parallelism e No parallelization of input processing Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch D Shared memory parallelism Parallel production of output outer loop parallelism marked with below2. TYPE ty_fld_orig INTENT IN data_origin Data origin INTEGER DIMENSION INTENT IN 2 tot_nbr_input CHARACTER LE DIMENSION INTENT IN out_paths x multi_pass_mode just_on_time z last_call True if last call datacache Multiple pass mode True if prod now Data cache file Expected nbr input Output files names TYPE ty_out_spec DIMENSION INTENT IN out_types types TYPE ty_out_mode DIMENSION INTENT IN out_modes modes INTEGER DIMENSION INTENT IN out_grib_keys grib specs INTEGER DIMENSION INTENT IN out_subset_size subset size INTEGER DIMENSION INTENT IN out_subdomain subdomain definition INT EGER D MENSION INTEN T N out_gplist gp definition CH AR H DIMENSION CHAR j TYPE ty CHARAI INTEGER DIME TYPE ty_fld_spec_root DIMENSION INTENT I INTEGER INTENT OUT zz jerr CHARACTER LEN out loclist Error status Local parameters CHARACTER LEN PARAMETER nm generate_output Tag Local variables LOGICAL L unique_ftype multiple grid exist LOGICAL DIMENSION 3 mx_iteration 1 AL DIMENSION 3 mx_iteration 1 z exception_detected exception use_postfix INTEGER il i2 13 i fd i vd INTEGER nbr_input INTEGER out_idx ios idx_vd_defined CHARACTER LEN strlen TYPE ty_fld_type messg temporal_res out_path out_ftype i
3. local use sections local lt centre gt 250 fieldextra already supports this mechanism e Local usage local use section local usage in tables local tables e Usage description on COSMO web site Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 14 Beyond release 11 0 e Priorities and resources not yet defined e Add functionalities to allow usage of fieldextra for COSMO DE EPS e Version light without license fees for SRNWP I e Consolidate ASCII output e g uniform improved header code clean up e Support new COSMO developments e g tiles snow model e Add or consolidate support for additional products e g radar pSeudo satellite e Set of small improvements for COSMO LEPS e Wrapper scripts to offer simplified usage for common tasks e g cropping e Finalize developer documentation e Parallel input e NetCDF input e Support ICON grid Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 15 SUBROUTINE generate_output multi_pass_mode just_on_time last_call datacache data_origin tot_nbr_input amp out_paths out_types out_modes amp out_grib_keys out_spatial_filters amp out_subset_size out_subdomain out_gplist out_loclist amp out_data_reduction out_postproc_modules amp nbr_gfield_spec gen_spec ierr errmsg y arguments LOGICAL INTENT IN L DIMENSION INTENT IN LOGICAL INTENT IN CHARACTER LEN INTENT IN
4. Conf d ration suisse Federal Office of Meteorology and Climatology MeteoSwiss Confederazione Svizzera Confederaziun svizra ch Schweizerische Eidgenossenschaft Federal Department of Home Affairs FDHA Swiss Confederation Fieldextra Jean Marie Bettems MeteoSwiss 13 09 2012 Lugano CH Identity card 1 e Generic tool to process model data and gridded observations implement a set of primitive operations which can be freely combined and iterated toolbox e single Fortran program controlled by namelists e File based input output e support both GRIB1 and GRIB2 input output e support local extension of GRIB standard e understand naming conventions of COSMO files e rich set of output format in addition to GRIB NetCDF CSV XML e Primary focus is the production environment e high quality standard design implementation exceptions testing e optimized code io memory cpu and elapsed time e comprehensive diagnostic and profiling e inter process communication support parallel production suite Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 2 Identity card 2 e About 90k lines of Fortran 2003 e 10k lines yearly 20k lines last year e Linked with DWD grib library GRIB1 ECMWF grib API GRIB2 JasPer JPEG in GRIB2 NetCDF library NetCDF hdf5 library for NetCDF zlib library for NetCDF and some COSMO modules e OpenMP implementation for shared memory paralleli
5. E IF nbr_input gt tot_nbr_input out_idx THEN messg all associated input collected ELSE messg ENDIF jean marie bettems meteoswiss ch 16
6. LSE IF LEN_TRIM out_postfix 0 AND data il ofile_usepostfix AND E NOT data il ofile_firstwrite AND data il ofile complete amp use_postfix TRU Wi 1S enti I ny n m x_iteration S IF tmp_value_alloc i2 DEALLOCATE data_tmp i2 values data_tmp i2 defined IF tmp_flag alloc i2 DEALLOCATE data_tmp i2 flag IF tmp_fddata_alloc i2 THEN DEALLOCATE data_tmp i2 field_type data_tmp i2 field_origin amp data_tmp i2 field_name data_tmp i2 field_grbkey amp data_tmp i2 field_trange amp data_tmp i2 field_level data_tmp i2 field_Itype amp data_tmp i2 field_prob data_tmp i2 field_epsid amp data_tmp i2 field_vref data_tmp i2 field_ngrid amp data_tmp i2 field_scale data_tmp i2 field_offset amp data_tmp i2 field_vop data_tmp i2 field_vop_usetag amp data_tmp i2 field_vop_nlev data_tmp i2 field_vop lev E data_tmp i2 field_pop data_tmp i2 field_hop amp data_tmp i2 field_top data_tmp i2 onbr_level amp data_tmp i2 level_idx data_tmp i2 nbr_eps_member amp data_tmp i2 eps_member_idx data_tmp i2 field_idx ENDIF IF tmp_gpdata_alloc i2 THEN DEALLOCATE data_tmp i2 gp_coord data_tmp i2 gp_idx amp data_tmp i2 gp_lat data_tmp i2 gp_lon data_tmp i2 gp_h ENDIF END DO Prepare data for print out calculate new fields populate data_pout Info message IF just_on_time out_idx THEN messg just on time output ELS
7. Loop over each record of each input Once a complete set of records is available Storage 1 Output 1 Storage 2 Output 2 Storage 3 Output 3 Storage 4 Output 4 For each output the following operations are applied in parallel 1 For each record in turn check use of current record process and store record 2 Once a complete set of records is available iterative processing of parent fields format and write output Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch D Shared memory parallelism Algorithm parallelization inner loop parallelism marked with below input 1 Storage 1 Output 1 bh EE E Derivation of new field P re gridding Output 2 Storage 3 Output 3 Storage 4 Output 4 Within the current processing iteration for the current output For each operator in turn parallel computation when the same transformation is applied on multiple fields or parallel computation on multiple partitions of the horizontal domain Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch Fieldextra Performances 1 e Configuration for following performance results Fieldextra 10 5 3 Code compiled with gfortran with O3 optimization level One Cray XE6 node 4x 2 1 GHz AMD MagnyCour processors for a total of 24 cores Lustre parallel filesystem Report total elapsed time rof time for decoding input in time for product generation prod and me
8. e situation of a production environment where the size of the model output remains constant but the number of products increases with the time Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 12 What shall expect next Next public release e Releases 11 0 Nov 2012 e New operators geostr vorticity vorticity advection thickness advection wind divergence humidity convergence frontogenese function CAT index Interpolation on theta surfaces e Consolidated test environment With support of H Asensio DWD e Cookbook with real life examples e The release 11 will fulfill all the requirements defined at the postprocessing workshop Langen 26 02 2009 Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 13 What shall I expect next GRIB2 coordination e Short names Master table on COSMO web site provided by DWD Excel table Tool to derive fieldextra dictionary from master table e Model name no WMO standard mechanism e Derived from the following set of keys center subCenter productDefinitionT emplateNumber generatingProcessldentifier Each COSMO member define a unique combination of these keys for each model operated at their center and use them consistently this is documented on the web fieldextra already supports this mechanism e Experiment tag no WMO standard mechanism localNumberOfExperiment as compulsory entry in all local use sections Default
9. err 0 errmsg exception_detected FALSE tmp_fddata_alloc FALSE tmp_gpdata_alloc FALSE tmp_value_alloc FALSE tmp_flag_alloc FALSE Create update data cache file state of collect output i e before any field manipulation done in prepare_pout Fieldextra COSMO GM Lugano 13 09 2012 P specifications gen_spec l fields to generate locations definition efining filter El pos LR DO i2 INTENT OUT 1 errmsg error message tmp_fddata_alloc tmp_gpdata_alloc tmp_value_alloc tmp_flag alloc output_file loop amp DO il 1 nbr_ofile out_idx data il ofile_idx nbr_input COUNT data il ifile_used Skip bogus output IF data il ofile_bogus CYCLE output_file_loop Skip completed output IF data il ofile_complete Skip empty data array IF ALL NOT data il defined CYCLE output_file_loop Only prepare output when all possible associated data have been collected CYCLE output_file_loop or when just on time production is active IF MOT last_call AND amp nbr_input lt tot_nbr_input out_idx AND amp NOT just_on_time out_idx CYCLE output_file_loop At this point the corresponding output file will be produced Keep track of completed output file IF nbr_input gt tot_nbr_input out_idx data il ofile_complete TRUE Build name of output considering a possible temporary postfix use_postfix FA
10. mory high water mark hwm Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch 10 Fieldextra Performances 2 e 24h COSMO 1 hourly GRIB 2 output with DD amp FF on all levels COSMO 1 grid size 1062 x 774 x 80 Input size about 60GB output size 6 6GB 25 products 24h COSMO 1 production time is about 42 1x1 thread 1 tot 4 in 10 prod 1x6 threads 10 tot 4 in prod sl e 72h COSMO 7 1200 products operational COSMO 7 grid size 393 x 338x 60 Input size about 23GB output size about 10GB about 1200 products 72h COSMO 7 production time is about 30 1x1 thread 25 tot 5 in 19 prod 1 other 6x4 threads 12 tot 5 in prod 1 other 4 4GB hwm Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch Fieldextra Performances 3 e Speedup between release 10 5 1 and 10 5 3 optimization OpenMP e COSMO 7 production from gt 3200 s to 720 s Speedup about 4 5 6x4 threads e FABEC production from 4400 s to 280 s Speedup about 15 1x6 threads e CAPE_MU production on COSMO 7 domain from 93 s to 22 s Speedup about 4 5 1x6 threads e Poor performance of lateral regridding with gfortran compiled code fixed in 10 5 2 e Problem was much less acute with Pathscale compiled code e A weak scalability of fieldextra is obtained when the GRIB decoding time is neglected e Typically th
11. sm Standalone package available on COSMO web site including source code for all above mentioned libraries http www cosmo model org content support software default htm Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch Identity card 3 e Portable code e Test platforms Cray Opteron IBM Power e Test compilers GNU Intel IBM Should work on any UNIX Linux Mac platform e Documented code e User manual examples FAQ developer manual e Community support e cosmo fieldextra cosmo model org e Limitations e Complex namelists steep learning curve Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch Usage e COSMO software licensed e COSMO adaptor for the EUMETNET programme SRNWP interoperability e Core non graphical NWP production tool at MeteoSwiss e About 15 000 products per day generated with fieldextra representing more than 200 GB data e Products derived from COSMO 2 COSMO 7 COSMO LEPS PEPS IFS e Thresholds and regions based warnings for the Common Information Platform for Natural Hazards developed for the Swiss government e COSMO LEPS production at ECMWF e FABEC production at DWD e Additional products for the German flight control e Others e NMA RHM Fieldextra COSMO GM Lugano 13 09 2012 jean marie bettems meteoswiss ch ck Activities since last COSMO GM e COSMO GM 2011 release 10 4 0 COSMO GM 2012 release 10
Download Pdf Manuals
Related Search
Fieldextra field extractor field extraction in splunk field extraction field extractor splunk field extraction training