Home

NCO User Guide

1. 0000 126 cell based grid 0 eee eee eee eee eee 68 cell area meeer ie dee etr p menie des 76 cell_methods ccc cece cece eens 76 126 290 OB costs Roca eee T To gendoh s 75 CF compliance checker 0000s eee 54 CF conventions 48 67 81 93 124 191 215 cfche cker perles cns eee RM IBN EE 54 change miss suspin Ween ve eek a ees 145 A uses halle a e a bales Eus 148 characters special 00 eee eee eee eee 186 Charlie Zender 0 00 cee eee eee eee 1 241 Chocolates sorron ea eei aa tarden pda cae 241 Chris Barker 21222 2 2i doused ees 88 AA Iba eR bx 12 85 208 chunking map 0 eee ee 86 88 chunking policy ise tos eae ie Eros 86 cliunksize i eiie wide ind 86 Citation ea dte roe e egeta 243 CM ii Et 8 CLASSIC fil s ae fee ii te 38 Client Server cece cee eee eee ene eeee 35 Climate and Forecast Metadata Convention 81 climate model 17 20 31 198 238 climatology attribute 93 125 215 climatology convention 085 125 clipping operators 0 eee eee eee eee 134 OMIP eerie ctu de ire bid bein 194 197 CMIP5 ected van pale hai aiala pa hale data fondle 247 seik a say poles tanta fae d bashes baad 86 AAA Lp UC Uu RTL 88 teni god e c o ou tT AT A HM S AS IAS 86 Met A EELE 86 Cenk lfp yas hit dk eee aS ees Y aW METRI Y 88 Chk map sse toh hone pakistan yous eee ah
2. and will not be filtered and will be included in the attribute Note that the NUL character NO which terminates C language strings is assumed and need not be explicitly specified If NO is input it is translated to the NUL character However this will make the subsequent portion of the string if any invisible to C standard library string functions And that may cause unintended consequences Because of these context sensitive rules one must use ncatted with care in order to store data rather than text strings in an attribute of type NC_CHAR Note that ncatted interprets character attributes i e attributes of type NC CHAR as strings EXAMPLES Append the string Data version 2 0 n to the global attribute history ncatted a history global a c Data version 2 0 n in nc Note the use of embedded C language printf style escape sequences Change the value of the long_name attribute for variable T from whatever it currently is to temperature ncatted a long name T o c temperature in nc Chapter 4 Reference Manual 187 NCO arithmetic operators may not work as expected on IEEE NaN short for Not a Number and NaN like numbers such as positive infinity and negative infinity One way to work around this problem is to change IEEE NaNs to normal missing values As of NCO 4 1 0 March 2012 ncatted works with NaNs First set the missing value i e the value of the _FillValue attribute for the variable s in
3. sss 200 T write tn p fl 2x ewido te ENTRAR 18 cOWEU Umnp fl c0 en ah Weitere dl cits OPER 18 SERCH acest sas dro as do 43 213 UXl l oe aoe Shanna aie eee i qe E PEERS 210 xml no Vocation eer snesena tei soe is 210 PMA Spr CDP ni ae wee eas 210 ML SP teil DNE 210 Oe was te Hares cee iD dude eee E Me xs 58 pnr oe Seis v RPM 13 38 B ts ete s tuus este dt en cu t Y 13 38 O SS IDEM DD A 38 204 TO6 ols A s uude baits idu 38 A oue Ue pe RARE ETIAM C an E EET Lm 38 E pav E m 204 213 Ass Uf o ibas 18 121 213 221 A EEEE 178 191 205 SB mask CON lt 8 5 scere RW va 235 236 Seide hin dieci beta ee a heey 47 126 SUUM LA atre aes gee punt DORE 47 126 178 d dim min max stride subcycle 62 d dim min max stride 60 287 d dim min max stride 58 64 66 79 d dim min max1 suse 235 Dra 15 D debug level sss 16 24 29 SE Ss ne ea neqageic ii eU CIR auiem E 180 Lic aed 57 porq pu LEID amatees tasks 43 A A sts acne cents 48 da 122 183 Lor 123 207 ST cian a is 236 boutput p tho lE errare 33 34 ux 101 SMe telah phim iM ILE aes Mad ea 208 numaskcVari elis dee t Lepus Urt 235 SBMolcnxiignreit rre de 39 198 207 M CORE maps ceca esse ie ip c os 3 606a eri 88 SM DG Map eeu iu VOL REA Id eb 218 pum Em 21 23 30 SN toy x ied Lreb EAR PERDIDA E E 109 237
4. 64bit 7 fl_fmt netcdf4 All NCO operators support read and write all three or four depending on how one counts file formats supported by netCDF4 The default output file format for all operators is the input file format The operators listed under Availability above allow the user to specify the output file format independent of the input file format These operators allow the user to convert between the various file formats The operators ncatted and ncrename do not support these switches so they always write the output netCDF file in the same format as the input netCDF file 3 9 1 File Formats netCDF supports four types of files CLASSIC 64BIT NETCDF4 and NETCDF4_CLASSIC The CLASSIC format is the traditional 32 bit offset written by netCDF2 and netCDF3 As of 2005 nearly all netCDF datasets were in CLASSIC format The 64BIT format was added in Fall 2004 As of 2010 many netCDF datasets were in 64BIT format As of 2013 an increasing number of netCDF datasets were in NETCDF4_CLASSIC format The NETCDF4 format uses HDF5 as the file storage layer The files are usually created accessed and manipulated using the traditional netCDF3 API with numerous extensions The NETCDF4_CLASSIC format refers to netCDF4 files created with the NC_CLASSIC_MODEL mask Such files use HDF5 as the back end storage format unlike netCDF3 though they incorporate only netCDF3 features Hence NETCDF4_CLASSIC fil
5. divide division Examples ncbo op_typ 1 nc 2 nc 3 nc ncdivide 1 nc 2 nc 3 nc 190 NCO 4 5 3 alpha03 User Guide Care should be taken when using the shortest form of key values i e and Some of these single characters may have special meanings to the shell Place these characters inside quotes to keep them from being interpreted globbed by the shell For example the following commands are equivalent ncbo op typ 1 nc 2 nc 3 nc Dangerous shell may try to glob ncbo op_typ 1 nc 2 nc 3 nc Safe protected from shell ncbo op typ 1 nc 2 nc 3 nc Safe protected from shell ncbo op typ mlt 1 nc 2 nc 3 nc ncbo op typ mult 1 nc 2 nc 3 nc ncbo op typ multiply 1 nc 2 nc 3 nc ncbo op typ multiplication 1 nc 2 nc 3 nc ncmult 1 nc 2 nc 3 nc First do 1n s ncbo ncmult ncmultiply 1 nc 2 nc 3 nc First do 1n s ncbo ncmultiply No particular argument or invocation form is preferred Users are encouraged to use the forms which are most intuitive to them Normally ncbo will fail unless an operation type is specified with y equivalent to op_typ You may create exceptions to this rule to suit your particular tastes in conformance with your site s policy on symbolic links to executables files of a different name point to the actual executable For many years ncdiff was the main binary file operator As a result
6. drc_out f1d_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_195601 200512 nc Average along time ncra 0 drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_195601 200512 nc Chapter 7 CMIP5 Example 255 drc_out fld_out var_id var var_id _ mdl nc echo Model mdl done done Remove temporary files rm drc_out fld_out var_id var var_id historical nc Store models as groups in the output file ncecat 0 gag drc_out fld_out var_id var var_id _ nc drc_out fld_out var_id var var_id _ rlm _ all md1_ xpt 0 _all nsm_clm nc echo Var var var_id done done With the use of group the above script will be shortened to ONE LINE Data from cmb_fl_grp sh ensemble averaging nces 0 nsm grp nsm_sfx _avg sn LImon all mdl all xpt all nsm 200001 200512 nc sn LImon all mdl all xpt nsm avg nc The input file sn LImon all mdl all xpt all nsm 200001 200512 nc produced by cmb_f1_grp sh includes all the ensemble members as groups The option nsm_grp de notes that we are using group ensembles mode http nco sf net nco htmlinsm grp of nces instead of file ensembles mode http nco sf net nco htmlitnsm f1 nsm fl The option nsm_sfx _avg instructs nces to store the output as a new child group mode1 model name avg var otherwise the output will be stored directly in the
7. glb h hdf hdr_pad nbr L dfl 1vl1 1 path mro N n loop no tmp f1 0 o output file p path ppc R r ram all rec_apn rth db1l f1t t thr nbr unn v var w wgt X x y op typl input files output file DESCRIPTION ncra averages record variables across an arbitrary number of input files The record dimension is by default retained as a degenerate size 1 dimension in the output variables See Section 2 6 Statistics vs Concatenation page 20 for a description of the distinctions between the various statistics tools and concatenators As a multi file operator ncra will read the list of input files from stdin if they are not specified as positional arguments on the command line see Section 2 7 Large Numbers of Files page 21 Input files may vary in size but each must have a record dimension The record coordi nate if any should be monotonic or else non fatal warnings may be generated Hyperslabs of the record dimension which include more than one file work correctly ncra supports the stride argument to the d hyperslab option see Section 3 15 Hyperslabs page 58 for the record dimension only stride is not supported for non record dimensions ncra always aver ages coordinate variables e g time regardless of the arithmetic operation type performed on non coordinate variables see Section 3 35 Operation Types page 107
8. 0 c eee ee eee 161 4 1 20 GSL interpolation 0 0 cece e 169 4 1 21 GSL least squares fitting 0 cece ee eee ee 170 4 1 22 GSE statistics cci easiness pee eee ME TS 172 4 1 23 GSL random number generation 2000 174 4 1 24 Examples ncap2 ssssseessse eh 176 4 1 25 Intrinsic mathematical methods 0 00 178 4 1 26 Operator precedence and associativity 2 180 4 1 27 ID Quoting s i dee eee a REED RR 181 4 2 ncatted netCDF Attribute Editor 2005 183 4 3 ncbo netCDF Binary Operator esses 189 4 4 nces netCDF Ensemble Statistics 0 0 c eee eee 194 4 5 ncecat netCDF Ensemble Concatenator 045 197 4 6 ncflint netCDF File Interpolator 004 200 4 7 ncks netCDF Kitchen Sink 0 e eee eee 203 Options specific to NCKS 2 eee eee ens 204 AZ Filters for ckS cose a Eben eis 211 4 8 ncpdq netCDF Permute Dimensions Quickly 217 Packing and Unpacking Functions 0 sees e eee 217 Dimension Permutation 0 cece cece eee eens 220 4 9 ncra netCDF Record Averager 00 cece eee eee eee 225 4 10 ncrcat netCDF Record Concatenator 004 228 4 11 ncrename netCDF Renamer esses 230 4 12 ncwa netCDF Weighted Averager sssssssessssss 235 4 12 1 Mask condition sssessssesesese
9. As of NCO version 4 4 9 released in May 2015 ncra accepts user specified weights with the w or long option equivalent wgt wgt_var or weight switch When no weight is specified ncra weights each record e g time slice in the input files equally ncra does not attempt to see if say the time coordinate is irregularly spaced and thus would require a weighted average in order to be a true time average Weights specified with w wgt may take one of two forms In the first form the wgt argument is a comma separated list of values by which to weight each file Thus the number of values must equal the number of files specified in the input file list or else the program will exit In the second form the wgt argument is the name of a weighting variable present in every input file The variable may be a scalar or a one dimensional record variable Scalar weights are applied uniformly to the entire file i e a per file weight One dimensional weights apply to each corresponding record i e per record weights and are suitable for dynamically changing timesteps By default any weights specified whether by value or by variable name are nor malized to unity by dividing each specified weight by the sum of all the weights This means for example that w 0 25 0 75 is equivalent to w 2 0 6 0 since both are equal when normalized This behavior simplifies specifying weights based on co
10. Chapter 3 Shared Features 37 NCO obtains remote files from the OPeNDAP server e g www cdc noaa gov rather than the local machine Input files are first copied to the local machine then processed The OPeNDAP server performs data access hyperslabbing and transfer to the local machine This allows the I O to appear to NCO as if the input files were local The local machine performs all arithmetic operations Only the hyperslabbed output data are transferred over the network to the local machine for the number crunching to begin The advantages of this are obvious if you are examining small parts of large files stored at remote locations Natually there are many versions of OPeNDAP servers supplying data and bugs in the server can appear to be bugs in NCO However with very few exceptions an NCO command that works on a local file must work across an OPeNDAP connection or else there is a bug in the server This is because NCO does nothing special to handle files served by OPeNDAP the whole process is supposed to be completely transparent to the client NCO software Therefore it is often useful to try NCO commands on various OPeNDAP servers in order to isolate whether a problem may be due to a bug in the OPeNDAP server on a particular machine For this purpose one might try variations of the following commands that access files on public OPeNDAP servers Strided access to HDF5 file ncks v Time d Time 0 10 2 http eosdap hdfgroup
11. MERRA300 prod assim inst3_3d_asm_Cp 20130601 hdf Type LLC PPC Pck Size H4 H4 N4 N7 N4 N4 N3 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 N4 7 5 B1 5 5 B1 B9 O e erererrrrrere 244 244 214 210 215 214 617 694 223 207 347 133 223 225 221 201 185 150 100 DOWFPRRFDARFPWNDWKRFPAWAKL AD IN W 100 100 87 86 88 87 252 284 91 84 142 54 91 92 90 82 75 61 41 OWOPRPORARFPFWNRrFPOWODONN AF O Flags and Notes Original is compressed bzip2 1 bzip2 bzip2 L O0 L 1 L 9 ncpdq ncpdq ppe ppe ppe ppe ppe ppe ppe 1 L O0 L f default 7 default 6 default 5 default 4 default 3 default 2 default 1 OMI Aura_L2 OMIAuraS02_2012m1222 044888_v01 00 2014m0107t114720 h5 Flags and Notes Original is compressed Type LLC PPC Pck H5 H5 E et ot R ge Gm am gm Gm BS ROA zu DS gt BS 5 B1 5 B1 B9 or kee Size 29 29 29 29 29 50 29 29 27 12 29 29 27 5 3 5 3 3 7 8 4 7 9 7 7 3 100 99 100 99 99 172 101 99 94 43 100 100 92 00 Y3 0O0O0UyR gt oOOoOoO O bzip2 1 bzip2 1 bzip2 9 L 0 bo L 9 ncpdq ncpdq L 0 zb TIPPS ppe T PPS default 7 default 6 default 5 Chapter 3 Shared Features 101 N4 1 4 23 8 80 7 ppc default 4 N4 1 3 20 3 69 0 ppc
12. 75 rgr lon_grd lon_grd eee e eens 70 rgr lon nbr lon nbr ss 72 General Index rgr snwe lat sth lat nrt lon wst lon est onn docu des nete D erum d Errem M cate qas e 72 rgr wesn lon_wst lon_est lat_sth lon_nrt MEME ae Bes tte ec ah eee Reacts T2 Perera pus t ess etd ig td eal eee ee fee ibas 73 TUSBEL IDESGIIDL Ri bite te gee eM Aah ee eee 74 AT S6 eis uvis wo a dec AA eot 74 HPA MA a a AS 209 o e A A e D tae aed I es 115 rth flt iicuuogu dou lli Era Eea ag Mees 115 SSD A qae 3T PAC ies 4 eu dis loo a4 ol ee tbt i um ed 2d 209 AECA LA 130 TIBCTIPyt LV a ia 130 SL TON 209 AA A Meee ech 209 214 A il ig DenbPPPRE ROI SR ike tices 130 Fi etd te uw dua E 209 SEBAS o A eek eme ERG 209 214 suppress global metadata 198 Sthrenbr thr nbr ua Es 27 threads thr nbr cee 27 ylm_nm ulm nm eee 197 SRUNLON oes ie ies toe tain peas hiana oei 43 45 FAUD ES kbd oes LARA a za 209 SUT iced rimo bate e enh dest CRM ENTRE 43 45 Unpaid eiue See ext 218 mUpEGoiliilag2 bbb RE De IRA et ieee ET 218 WVar Val io cad IRL NAR ee ees 209 215 S Varlabletvar yd ccion vers wer reed 43 213 7VOerSiOn sisseiite e RE RETE 16 127 TUVESSSA aider ve DNGPRE E duris UT 16 127 weight weight segta irise pne ets 235 weight wgtil wgt2 sess 200 wgt var weight 6 cece eee eens 235 wgt var wgti wgt2
13. 8192 E gw stands for Gaussian weight in many climate models Chapter 4 Reference Manual 239 Compute the area mean over the globe using only points for which ORO lt 0 5 ncwa B ORO 0 5 W gw a lat lon in nc out nc ncwa m ORO M 0 5 T 1t w gw a lat lon in nc out nc It is considerably simpler to specify the complete mask cond with the single string argument to B than with the three separate switches m T and M If in doubt enclose the mask_cond within quotes since some of the comparators have special meanings to the shell Assuming 70 of the gridpoints are maritime then here the tally is 0 70 x 8192 5734 Compute the global annual mean over the maritime tropical Pacific ncwa B ORO lt 0 5 w gw a lat lon time d 1at 20 0 20 0 d lon 120 0 270 0 in nc out nc ncwa m ORO M 0 5 T 1t w gw a lat lon time d 1at 20 0 20 0 d lon 120 0 270 0 in nc out nc Further examples will use the one switch specification of mask cond Determine the total area of the maritime tropical Pacific assuming the variable area contains the area of each gridcell ncwa N v area B ORO lt 0 5 a lat lon d 1at 20 0 20 0 d lon 120 0 270 0 in nc out nc Weighting area e g by gw is not appropriate because area is already area weighted by definition Thus the N switch or equivalently the y ttl switch correctly integrate the cell areas into a total regional ar
14. Equi Angular grid 180x360 rgr grid DATA grids 180x360_SCRIP rgr lat nbr 180 rgr lon nbr 360 rgr lat typ uni rgr lon typ grn ctr zender nco data in nc foo nc 129x256 CAM FV grid first longitude centered at Greenwich ncks rgr grd ttl CAM FV scalar grid 129x256 rgr grid DATA grids 129x256_SCRIP Chapter 3 Shared Features 73 rgr lat_nbr 129 rgr lon nbr 256 rgr lat typ fv rgr lon_typ grn_ctr zender nco data in nc foo nc 128x256 Equi Angular grid Greenwich west edge of first longitude This is the CAM FV offset grid for the 129x256 CAM FV scalar grid above ncks rgr grd_ttl Equi Angular grid 128x256 rgr grid DATA grids 128x256_SCRIP rgr lat nbr 128 rgr lon nbr 256 rgr lat typ uni rgr lon typ grn wst zender nco data in nc foo nc T42 Gaussian grid first longitude centered at Greenwich ncks rgr grd_ttl T42 Gaussian grid rgr grid DATA grids t42_SCRIP 20150901 nc rgr lat nbr 64 rgr lon nbr 128 rgr lat typ gss rgr lon typ Grn ctr zender nco data in nc foo nc 3 23 Regridding T Availability ncks Short options None Long options map map file or rgr map map file rgr key val multiple invocations allowed rnr wgt thr or rgr rnr wgt thr or renormalize wgt thr As of NCO version 4 5 0 June 2015 ncks supports horizontal regridding of global datasets where the grids and weights are all store
15. for yyyy in 1990 2005 do Loop over years for moy in 1 12 do Loop over months mm printf 02d moy Change to 2 digit format Average specific month yyyy mm ncra 0 d time yyyy mm 01 yyyy mm 31 in nc in_ yyyy mm nc done done Concatenate monthly files together Annual average for yyyy in 1990 2005 do Loop over years ncra 0 d time yyyy 01 01 yyyy 12 31 in nc in_ yyyy nc done Concatenate annual files together The 0 switch means to overwrite the pre existing files see Section 3 37 Batch Model page 121 The d option is to specify the range of hyperslabs see Section 3 15 Hyperslabs page 58 There are detailed instructions on ncra see Section 4 9 ncra netCDF Record Averager page 225 and ncrcat see Section 4 10 ncrcat netCDF Record Concatenator page 228 NCO supports UDUnits so that we can use readable dates as time dimension see Section 3 24 UDUnits Support page 79 6 2 Monthly data in one file Inside the input file in nc the record dimension time is from Jan 1990 to Dec 2005 Seasonal average e g DJF ncra 0 mro d time 1990 12 01 12 3 in nc out nc Annual average 246 NCO 4 5 3 alpha03 User Guide ncra 0 mro d time 12 12 in nc out nc Here we use the subcycle feature i e the number after the fourth comma 3 in the seasonal example and the second 12 in the annual example to retrieve groups of records separ
16. sees 242 lossy CompresSion 00 000 eee eee eee ee 91 lrint 2 105 A ie E REUS 115 lround i eo tea Sabet REPE ER Ep 115 DIS Dist tess active MEA ERATUM NES ET EU 92 Duk Claes tein sie diy a Sess ods es 242 M labs oteisele RUMP ad eee thats 107 Y abs sitenieigerdy rd era pia 146 Macintosh esiosa eee id ed 8 Makefiles it isin ted eae RET 8 10 35 malloc is eror a a e E a a 25 mantissa isins din dr pda 120 manual type conversion 000 113 MAPA ti 73 Map Am da 88 MP A Tu ended ias 88 map NCA nn n b EIEEE RETE SE Pep su en UE OY 88 Map PEA ia 88 Jnapardi esee a 88 Tap crew nil pieni a Rd 88 map RS EOE 88 ap XSbt IN 88 mapping files iii io GE MS ES 76 294 Mark Plann r 2 22c rS TREE Rcs 242 Markus Liebig 2 II Recht ERR tes 242 Martin DI EE DRE 242 Martin Otte unesco Ar 243 Martin Schmidt 00 eee eee eee eee 242 Martin Schultz 2 eee ee eee 54 maskin keea ese dos 157 177 mask condition esse ee eee 236 239 masked average 0 cece eee e eee eee 235 Mass Store System 0 000 e eee 33 Matej Vela ito ad ERU ives 242 mathematical functions 66 178 Matlab soo tile he Re IN eed 17 MAX ese snk ew Alea ge had titer 107 Ma ses btt ato 146 MAXIMUM A A eE 107 MDS dE uan nas E 102 MM E A AS ws 107 MODOS ii E E e a E TH IRE 107 mebs uincorsrkhteecituuvers wet Ele RW 146 memory available o ooom oo
17. snc time lat lon snd time lat lon historical CCSM4 dt dk dk dt dt dt db db dt dt db db OH OF OF 252 NCO 4 5 3 alpha03 User Guide CCSMA 00 1 snc time lat lon snd time lat lon CCSM4_01 snc time lat lon snd time lat lon CCSM4_02 CCSM4_03 CCSMA 04 4 CESM1 BGC 1 CESM1 BGC 00 CESM1 CAM5 riilpi CESM1 CAMB O1 CESM1 CAMB 02 d 3k dk dt HH db di dt di dt dt di dt di dk dt Gi dk db db dt OH OH OH ot vw v Chapter 7 CMIP5 Example 253 7 2 Global Distribution of Long term Average Global Annual Average Snow Cover Fraction 96 bcc csmi 1 EH a o I EENENN 10 3040 6070 9095 Figure 7 1 Global Distribution of Long term Average This section illustrates how to calculate the global distribution of long term average see Figure 7 1 with either flat files or group file http nco sourceforge net nco html index groups Key steps include 1 Average ensemble members of each model using nces see Section 4 4 nces netCDF Ensemble Statistics page 194 2 Average the record dimension using ncra see Section 4 9 ncra netCDF Record Aver ager page 225 3 Store results of each model as a distinct group in a single output file using ncecat see Section 4 10 ncrcat netCDF Record Concatenator page 228 with the gag option The first example shows how to process flat files bin bash 254 dk dk dk
18. 0 0 eee eee 133 Ditimask soe sis ic Aaa OCA er Ra t RR 93 le e 208 DIO CKSIZE caca y DOM TRE nai 86 bounds 5 5 ax Wear pet P PRAE e 75 bounds attribute 93 125 215 bounds convention 0 0 e eee eee eee 125 Bourne Shell 0 0 ee eee eee eee 61 193 Brian Mays s scceeuvnnshe nie 242 broadeasting 222 sae Mad te b 192 broadcasting groupS 005 12 191 192 broadcasting variables 132 190 202 235 BSD i toda aa dann eles 29 B ffer sizes po nudo Secrets tan Rees wee REUS lala 103 PA fale rete tee s e LERS ag tpud 26 bugs reporting c os ete cet pide eta 15 Burrows Wheeler algorithm 98 Dti su efe pixel ERU re EY 148 A Sates eR ul ute S eara 98 C CELOS asda usce du noe ected aon Sal eus 8 C index convention seen eee eee 57 C language 9 85 114 115 131 186 209 C Shelb zz acide idad e apis 61 193 A 8 CA OPMBE as a tie deque aedes 26 BD ncs tros doles AE ade dep dede aa 8 9 COON den e CPP 9 10 94 CAM creda wala ae tad tao neg estes 101 CQAMEEV grid y ha RE Re edna 69 CAMB isan ETE 118 CC has weld Aa hee Nerio dea hale e do 8 Ca TP 8 CCM Processor llseseeeeees 30 226 228 CES Mica ited aide ieee oris t 243 275 CCSM conventions sees 124 CDD cuui e baee x een Er aA 206 Gell eii mrt supe ane repr Ta RUM 178 ceiling function 00 0 22 178 cell methods convention
19. 273 1 273 2 273 3 273 4 273 5 273 6 273 7 273 8 273 9 274 tpt flt 273 1 273 2 273 3 273 4 273 5 273 6 273 7 273 8 273 9 274 Create file with five ten month years i e 50 timesteps of temperature data ncrcat 0 v tpt flt tpt dbl p nco data in nc in nc in nc in nc in nc foo nc 32 See page 21 in Section 1 2 of the First edition for this gem One does not need much experience in scientific computing to recognize that the implicit conversion rules are in fact sheer madness In effect they make it impossible to write efficient numerical programs 33 For example the CMIP5 archive tends to distribute monthly average timeseries in 50 year chunks Chapter 3 Shared Features 117 Average ist five Januaries elements 1 11 21 31 41 ncra flt 0 F d time 1 10 foo nc foo_avgi nc Average 2nd five Januaries elements 2 12 22 32 42 ncra flt O F d time 2 10 foo nc foo avg2 nc Average 3rd five Januaries elements 3 13 23 33 43 ncra flt 0 F d time 3 10 foo nc foo avg3 nc Method 1 Obtain ensemble January average by averaging the averages ncra flt 0 foo avgi nc foo avg2 nc foo avg3 nc foo avg mthi nc Method 2 Obtain ensemble January average by averaging the raw data Employ ncra s subcycle feature http nco sf net nco html ssc ncra flt O F d time 1 10 3 foo nc foo avg mth2 nc Difference the two methods ncbo 0 foo avg mth
20. Chapter 4 Reference Manual 231 Note that renaming a dimension to the name of a dependent variable can be used to invert the relationship between an independent coordinate variable and a dependent variable In this case the named dependent variable must be one dimensional and should have no missing values Such a variable will become a coordinate variable According to the netCDF User Guide renaming objects in netCDF files does not incur the penalty of recopying the entire file when the new name is shorter than the old name Thus ncrename may run much faster at least on netCDF3 files if judicious use of header padding see Section 3 2 Metadata Optimization page 27 was made when producing the input file Similarly using the hdr_pad option with ncrename helps ensure that future metadata changes to output file occur as swifly as possible OPTIONS a old name new name Attribute renaming The old and new names of the attribute are specified with a or attribute by the associated old name and new name values Global attributes are treated no differently than variable attributes This option may be specified more than once As mentioned above all occurrences of the attribute of a given name will be renamed unless the form is used with one exception To change the attribute name for a particular variable specify the old name in the format old var nameQold att name The symbol delimits the v
21. NCO User Guide A suite of netCDF operators Edition 4 5 3 alpha03 for NCO Version 4 5 3 alpha03 September 2015 by Charlie Zender Departments of Earth System Science and of Computer Science University of California Irvine Copyright 1995 2015 Charlie Zender This is the first edition of the NCO User Guide and is consistent with version 2 of texinfo tex Published by Charlie Zender Department of Earth System Science 3200 Croul Hall University of California Irvine Irvine CA 92697 3100 USA Permission is granted to copy distribute and or modify this document under the terms of the GNU Free Documentation License Version 1 3 or any later version published by the Free Software Foundation with no Invariant Sections no Front Cover Texts and no Back Cover Texts The license is available online at http www gnu org copyleft fdl html The original author of this software Charlie Zender wants to improve it with the help of your suggestions improvements bug reports and patches Charlie Zender surname at uci dot edu yes my surname is zender Department of Earth System Science 3200 Croul Hall University of California Irvine Irvine CA 92697 3100 Table of Contents Foreword 1d dcos tO pietate etes dels dates 1 SUDAR A Met Re 9 Ll Introduction Fa e ri Mj RC ut T Tad Availability ica eter tI eec RR CE ERR orate E EPOR 7 1 2 How to Use This Guide 0 0 cee eee eee eee eee 7 1 8 Operating systems
22. ans between 7 2 and 7 2 Example atanh 3 14159265358979323844 1 0 Ceil Ceiling of x Smallest integral value not less than argument Example ceil 0 1 1 0 Cosine Cosine of x where x is specified in radians Example cos 0 0 1 0 Hyperbolic cosine Hyperbolic cosine of x where x is specified in radians Ex ample cosh 0 0 1 0 Error function Error function of x where x is specified between 1 and 1 Example erf 1 0 0 842701 Complementary error function Complementary error function of x where x is specified between 1 and 1 Example erfc 1 0 0 15729920705 x Exponential Exponential of X e exp 1 0 2 71828182845904523536 Example Floor Floor of x Largest integral value not greater than argument Example floor 1 9 1 Gamma function Gamma function of x T x The well known and loved con tinuous factorial function Example gamma 0 5 yr gamma_inc_P x 1n x log x log10 x Incomplete Gamma function Incomplete Gamma function of parameter a and variable x P a x One of the four incomplete gamma functions Example gamma inc P 1 1 1 e Natural Logarithm Natural logarithm of x ln z Example In 2 71828182845904523536 1 0 Natural Logarithm Exact synonym for 1n x Base 10 Logarithm Base 10 logarithm of x log a Example log 10 0 1 0 nearbyint x pow x y rint x round x sin x Round inexactly Nearest integer to x is returned in floating p
23. dbl 0 F d time 1 10 foo nc foo_avgi nc Average 2nd five Januaries elements 2 12 22 32 42 ncra dbl 0 F d time 2 10 foo nc foo avg2 nc Average 3rd five Januaries elements 3 13 23 33 43 ncra dbl 0 F d time 3 10 foo nc foo avg3 nc Method 1 Obtain ensemble January average by averaging the averages ncra dbl 0 foo avgi nc foo avg2 nc foo avg3 nc foo avg mth nc Method 2 Obtain ensemble January average by averaging the raw data Employ ncra s subcycle feature http nco sf net nco html ssc ncra dbl 0 F d time 1 10 3 foo nc foo avg mth2 nc Difference the two methods ncbo 0 foo avg mthi nc foo avg mth2 nc foo avg dff nc Show differences 118 NCO 4 5 3 alpha03 User Guide ncks cdl foo avg dff nc tpt dbl 5 6843418860808e 14 tpt flt 0 The dbl switch has no effect on the results computed from double precision inputs But now the two methods produce bit for bit identical results from the single precision inputs This is due to the happenstance of rounding along with the effects of the db1 switch The flt and rth flt switches are provided for symmetry They enforce the traditional NCO and Fortran convention of keeping single precision arithmetic in single precision unless a double precision number is explicitly involved We have shown that forced promotion of single to double precision prior to arith metic has advant
24. long version command line option ncks D 3 in nc Short option ncks dbg_lvl 3 in nc Long option preferred form ncks dbg_lvl 3 in nc Long option alternate form The last example is preferred for two reasons First dbg_1v1 is more specific and less ambiguous than D The long option form makes scripts more self documenting and less error prone Often long options are named after the source code variable whose value they carry Second the equals sign joins the key i e Jong name to the value in an unin terruptible text block Experience shows that users are less likely to mis parse commands when restricted to this form GNU implements a superset of the POSIX standard which allows any unambiguous trun cation of a valid option to be used ncks D 3 in nc Short option ncks dbg_lvl 3 in nc Long option full form 1 Ifa getopt_long function cannot be found on the system NCO will use the getopt_long from the my_getopt package by Benjamin Sittler bsittler iname com This is BSD licensed software available from http www geocities com ResearchTriangle Node 9405 my_getopt 30 NCO 4 5 3 alpha03 User Guide ncks dbg 3 in nc Long option unambiguous truncation ncks db 3 in nc Long option unambiguous truncation ncks d 3 in nc Long option ambiguous truncation The first four examples are equivalent and will work as expected The final example will exit with an error since ncks cannot disambiguat
25. msk cmp typ mask comp ssssse 236 msk cnd mask cond sss 235 msk_cnd_sng mask_cond 236 msk nm maSk_var ccc ccc eee eee 235 msk val maSk_val ccc ccc cee 236 s msk var Mask Var ince ree cide a wees bs 235 el A eshte bib besdeielaus 208 SOMA i sin vas eile ees A 207 multiple_record_dimensions 83 CH CI zu uir tol dene MPG E Se dts 210 mnetcdf4 lul up xa laca owes Y rd 38 Hnintap loop tii eet Bae asa bos 30 SSnOabLank coc viii sawn tak remm oa ars od 208 SNO COOTdS om ys ada heed oi hs oes 47 126 SCUOP CIdz s lica as EE RON ere RE E 47 126 nozblank xia eee prete ee 208 214 no cell methods 0 cece eee eee 126 NCO 4 5 3 alpha03 User Guide OL CLM MEN sisson nee Cais ne ay Mee ers naa ES 126 z no dmn var nm cssseeke eee TP Pu 209 215 no glb mtd i pei di ete VS 198 hOo cnm pfni a 2se 4 etm ey ieee et 209 215 no rec dmn diM o oocoocoooo do minara 205 HSNO tmp f esis cie E xS 18 105 EMO OA 6 di Ron Oo eee wan hh Rs 208 AAA RU LSQexrstwedbet 194 SSNS grpisuiedete Me BER aede etude m adi cs 194 NSM ASE Kae e 5 Die dea ne 09b e M ares e 8 194 PODS Ka dh eins Heed Ce be ede se Pee ed ls i alan 45 omp num threads thr nbr 27 op rlt mask comp esses 236 op typ op EYP ir Er ETENEE eee eee 107 189 0pen ram 5 cue par esis EL 18 24 104 operation OP Piscina nina 10
26. ncrng var nm fl nm What is range of variable function ncrng ncap2 0 C v s foo min 1 min foo max 11 max print foo mi ncmode var nm f1 nm What is mode of variable function ncmode ncap2 0 C v s foo gsl stats median from sorted data 1 sort ncrecsz f1 nm What is record dimension size function ncrecsz 1 ncks M 1 grep E i Record dimension cut f 8 d Csh shell bin csh users place these in cshrc ncattget ncks M m 13 grep E i 2 attribute 0 9 11 cut f 11 ncdmnsz ncks m M 12 grep E i 11 size cut f 7 d unig gt nclist ncks m 11 grep E type cut f 1 d sed s sort ncrecsz ncks M 1 grep E i Record dimension cut f 8 d Sh shell bin sh users place these in profile ncattget ncks M m 13 grep E i 2 attribute 0 9 11 cut f 11 ncdmnsz ncks m M 42 grep E i 11 size cut f 7 d unig nclist ncks m 11 grep E type cut f 1 d sed s sort ncrecsz ncks M 1 grep E i Record dimension cut f 8 d EXAMPLES View all data in netCDF in nc printed with Fortran indexing conventions ncks F in nc Copy the netCDF file in nc to file out nc ncks in nc out nc Now the file out nc contains all the data from in nc There are however tw
27. page 30 for an explanation of the distinctions between these methods Chapter 4 Reference Manual 229 Assume the files 85 nc 86 nc 89 nc each contain a record coordinate time of length 12 defined such that the third record in 86 nc contains data from March 1986 etc NCO knows how to hyperslab the record dimension across files Thus to concatenate data from December 1985 February 1986 ncrcat d time 11 13 85 nc 86 nc 87 nc 8512 8602 nc ncrcat F d time 12 14 85 nc 86 nc 87 nc 8512 8602 nc The file 87 nc is superfluous but does not cause an error When ncra and ncrcat en counter a file which does contain any records that meet the specified hyperslab criteria they disregard the file and proceed to the next file without failing The F turns on the Fortran 1 based indexing convention The following uses the stride option to concatenate all the March temperature data from multiple input files into a single output file ncrcat F d time 3 12 v temperature 85 nc 86 nc 87 nc 858687 03 nc See Section 3 16 Stride page 60 for a description of the stride argument Assume the time coordinate is incrementally numbered such that January 1985 1 and December 1989 60 Assuming only expands to the five desired files the following concatenates June 1985 June 1989 ncrcat d time 6 54 nc 8506 8906 nc 230 NCO 4 5 3 alpha03 User Guide 4 11 ncrename netCDF Renamer SYNTAX ncrename a old name new na
28. 0 temp idx jdx 0 0 Are values of co ordinate variable double lat lat monotonic sz lat size for idx 1 idx lt sz idx if lat idx lat idx 1 lt 0 0 break if idx sz print lat co ordinate is monotonic n else print lat co ordinate is NOT monotonic n 152 NCO 4 5 3 alpha03 User Guide Sum odd elements idx 0 sz lat_nw size sum 0 0 while idx lt sz if lat idx 2 sum lat idx idx ram_write sum print Total of odd elements print sum print n 4 1 15 Include files The syntax of an include file is include script nco The script filename is searched relative to the run directory It is possible to nest include files to an arbitrary depth A handy use of inlcude files is to store often used constants Use RAM variables if you do not want these constants written to output file script nco Sample file to include in ncap2 script pi 3 1415926535 RAM variable not written to output h 6 62607095e 34 RAM variable not written to output e 2 71828 Regular disk variable written to output 4 1 16 sort methods In ncap2 there are multiple ways to sort data Beginning with NCO 4 1 0 March 2012 ncap2 support six sorting functions var out sort var in amp srt map Ascending sort var out asort var in amp srt map Accending sort var out dsort var in amp srt map Desending sort var out remap var in srt map Apply srt
29. 95 93 75 92 5 91 25 90 88 75 87 5 86 25 85 83 75 82 5 81 25 80 78 75 77 5 76 25 75 73 75 72 5 71 25 70 68 75 67 5 66 25 65 63 75 62 5 61 25 60 58 75 57 5 56 25 55 53 75 52 5 51 25 50 48 75 47 5 46 25 45 43 75 42 5 41 25 40 38 75 37 5 36 25 35 33 75 32 5 31 25 30 28 75 27 5 26 25 25 23 75 22 5 21 25 20 18 75 17 5 16 25 15 13 75 12 5 11 25 10 8 75 7 5 6 25 5 3 75 2 5 1 25 0 1 25 2 5 3 75 5 6 25 7 5 8 75 10 11 25 12 5 13 75 15 16 25 17 5 18 75 20 21 25 22 5 23 75 25 26 25 27 5 28 75 30 31 25 32 5 33 75 35 36 25 37 5 38 75 40 41 25 42 5 43 75 45 46 25 47 5 48 75 50 51 25 52 5 53 75 55 56 25 ZA A A A AA A A Chapter 7 CMIP5 Example 7 6 Main 269 57 5 58 75 60 61 25 62 5 63 75 65 66 25 67 5 68 75 70 71 25 72 5 73 75 75 76 25 77 5 78 75 80 81 25 82 5 83 75 85 86 25 87 5 88 75 90 91 25 92 5 93 75 95 96 25 97 5 98 75 100 101 25 102 5 103 75 105 106 25 107 5 108 75 110 111 25 112 5 113 75 115 116 25 117 5 118 75 120 121 25 122 5 123 75 125 126 25 127 5 128 75 130 131 25 132 5 133 75 135 136 25 137 5 138 75 140 141 25 142 5 143 75 145 146 25 147 5 148 75 150 151 25 152 5 153 75 155 156 25 157 5 158 75 160 161 25 162 5 163 75 165 166 25 167 5 168 75 170 171 25 N 172 5 173 75 175 176 25 177 5 178 75 180 Copy of CCSM4 longitude out time 1atn 10nn 20 0 Output structure Bi linear interpolation bsa sw less biline
30. Add new grp att group attribute to single group ncatted 0 a gi_grp_att gi c sng new group attribute in grp nc Add new glb att global attribute to root group ncatted O a new_glb_att global c sng new global attribute in grp nc Note that regular expressions work well in conjuction with group path support In other words the variable name including group path component and the attribute names may both be extended regular expressions Demonstrate input of C language escape sequences e g Nn and other special characters e g ncatted h a special global o c nDouble quote nTwo consecutive double quotes n Single quote Beyond my shell abilities nBackslash n Two consecutive backslashes nQuestion mark n in nc Note that the entire attribute is protected from the shell by single quotes These outer single quotes are necessary for interactive use but may be omitted in batch scripts Chapter 4 Reference Manual 189 4 3 ncbo netCDF Binary Operator SYNTAX ncbo 3 4 6 7 A C c cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stridel1 F G gpe_dsc g grp glb h hdr_pad nbr L dfl lvl 1 path no tmp f1 0 o file 3 p path R r ram al1 t thr nbr unn v var X x y op typl file 1 file 2 file 3 DESCRIPTION ncbo perf
31. INT64 and NC_UINT64 Eight byte integer support is an espe cially useful improvement from netCDF3 All NCO operators support these types e g ncks copies and prints them ncra averages them and ncap2 processes algebraic scripts with them ncks prints compression information if any to screen NCO version 3 9 1 June 2007 added support for netCDF4 Lempel Ziv deflation Lempel Ziv deflation is a lossless compression technique See Section 3 30 Deflation page 101 for more details NCO version 3 9 9 June 2009 added support for netCDF4 chunking in ncks and ncecat NCO version 4 0 4 September 2010 completed support for netCDF4 chunking in the remaining operators See Section 3 28 Chunking page 85 for more details NCO version 4 2 2 October 2012 added support for netCDF4 groups in ncks and ncecat Group support for these operators was complete e g regular expressions to select groups and Group Path Editing as of NCO version 4 2 6 March 2013 See Section 3 13 Group Path Editing page 48 for more details Group support for all other operators was finished in the NCO version 4 3 x series completed in December 2013 T One must link the NCO code to the HDF4 MFHDF library instead of the usual netCDF library Apparently MF stands for Multi file not for Mike Folk In any case until about 2007 the MFHDF library only supported netCDF2 calls Most people will never again install NCO 1 2 x and so will never use NCO to write
32. It is safest to write scripts using mask_condition Chapter 4 Reference Manual 237 the i th element of the hyperslab to be averaged the mask condition is mask mask_comp mask_val Each z is also associated with an additional weight w whose value may be user specified The value of w is identically 1 unless the user specifies a weighting variable weight with w weight or wgt_var In this case the value of w is determined by the weight variable in the input file As noted above weight is broadcast if possible to conform to the variable being averaged M is the number of input elements x which actually contribute to output element z M is also known as the tally and is defined as i N M 2 M um i 1 M is identical to the denominator of the generic averaging expression except for the omission of the weight w Thus M N whenever no input points are missing values or are masked Whether an element contributes to the output and thus increments M by one has more to do with the above two criteria missing value and masking than with the numeric value of the element per se For example x 0 0 does contribute to x assuming the _FillValue attribute is not 0 0 and location 7 is not masked The value x 0 0 will not change the numerator of the generic averaging expression but it will change the denominator unless its weight w 0 0 as well 4 12 2 Normalization and Integration ncwa has o
33. Mask flag and sort map are same size as variable of interest msk flg var in Chapter 4 Reference Manual 155 srt map var in In this example we mask for all values evenly divisible by 3 This is the key problem specific portion of the template Replace this where condition by that for your problem Mask variable is Boolean 1 Meets condition O Fails condition where var_in 3 0 msk flg 1 elsewhere msk flg 0 print msk flg print msk flg For debugging The sort routine is overloaded and takes one or two arguments The second argument optional is the sort map srt map below Pass the sort map by reference i e prefix with an ampersand If the sort map does not yet exist then it will be created and returned as an integer type the same shape as the input variable The output of sort on the LHS is a sorted version of the input msk flg is not needed in its original order after sort Hence we use msk flg as both input to and output from sort Doing this prevents the need to define a new unneeded variable msk flg sort msk flg amp srt map Count number of valid points in mask by summing the one s msk nbr msk flg total O Define output dimension equal in size to number of valid points defdim crd out msk nbr Now sort the variable of interest using the sort map and remap The output on the LHS is the input re arranged so that all points meeting the mask condition
34. NCO versions 4 3 6 and earlier do not explicitly support HDF4 yet should work with HDF4 if compiled with a version of netCDF 4 3 2 or later that does not unexpectedly die when probing HDF4 files with standard netCDF calls NCO versions 4 3 7 4 3 9 October December 2013 use a special flag to workaround netCDF HDF4 issues The user must tell these versions of NCO that an input file is HDF4 format by using the hdf4 switch When compiled with netCDF version 4 3 1 20140116 or later NCO versions 4 4 0 Jan uary 2014 and later more gracefully handle HDF4 files In particular the hdf4 switch is obsolete Current versions of NCO use netCDF to determine automatically whether the un derlying file is HDF4 and then take appropriate precautions to avoid netCDF4 API calls that fail when applied to HDF4 files e g nc inq var chunkingO nc inq var deflate When compiled with netCDF version 4 3 2 20140423 or earlier NCO will report that chunking and deflation properties of HDF4 files as HDF4 UNKNOWN because determining those properties was impossible When compiled with netCDF version 4 3 3 rc2 20140925 or later NCO versions 4 4 6 October 2014 and later fully support chunking and deflation features of HDF4 files The hdf4 switch is supported for backwards compatibility yet redundant i e does no harm with current versions of NCO and netCDF Converting HDF4 files to netCDF Since NCO reads HDF4 files native
35. Value s specified in att_val will be stored as netCDF intrinsic type NC INT64 ull uint64 Uint64 Value s specified in att val will be stored as netCDF intrinsic type NC UINT64 sng string String Value s specified in att val will be stored as netCDF intrinsic type NC STRING Note that ncatted handles type NC STRING attributes correctly beginning with version 4 3 3 released in July 2013 Earlier versions fail when asked to handle NC STRING attributes In Delete mode the specification of att type is optional and is ignored if supplied The value of att val is what you want to change attribute att nm to contain The specification of att val is optional in Delete and is ignored mode Attribute values for all types besides NC CHAR must have an attribute length of at least one Thus att val may 186 NCO 4 5 3 alpha03 User Guide be a single value or one dimensional array of elements of type att type If the att val is not set or is set to empty space and the att type is NC CHAR e g a units T o c or a units T o c then the corresponding attribute is set to have zero length When specifying an array of values it is safest to enclose att val in single or double quotes e g a levels T o s 1 2 3 4 or a levels T o s 1 2 3 4 The quotes are strictly unnecessary around att val except when att val contains characters which would confuse the calling shell such as spaces commas and wildcard characters NCO process
36. after panicking treat this as a clue to re try your commands with the hdf_upk switch See Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 for an easy tech nique to unpack data packed with the HDF convention and then re pack it with the netCDF convention Default Handling of Packed Data All NCO arithmetic operators understand packed data The operators automatically unpack any packed variable in the input file which will be arithmetically processed For example ncra unpacks all record variables and ncwa unpacks all variable which contain a dimension to be averaged These variables are stored unpacked in the output file On the other hand arithmetic operators do not unpack non processed variables For example ncra leaves all non record variables packed and ncwa leaves packed all variables lacking an averaged dimension These variables called fixed variables are passed unaltered from the input to the output file Hence fixed variables which are packed in input files remain packed in output files Completely packing and unpacking files is easily accomplished with ncpdq see Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 Pack and unpack individual variables with ncpdq and the ncap2 pack and unpack O functions see Section 4 1 11 Methods and functions page 145 3 35 Operation Types 108 NCO 4 5 3 alpha03 User Guide Availability ncap2 ncra nces ncwa Short options y Long opti
37. and missing points The WRF grid documentation should describe this For WRF files creating regional masks looks in general like mask var XLAT gt lat min amp amp XLAT lt lat max amp amp XLONG gt lon min amp amp XLONG lt lon m A few notes Irregular regions are the union of arrays lat lon_min max s The mask procedure is identical for all R 4 1 18 Bilinear interpolation As of version 4 0 0 NCO has internal routines to perform bilinear interpolation on gridded data sets In mathematics bilinear interpolation is an extension of linear interpolation for interpolating functions of two variables on a regular grid The idea is to perform linear interpolation first in one direction and then again in the other direction Suppose we have an irregular grid of data temperature lat lon with co ordinate vars lat lat lon lon We wish to find the temperature at an arbitary point X Y within the grid If we can locate lat min lat max and lon min lon max such that lat min lt X lt lat max and lon min lt Y lt lon max then we can interpolate in two dimensions the temperature at X Y The general form of the ncap2 interpolation function is var out bilinear interp grid in grid out grid out x grid out y grid in x grid in y where grid in Input function data Usually a two dimensional variable It must be of size grid in x size grid in y size O grid out This variable is the shape of var out Usually a
38. below and in the man pages was not written until July 2013 Multislabs overcome many restraints that limit simple hyperslabs A single d op tion can only specify a contiguous and or a regularly spaced multi dimensional data ar ray Multislabs are constructed from multiple d options and may therefore have non regularly spaced arrays For example suppose it is desired to operate on all longitudes from 10 0 to 20 0 and from 80 0 to 90 0 degrees The combined range of longitudes is not selectable in a single hyperslab specfication of the form d dimension min max or d dimension min max stride because its elements are irregularly spaced in coordinate space and presumably in index space too The multislab specification for obtaining these values is simply the union of the hyperslabs specifications that comprise the multislab i e ncks d lon 10 20 d lon 80 90 in nc out nc ncks d lon 10 15 d 10on 15 20 d lon 80 90 in nc out nc Any number of hyperslabs specifications may be chained together to specify the multislab MSA creates an output dimension equal in size to the sum of the sizes of the multislabs This can be used to extend and or pad coordinate grids Users may specify redundant ranges of indices in a multislab e g ncks d 10on 0 4 d 10on 2 9 2 in nc out nc This command retrieves the first five longitudes and then every other longitude value up to the tenth Elements 0 2 and 4 are specified by both hyp
39. dimensions i e latitude by longitude hyam is the weight given to surfaces of constant density one spatial dimension pressure which is orthogonal to the horizontal dimensions and hybm is the weight given to surfaces of constant elevation also one spatial dimension This command constructs a four dimensional pressure prs_mdp from the four input variables of mixed rank and orthogonality ncap2 s prs mdp time lat lon lev PO hyam PS hybm in nc out nc Manipulating the four fields which define the pressure in a hybrid coordinate system is easy with left hand casting 4 1 5 Arrays and hyperslabs Generating a regularly spaced one dimensional array with ncap2 is simple with the array O function The syntax is var out array val srt val ncr dmn nm One dimensional output Chapter 4 Reference Manual 137 var out array val srt val ncr var tpl Multi dimensional output where the arguments are the starting value val srt incremental value val ncr and for one dimensional output the single dimension dmn nm or for multi dimensional output a template variable var tpl i e a variable with the same shape as the desired output The type of var out will be the same as val srt Be sure to encode this type with the appropriate decimal point and floating point suffix when val srt is a naked constant rather than a variable For example to produce an array of shorts signed two byte integers integers signed four byte integ
40. diskless_all are convenient shortcuts for specify ing both create_ram and diskless_ram Thus ncks in nc out nc Default Open in nc on disk write out nc to disk ncks open ram in nc out nc Open in nc in RAM write out nc to disk Chapter 3 Shared Features 105 ncks create ram in nc out nc Create out nc in RAM write to disk Open in nc in RAM create out nc in RAM then write out nc to disk ncks open ram create ram in nc out nc ncks ram all in nc out nc Same as above ncks diskless all in nc out nc Same as above It is straightforward to demonstrate the efficacy of RAM disks For NASA we constructed a test that employs ncecat an arbitrary number set to one hundred thousand of files are all symbolically linked to the same file Everything is on the local filesystem not DAP Create symbolic links for benchmark cd DATA nco Do all work here for idx in 1 99999 do idx fmt printf 05d idx bin ln s DATA nco LPRM AMSR_E_L3_D_SOILM3_V002 20120512T111931Z_20020619 nc DATA nco idx_fmt nc done Benchmark time to ncecat one hundred thousand files time ncecat create ram 0 u time v ts d Latitude 40 0 d Longitude 105 0 p DATA nco n 99999 5 1 00001 nc foo nc Run normally on a laptop in 201303 this completes in 21 seconds The create ram reduces the elapsed time to 9 seconds Some of this speed may be due to using symlinks and caching However the ef
41. filberg c sce Y aimee e e RA Re RE RN RA 211 findgen equivalent 2 0000 136 fix record dimension 197 199 206 fixed dimension 197 199 205 206 Ilagss cte salto i 176 flatten groups 00 02 cee eee eee eee 49 flattening iio as a denne Bep 42 Plat sais ye uim a An 180 float ct Eere victae terri Du cw edt 148 MOOK AAE gato be gee eh dada Rie pec SMS 178 floor function 2452 eee Stes eed pP a EM 178 dlt byt e a Me RDUM ot Minho ened 218 FIF SET coule a ii tie ares 218 fOr ul hentia E 151 force append se ccnl beu tel dibus 121 force overwrite 0 0 0 cece eee eee eee 121 foreword aia 1 o iege A ep EIL 115 226 228 Fortran index convention sssssses 5T FORTRAN format 0 000 cece eee eens 26 Francesco Lovergine 0 0 cece eee 242 LED i Matar cia ameneeth idas 10 33 ETL ne dd Ru edet 37 A AN 243 EVER A E ERRARE 69 74 G AA A TEN 8 10 EO e a in 86 O 86 GAG iit ed aga emen lebe us 197 E AAA wie hee Eee aka MESS TENAN 9 178 gamma function cu sepdertie fy eb eter 161 178 Gary Strand susce retener ro 242 Gaussian grid 0 eee eee eee 69 74 Gaussian weight sees 76 Gaussian weights 0 0 0000 238 Gavin Burris 2 2 3 0 ani nero 242 Gayathri Venkitachalam 242 ECGunnoueuyoLet a O 8 10 COM E IA esf 17 101 118 geographical weight 00
42. gsl sf eta e Y gslsf eta dbl expr 4 1 20 GSL interpolation As of version 3 9 9 released July 2009 NCO has wrappers to the GSL interpolation func tions Given a set of data points xl yl xn yn the GSL functions computes a continuous interpolating function Y x such that Y xi 2 yi The interpolation is piecewise smooth and its behavior at the end points is determined by the type of interpolation used For more information consult the GSL manual Interpolation with ncap2 is a two stage process In the first stage a RAM variable is created from the chosen interpolating function and the data set This RAM variable holds in memory a GSL interpolation object In the second stage points along the interpolating function are calculated If you have a very large data set or are interpolating many sets then consider deleting the RAM variable when it is redundant Use the command ram delete var nm A simple example x_in lon 1 0 2 0 3 0 4 0 y_in lon 1 1 1 2 1 5 1 8 Ram variable is declared and defined here gsl_interp_cspline amp ram_sp x_in y_in x_out lon_grd 1 1 2 0 3 0 3 1 3 99 y_out gsl_spline_eval ram_sp x_out y2 gsl_spline_eval ram_sp 1 3 y3 gsl_spline_eval ram_sp 0 0 170 NCO 4 5 3 alpha03 User Guide ram delete ram sp print y out 1 10472 1 2 1 4 1 42658 1 69680002 print y2 1 12454 print y3 PINAR Note in the above example y3 is set to missing value because 0 0 i
43. i e with all supported HDF5 features like compression and chunking but without groups or new atomic types Operators given the 3 or 3 switch without arguments will attempt to produce netCDF3 CLASSIC output even from netCDF4 input files Note that NETCDF4 and NETCDF4 CLASSIC are the same binary format The latter simply causes a writing application to fail if it attempts to write a NETCDF4 file that cannot be completely read by the netCDF3 library Conversely NETCDF4 CLASSIC indicates to a reading application that all of the file contents are readable with the netCDF3 library NCO has supported reading writing basic NETCDF4 and NETCDF4 CLASSIC files since October 2005 3 9 2 Determining File Format Input files often end with the generic nc suffix that leaves perhaps by intention the internal file format ambiguous There are at least three ways to discover the internal format of a netCDF supported file These methods determine whether it is a classic 32 bit offset or newer 64 bit offset netCDF3 format or is a netCDF4 format Each method returns the information using slightly different terminology that becomes easier to understand with practice First examine the first line of global metadata output by ncks M 4 ncks M foo 3c nc Summary of foo 3c nc filetype NC FORMAT CLASSIC O groups ncks M foo 364 nc Summary of foo 364 nc filetype NC FORMAT 64BIT O groups 4 ncks M foo A4c nc
44. mm 31 23 59 9 9 drc_out f1ld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN nc Chapter 7 CMIP5 Example 265 drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _ all nsm_GN_ yr mm nc Subtract the annual average from the monthly data ncbo 0 op_typ drc_out fld_out var_id var var_id _ r1m _ md1 _ xpt 0 _all nsm_GN_ yr mm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _ all nsm_GN_ yr nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN_ yr mm _anm nc done Average over years ncra 0 drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN_ 7 mm _anm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN_ mm _anm nc done Concatenate months together ncrcat 0 drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN_ _anm nc drc_out f 1d_out var_id var var_id _ mdl1l nc echo Model md1 done done rm f drc_out fld_out var_id var var_id historical Store models as groups in the output file ncecat 0 gag v var var_id A drc_out fld_out var_id var var_id _ nc drc_out fld_out var_id var var_id _ rlm _all md1_ xpt 0 _all nsm_GN_mthly anm nc echo Var var var_id done done Using group feature and hyperslabs http nc
45. nco data in nc ncks INFO MD5 md5_a Occ175b9c0f1b6a831c399e269772661 md5_a a ncks INFO MD5 md5_abc Occi75b9cOf b6a831c399e269772661 lev 0 100 md5_abc 0 0 a In fact these examples demonstrate the validity of the hash algorithm since the MD5 hashes of the strings a and abc are widely known The second example shows that the hyperslab of variable md5 abc abc consisting of only its first letter a has Chapter 3 Shared Features 103 the same hash as the variable md5_a a This illustrates that MD5 digests act only on variable data not on metadata When activating MD5 digests with ncecat or ncrcat it is assumed that the user wishes to verify that every variable written to disk has the same MD5 digest as when it is subsequently read from disk This incurs the major additional overhead of reading in each variable after it is written and performing the hash algorithm again on that to compare to the original hash Moreover it is assumed that such operations are generally done production mode where the user is not interested in actually examining the digests herself The digests proceed silently unless the debugging level exceeds three gt ncecat 0 D 4 md5 p nco data in nc in nc foo nc grep MD5 ncecat INFO MD5 wnd_spd bec190dd944f2ce2794a7a4abf224b28 ncecat INFO MD5 digests of RAM and disk contents for wnd_spd agree gt ncrcat 0 D 4 md5 p nco data in nc i
46. of global and group metadata when copying subsetting or appending to an output file The various combinations of printing switches can be confusing In an attempt to anticipate what most users want to do ncks uses context sensitive defaults for printing Our goal is to minimize the use of switches required to accomplish the common operations We assume that users creating a new file or overwriting e g with 0 an existing file usually wish to copy all global and variable specific attributes to the new file In contrast we assume that users appending e g with A an explicit variable list from one file to another usually wish to copy only the variable specific attributes to the output file The switches H M and m switches are implemented as toggles which reverse the default behavior The most confusing aspect of this is that M inhibits copying global metadata in overwrite mode and causes copying of global metadata in append mode ncks in nc Print VAs and GAs ncks v one in nc Print VAs not GAs ncks M v one in nc Print GAs only ncks m v one in nc Print VAs only ncks M m v one in nc Print VAs and GAs ncks 0 in nc out nc Copy VAs and GAs ncks 0 v one in nc out nc Copy VAs and GAs ncks 0 M v one in nc out nc Copy VAs not GAs ncks 0 m v one in nc out nc Copy GAs not VAs ncks 0 M m v one in nc out nc Copy only data no atts ncks A in nc out nc Append
47. or cnk_sz_scl then the existing policy and map are retained and the output chunksizes are modified where necessary in accord with the user specified parameter When cnk_map is specified and cnk_plc is not then NCO picks what it thinks is the optimal chunking policy This has always been policy map_g2d When cnk_plc is specified and cnk_map is not then NCO picks what it thinks is the optimal chunking map This has always been map map_rd1 To start afresh and return to netCDF4 chunking defaults select cnk_map nc4 Simple chunking and unchunking ncks 0 4 cnk_plc all in nc out nc Chunk in nc ncks 0 4 cnk plc unchunk in nc out nc Unchunk in nc Chunk data then unchunk it printing informative metadata ncks 0 4 D 4 cnk plc all nco data in nc foo nc ncks 0 4 D 4 cnk plc uck foo nc foo nc Set total chunksize to 8192 B ncks 0 4 D 4 cnk plc all cnk byt 8192 nco data in nc foo nc More complex chunking procedures with informative metadata Chapter 3 Shared Features 91 ncks 0 4 D 4 cnk scl 8 nco data in nc foo nc ncks 0 4 D 4 cnk_scl 8 dstmch90 clm nc foo nc ncks 0 4 D 4 cnk dmn lat 64 cnk dmn lon 128 dstmch90 clm nc foo nc ncks 0 4 D 4 cnk plc uck foo nc foo nc ncks 0 4 D 4 cnk plc g2d cnk map rdi1 cnk dmn 1at 32 cnk dmn lon 128 dstmch90 clm 0112 nc foo nc Chunking works with al
48. sphF array gsl sf coulomb CL gsl sf coulomb CL array gsl sf coupling 3j gsl sf coupling 6j gsl sf coupling RacahW gsl sf coupling 9j gsl sf coupling 6j INCORRECT gsl_sf_dawson dbl_expr gsl_sf_debye_1 dbl_expr gsl_sf_debye_2 dbl_expr gsl_sf_debye_3 dbl_expr gsl_sf_debye_4 dbl_expr gsl_sf_debye_5 dbl_expr gsl_sf_debye_6 dbl_expr gsl_sf_dilog gsl_sf_complex_dilog_xy_e gsl_sf_complex_dilog gsl_sf_complex_spence_xy_e gsl_sf_multiply esl_sf_multiply_err esl_sf_ellint_Kcomp dbl_expr gsl_sf_ellint_Ecomp dbl_expr gsl_sf_ellint_Pcomp dbl_expr dbl_expr gsl_sf_ellint_Dcomp dbl_expr esl_sf_ellint_F dbl_expr dbl_expr esl_sf_ellint_E dbl_expr dbl_expr esl_sf ellint P dbl expr dbl expr dbl expr gsl sf ellint D dbl expr dbl expr dbl expr gsl_sf_ellint_RC dbl_expr dbl_expr gsl_sf_ellint_RD dbl_expr dbl_expr dbl_expr gsl_sf_ellint_RF dbl_expr dbl_expr dbl_expr gsl_sf_ellint_RJ dbl_expr dbl_expr dbl_expr dbl_expr gsl_sf_elljac gsl_sf_erfc dbl_expr gsl_sf_log_erfe dbl_expr gsl_sf_erf dbl_expr gsl_sf_erf_Z dbl_expr gsl_sf_erf_Q dbl_expr gsl_sf_hazard dbl_expr gsl_sf_exp dbl_expr gsl_sf_exp_el0 gsl_sf_exp_mult dbl_expr dbl_expr gsl sf exp mult e10 gsl_sf_expm1 dbl_expr 166 esl_sf_exprel_e gsl_sf_exprel_2_e gsl_sf_exprel_n_e gsl_sf_exp_err_e esl_sf_exp_err_el0_e gsl_sf_exp_mult_err_e esl_sf_exp_mult_err_el0_e gsl sf expint El e gsl sf expint E2 e gsl sf expint En e gsl_sf_
49. this selects the compression blocksize Minimal deflation dfl_lvl 1 achieves considerable storage compression with little time penalty Higher deflation levels require more time for compression File sizes resulting from minimal dfl_Ivl 1 and maximal dfl_Ivl 9 deflation levels typically differ by less than 10 in size To compress an entire file using deflation use ncks 4 L 0 in nc out nc No deflation fast no time penalty ncks 4 L 1 in nc out nc Minimal deflation little time penalty ncks 4 L 9 in nc out nc Maximal deflation much slower Unscientific testing shows that deflation compresses typical climate datasets by 30 6096 Packing a lossy compression technique available for all netCDF files see Section 3 34 Packed data page 105 can easily compress files by 50 Packed data may be deflated to squeeze datasets by about 8096 102 NCO 4 5 3 alpha03 User Guide ncks 4 L 1 in nc out nc Minimal deflation 30 60 compression ncks 4 L 9 in nc out nc Maximal deflation 31 63 compression ncpdq in nc out nc Standard packing 50 compression ncpdq 4 L 9 in nc out nc Deflated packing 80 compression ncks prints deflation parameters if any to screen see Section 4 7 ncks netCDF Kitchen Sink page 203 3 31 MD5 digests Availability ncecat ncks ncrcat Short options Long options md5_dgs md5_digest mdb wrt att md5 write attribute As of NCO version
50. value meter Eds 3dvar attribute 2 coordinates size 15 NC CHAR value N lat gds lon gds gds 3dvar attribute 3 purpose size 64 NC CHAR value Test auxiliary coordinates like those that define geodesic grids 17 CRD The coordinates attribute lists the names of the latitude and longitude coordinates lat_gds and lon_gds respectively The coordinates attribute is recommended though optional With it the user can immediately identify which variables contain the latitude and longitude coordinates Without a coordinates attribute it would be unclear at first glance whether a variable resides on a cell based grid In this example time is a normal record dimension and gds_crd is the cell based dimension The cell based grid file must contain two variables whose standard_name attributes are latitude and longitude ncks m C v lat_gds lon_gds nco data in nc lat gds type NC DOUBLE 1 dimensions 4 attributes chunked no compressed no packed no ID 37 lat gds RAM size is 8 sizeof NC DOUBLE 8 8 64 bytes lat gds dimension 0 gds crd size 8 NC FLOAT dim ID 17 CRD Chapter 3 Shared Features 69 lat gds attribute long name size 8 NC CHAR value Latitude lat gds attribute standard name size 8 NC CHAR value latitude lat gds attribute units size 6 NC CHAR value degree lat gds attribute 3 purpose size 62 NC CHAR value 1 D latitude coordinate referred to by geodes
51. which possess it the attribute hieght is changed to height in the variable tpt and in the variable prs sfc if it exists ncrename a FillValue missing value a units CGS units a tptOhieght height a prs sfcO hieght height in nc out nc The presence and absence of the and features cause this command to execute successfully only if a number of conditions are met All variables must have a _FillValue attribute and _FillValue must also be a global attribute The units attribute on the other hand will be renamed to CGS units wherever it is found but need not be present in the file at all either as a global or a variable attribute The variable tpt must contain the hieght attribute The variable prs sfc need not exist and need not contain the hieght attribute Rename the global or group attribute Convention to Conventions ncrename a Convention Conventions in nc Variable and group atts ncrename a Convention Conventions in nc Variable and group atts ncrename a Convention Conventions in nc Group atts only ncrename a Convention Conventions in nc Group atts only ncrename a global Convention Conventions in nc Group atts only ncrename a global Convention Conventions in nc Group atts only ncrename a global Convention Conventions in nc Global atts only ncrename a global Convention Conventions in nc Global atts only 234 NCO 4 5 3 alpha03 User Guide The examples without the character attem
52. 100 or of 1 100 respectively PPC algorithms should produce maximum absolute errors no greater than 0 5 in these units If the LSD is the hundreds digit then quantized versions of true values will be within fifty of the true value It is much easier to satisfy this tolerance for a true value of 100 only 50 accuracy required than for 999 5 accuracy required Thus the minimum accuracy guaranteed for nsd 1 ranges from 5 50 For this reason the best and worst cast performance usually occurs for true values whose LSD value is close to one and nine respectively Of course most users prefer prc gt 1 because accuracies increase exponentially with prc Continuing the previous example to prc 2 quantized versions of true values from 1000 9999 will also be within 50 of the true value i e have accuracies from 0 5 5 In other words only two significant digits are necessary to guarantee better than 596 accuracy in quantization We recommend that dataset producers and users consider quantizing datasets with nsd 3 This guarantees accuracy of 0 05 0 596 for individual values Statistics computed from ensembles of quantized values will assuming the mean error is small have much better accuracy than 0 5 This accuracy is the most that can be justified for many applications To demonstrate these principles we conduct error analyses on an artificial reproducible dataset and on an actual dataset of observational analysis values The table s
53. 15 to 17 significant decimal digits precision Hence double precision numbers represent about nine digits more precision than single precision numbers Given these properties there are at least two possible arithmetic conventions for the treatment of real numbers 1 Conservative aka Fortran Convention Automatic type conversion during arithmetic in the Fortran language is by default performed only when necessary All operands in an operation are converted to the most precise type involved the operation before the arithmetic operation Expressions which involve only single precision numbers are computed entirely in single precision Expressions involving mixed precision types 20 According to Wikipedia s summary of IEEE standard 754 If a decimal string with at most 6 significant digits is converted to IEEE 754 single precision and then converted back to the same number of significant decimal then the final string should match the original and if an IEEE 754 single precision is converted to a decimal string with at leastn 9 significant decimal and then converted back to single then the final number must match the original im According to Wikipedia s summary of IEEE standard 754 If a decimal string with at most 15 significant digits is converted to IEEE 754 double precision representation and then converted back to a string with the same number of significant digits then the final string should match the original and if an IEEE 7
54. 179 Info cuspis n ere EE Rea etn LES A T input files 24 irin eitie ii ranned 22 30 32 33 inpu t path ie eeen ere eE E ipaa DEI HAAS 30 34 installation eae sca da perebeenis 8 16 instantiate ceu ete ded ai as 140 instantiation n eisern foal Meee RR 140 Thal RETE A M T THESES 148 Int64 22 1cenesnesstad d danse S p CHR HS 148 IDLegratioH 2s eee and tn eae doe ERE ER 237 MOTA Poet eee a 102 Intel vole NEON 8 Internationalization eese 27 interoperability 000008 106 interpolation 0 ritea eee 200 266 intersections ee See ra E EEREN 43 45 introducti t ani ia rad ln 7 inyert map eee ds sendat Roa 152 IPC xo sn en 194 243 irregular grids 157 TS Orc ai cout oett aide uoce ipta ded na PRESE tts 8 James Gallagher ooooooooooommmmm 241 Jeff Whitaker 0 0 0 0 ccc cee ccc eens 95 Jin Edwards i ade ew the 242 JOG Hamm ceci dades 241 John Caro cerca ada res 95 241 Juliana Rew ti a da 242 K Karen Schuchardt 00 ccc cece eee ee 242 Keith Lindsay eee eben 242 kitchen sink llic icio Mave teases 203 Kyle WUCOR vest eer EA RUNE ERU abs 242 TIONIS S event tret eroe ct oer mo se E a s 27 large datasets 00 20 eee eee eee ee 23 28 Large File Support 0000s 23 42 latu Sei E E salada 75 l t3bnd znm A seda eo dae 75 lat bndS va dd anos 75 124 IAS OS adr D A E Rey 12 lat BSId4 AA 70 E A eee etes A E T2 MAGA S
55. 2 skips every other element etc ncks multislabs support strides and are more powerful than the regular hyperslabs supported by the other operators see Section 3 19 Multislabs page 64 Using the stride option for the record dimension with ncra and ncrcat makes it possible for instance to average or concatenate regular intervals across multi file input data sets The stride is specified as the optional fourth argument to the d hyperslab specification d dim min max stride Specify stride as an integer i e no decimal point following the third comma in the d argument There is no default value for stride Thus using d time 2 is valid but d time 2 0 and d time are not When stride is specified but min is not there is an ambiguity as to whether the extracted hyperslab Chapter 3 Shared Features 61 should begin with using C style 0 based indexes element 0 or element stride 1 NCO must resolve this ambiguity and it chooses element 0 as the first element of the hyperslab when min is not specified Thus d time stride is syntactically equivalent to d time 0 stride This means for example that specifying the operation d time 2 on the array 1 2 3 4 5 selects the hyperslab 1 3 5 To obtain the hyperslab 2 4 instead simply explicitly specify the starting index as 1 i e d time 1 2 For example consider a file 8501 8912 nc whi
56. 64 Multi Record Operator eee eee 62 NCO 4 5 3 alpha03 User Guide multiplication 0000 178 189 202 multiply oe cde hee dni 189 multiplying data 04 189 200 multislab 2 eere MESURER 64 IV ats een ay Tn itera gU 44 MVS it aad A eg i 8 10 N naked characters 00sec eee eee eee eee 190 NON pte jetratht eeb is cet gbedeex nhe 4 eee 186 NARR North American Regional Reanalysis a vedi aemebep Peace DECPeDebUSP Saad Mere na 150 NASA nesci terrine aA o A e X DA ELE e Te ps 243 NASA BOSDIS heira hes ends eens eh 21 National Virtual Ocean Data System 35 MONG ARPE eee a os 75 nco enddef i1 se eS eke ee 27 e si cass eet T pBEDX S4 e edu 14 NCA seks esa yb renedey bows ye RBS bee ieee 88 NC BYTE ai a aan 218 NC CHAR aeinn eeur o 60 141 191 218 NC DISKLES S iiu AE cil oa Sata tas 104 NC D UBLE easi ieren a Gees Beene T 180 218 NC FLUA Toni anaE E xenon bans 218 NC FORMAT DA Rican bere ve reper eee 40 NC FORMAT DAPA olio EROR Aes 40 NC FORMAT NC HDF4 eeeeeeee eese 40 NC_FORMAT_NC_HDF5 0200s ees 40 NC FORMAT NCS oo ic Repo Wl dog Cete RT tgs 40 NC FORMAT PNETCDF eeeeeeeeeeeee 40 NOT aaa we ate X io treu t tnb S La DI RET 218 NC INT64 eere Lx 12 218 NC2 SHORT stesso ene ts DATE a x I SER Heo e 218 NC STRING dicta AIT WO Xue 141 NCSUBYTE api uw Shu eren RE 12 218 NOCUTNTOS Lotes A iut 12 218 N
57. 88 onk C4 cssc V Lec ads 88 senk prd ts aust bere ptm met Ub Ta eim e 88 ecnk rid ecsokesjDX DIN GWY nha baw Rd 86 Sonik rdi iios ab MIA RU INR EU Yu 88 gol dh cr 88 sonk Scl ve soci ovS ON 88 do d on MEE HT 86 PM legendi CAE YRd OD 86 88 COLA a ria a ASE 75 GOMeAU A O A ete 8 command line options ooooooocccccoooo 29 command line switches 17 27 32 129 Comments 252 AA 131 COllO 2210 Ie Reales ROPRTepelbre0 pepe 8 Compag iesiti oi bi vel ees hie da 8 COMPALAtOL serne el eee eG eee ads 236 compatability s esp PERDERE 8 Compilers ud tie pta utu se on ges 33 complementary error function 178 compliance checker 0 cece eee eee 54 compression 0 eee eee eee eee eee 101 208 concatenation 19 197 220 228 conditional Operator ee eee eee 134 config gueSS mauere eee eee 16 config re eg i ices al 16 conservative regridding 000 74 constraint expressions esee 36 NCO 4 5 3 alpha03 User Guide CONTENTS eccl ad A aL T contributisg erp pae aa ea a ib 241 cOnttibubor amp add 241 coordinate limits 00 ccc cece eee eee 58 coordinate variable 80 108 126 191 236 coordinate variables oooooooooooo 232 coordinateS 0 0 eee eee 67 126 269 coordinates attribute o o o 93 215 coordinates convention oo o ooooooocooooo 126 coordinat
58. 90 lev 1 100 lat 1 90 three dmn var 2 1 lon 3 180 lev 1 100 lat 1 90 three dmn var 3 2 Printing hyperslab does not affect the variable or dimension indices since these indices are relative to the full variable as stored in the input file and the input file has not changed However if the hyperslab is saved to an output file and those values are printed the indices will change 4 ncks H d 1at 90 0 d 1ev 1000 0 v three dmn var in nc out nc lat 1 290 lev 2 1000 lon 0 0 three dmn var 20 20 lat 1 290 lev 2 1000 lon 1 90 three dmn var 21 21 lat 1 290 lev 2 1000 lon 2 180 three dmn var 22 22 lat 1 290 lev 2 1000 lon 3 270 three dmn var 23 23 4 ncks C v three dmn var out nc lat 0 90 lev 0 1000 lon 0 0 three dmn var 0 20 lat 0 290 lev 0 1000 lon 1 90 three dmn var 1 21 lat 0 290 lev 0 1000 lon 2 180 three dmn var 2 22 lat 0 290 lev 0 1000 lon 3 270 three dmn var 3 23 Turn on printing to screen or turn off copying global and group metadata This includes file summary information and global and group attributes Also Mtd 208 m NCO 4 5 3 alpha03 User Guide and Metadata By default ncks prints global metadata to screen if no netCDF output file and no variable extraction list is specified with v Use M to print global metadata to screen if a netCDF output is specified or if a variable extraction list is specified with v Use M to turn off copying
59. AMOO Luar a td 113 Dennis Heimbigner eee ee ee 241 derived fields 0 ccc ccc ccc cece eens 130 digesti sect ON 102 Digitales 8 dimension limits 000 c cece eee eee 58 General Index dimension names eee eee eee ee 230 dimension order 0 0 0 cee eee eee eee eee 256 dimensions growing 0 000 176 disaggregate 0 eee eee eee eee 51 disjoint Miles s scopus WP VERE steed NIS 19 diskless les cone 104 dismember chy rio a et RPIOR M RIaEe E 51 Distributed Data Reduction amp Analysis 243 Distributed Oceanographic Data System 35 divide iuuenes a d es 189 dividing data tral 189 division Colts pra is rs 178 AMO ta Sole eid oe seed Gaede VA ERE 88 documentation 0 0 eee eee ne 7 DOD Sips sitar in ee aegis Te a ne Mee 35 38 DODSERDO Tata e da 35 dot producir aras aaron sr bas 237 double onere elt t 148 double precision ssseeeeeeeeeeeee 180 DS Di narr daa tos 92 dsort 4o dd RR E 152 duration ulnis ac pNUEEiEhe ddb bei 0s 62 dynamic linking 0 00 eee cece eee ee 11 Ed Hartnettacsidsa sos ted ceediaecaauitediesd 241 A A 242 eddy c varan miii ia 177 editing attributes sse 183 Ta was ut Ue RP NAMENS Nue TR RD de 46 Elliptic integrals oooooomoommocm o 162 ensemble 2 22 benshdb lege D eek hb lees 20 194 ensemble average 0 0 cece eee eee 194 ensemb
60. C D 3 p http thredds test ucar edu thredds dodsC testdods 1 tmp in nc in nc foo nc NOAA DAP servers ncwa 0 C a lat lon time d lon 10 10 d lat 10 10 1 tmp p A http www esrl noaa gov psd thredds dodsC Datasets ncep reanalysis dailyavgs surface pres sfc 1969 nc foo nc LLNL PCMDI IPCC OPeNDAP Data Portal ncks M p http username password esgcet 11nl gov cgi bin dap cgi py ipcc4 sresaib n Earth System Grid ESG http www earthsystemgrid org caseid b30 025 E801 CCSM3 0 1 increasing C02 run T42_gx1v3 200 years starting in year 400 Atmospheric post processed data monthly averages e g data zender tmp b30 025 ES01 cam2 h0 TREFHT 0400 01_cat_0449 12 nc data zender tmp b30 025 ES01 cam2 h0 TREFHT 0400 01_cat_0599 12 nc ESG supports password protected FTP access by registered users NCO uses the netrc file if present for password protected FTP access Syntax for accessing single file is e g ncks 0 D 3 p ftp climate 11n1 gov sresalb atm mo tas ncar_ccsm3_0 run1 1 tmp tas_A1 SRESA1B_1 CCSM atmm 2000 01_cat_2099 12 nc foo nc Average surface air temperature tas for SRESA1B scenario This loop is illustrative and will not work until NCO correctly translates to FTP mget all remote files for var in tas do for scn in sresalb do for mdl in cccma cgcm3 1 cccma_cgcm3_1_t63 cnrm cm3 csiro_mk3_0 gfdl cm2 O0 gfdl cm2 1 giss aom giss model e h giss model
61. C the binary operators return an array of values There is no such thing as short circuiting with the AND OR operators Missing values are carried into the result in the same way they are with the arithmetic operators When an expression is evaluated in an if the missing values are treated as true The binary operators are in order of precedence Logical Not lt lt Less Than Selection gt gt Greater Than Selection gt Greater than gt Greater than or equal to lt Less than lt Less than or equal to Equal to Not equal to To see all operators see Section 4 1 26 Operator precedence and associativity page 180 Examples tmi time 2 amp amp time 7 tm1 0 O 1 1 1 1 O O O O double tm2 time 3 time gt 6 tm2 0 0 1 0 0 1 1 1 1 1 double tm3 int tm1 tm3 1 1 O O O O 1 1 1 1 int tm4 tmi amp amp tm2 tm4 0 0 1 0 0 1 0 0 0 O double tm5 tm4 tmb 1 1 O 1 1 O 1 1 1 1 double Regular Assign Operator vari expl If varl does not already exist in Output then varl is written to Output with the values and dimensions from expr1 If varl already exists in Output then the only requirement on exprl is that the number of elements must match the number already on disk The type of exprl is converted if necessary to the disk type 134 NCO 4 5 3 alpha03 User Guide Other Assign Operators vari ass op expl if expl is a variable and it d
62. Concatenating a variable packed with different scales across multiple datasets is be yond the capabilities of ncecat and ncrcat the other concatenator Section 2 6 1 Con catenation page 20 ncecat does not unpack data it simply copies the data from the input files and the metadata from the first input file to the output file This means that data compressed with a packing convention must use the identical packing parameters e g scale factor and add offset for a given variable across all input files Otherwise the concatenated dataset will not unpack correctly The workaround for cases where the pack Chapter 4 Reference Manual 199 ing parameters differ across input files requires three steps First unpack the data using ncpdq Second concatenate the unpacked data using ncecat Third re pack the result with ncpdq EXAMPLES Consider a model experiment which generated five realizations of one year of data say 1985 You can imagine that the experimenter slightly perturbs the initial conditions of the problem before generating each new solution Assume each file contains all twelve months a seasonal cycle of data and we want to produce a single file containing all the seasonal cycles Here the numeric filename suffix denotes the experiment number not the month ncecat 85_01 nc 85 02 nc 85 03 nc 85_04 nc 85_05 nc 85 nc ncecat 85 O 1 5 nc 85 nc ncecat n 5 2 1 85 O1 nc 85 nc These three commands produce identical answers See
63. ERE 5 DUN ei Mee gine iia laden din cupeigaeracses 8 SWAP SPAC eee eee e 23 24 104 Swath like Data SLD 0 2 0 150 Wilts rece tees ee DERI ee seen D Deis 273 SWItCheS cities 29 symbolic links 10 20 23 190 218 synchronous file accesS oooooooooocooooo 33 SYNONYM soe seid eet Seale ede des v atlas 10 SyIUAX A ev UR QS fade ie pP BICIS 131 System calls iuto pe ELDER ND 103 T Takeshi Enomoto seeeeesseeeeees 242 As ossa used views nte ele pueri data ung 178 tanhlisllilenierbtuedesp e Gee de oA RIA 178 temporary files esses 17 105 temporary output files 17 105 230 TEXID OuN s Vive ovis a oes yaa Ned sah re as neds ie bhrznbEg acu a REQUE 28 Td tit ene lend 24 25 27 timeo dM oLiisl ced LADO A 80 127 time averaging 178 245 246 253 256 263 time brndsS z c3 panera r ia pue Ss Wea ide d 75 timecoffse t uoc werk Rex Eg REID 127 208 time written co esso X a 125 bimestalnpsocesbeweensUeuiteqeer ERI YER ed 122 total uci Patents tees ds Pangea 107 iransposeete ve Gace EMS oa Hae WF n 57 220 TREFH ie e ERUIT S 118 TUNC bea cue futu ea ta iaa 178 trunc zssl RlfOe ve a br 114 truncate groups sssssss eee eee 49 truncation function 0 eee eee eee 178 truth condition 0 eee eee eee 236 239 A O wasnt SY er qe ll 107 tirada ede cae ee led esl 146 type COMVEFSION 6 eee eee 113 PO ces
64. FC19 Bug just fixed in texinfo tex version 2013 09 11 committed by Karl nco autobld texinfo tex now contains fixed version Breakage always occurs near here The square root of the mean of x y sqrt is E and pi MWLi Tj i N i Hii The definitions of some of these operations are not universally useful Mostly they were chosen to facilitate standard statistical computations within the NCO framework We are open to redefining and or adding to the above If you are interested in having other statistical quantities defined in NCO please contact the NCO project see Section 1 7 Help Requests and Bug Reports page 15 EXAMPLES Suppose you wish to examine the variable prs_sfc time lat lon which contains a time series of the surface pressure as a function of latitude and longitude Find the minimum value of prs_sfc over all dimensions ncwa y min v prs_sfc in nc foo nc Find the maximum value of prs sfc at each time interval for each latitude ncwa y max v prs sfc a lon in nc foo nc Find the root mean square value of the time series of prs sfc at every gridpoint ncra y rms v prs sfc in nc foo nc ncwa y rms v prs sfc a time in nc foo nc The previous two commands give the same answer but ncra is preferred because it has a smaller memory footprint A dimension of size one is said to be degenerate By default ncra leaves the degenerate time dimension in the output file which is usually useful whereas ncwa removes the
65. In the future perhaps with netCDF4 we hope to demonstrate significant threading improvements with operators like ncrcat by performing multiple simultaneous writes Compute intensive operators ncap2 ncwa and ncpdq benefit most from threading The greatest increases in throughput due to threading occur on large datasets where each thread performs millions at least of floating point operations Otherwise the system overhead of setting up threads probably outweighs the speed enhancements due to SMP parallelism However we have not yet demonstrated that the SMP parallelism scales beyond four threads for these operators Hence we restrict thr nbr lt 4 for all operators We encourage users to play with these limits edit file nco_omp c and send us their feedback Once the initial thr_nbr has been modified for any operator specific limits NCO requests the system to allocate a team of thr_nbr threads for the body of the code The operating system then decides how many threads to allocate based on this request Users may keep track of this information by running the operator with dbg lvl gt 0 By default threaded operators attach one global attribute nco openmp thread number to any file they create or modify This attribute contains the number of threads the op Chapter 3 Shared Features 29 erator used to process the input files This information helps to verify that the answers with threaded and non threaded operators are equal to wi
66. Long options mrd The netCDF3 file format allows only one record dimension and that dimension must be the first dimension i e the least rapidly varying dimension of any variable in which it appears This imposes certain rules on how operators must perform operations that alter the ordering of dimensions or the number of record variables The netCDF4 file format has no such restrictions Files and variables may have any number of record dimensions in any order This additional flexibility of netCDF4 can only be realized by selectively abandoning the constraints that would make operations behave completely consistently between netCDF3 and netCDF4 files NCO chooses by default to impose netCDF3 based constraints on netCDF4 files This reduces the number of unanticipated consequences and keeps the operators functioning in a familiar way Put another way NCO limits production of additional record dimensions so processing netCDF4 files leads to the same results as processing netCDF4 files Users can override this default with the mrd or multiple_record_dimension switch which enables netCDF4 variables to accumulate additional record dimensions How can additional record dimensions be produced Most commonly ncecat in record aggregate mode defines a new leading record dimension In netCDF4 files this becomes an additional record dimension unless the original record dimension is changed to a fixed 84 NCO 4 5 3 alpha03
67. Loop over models ncecat 0 glb mtd spp G xpt j mdl k mdl1 k _ Chapter 7 CMIP5 Example 251 drc_in var i _ rlm _ md1 k _ xpt j _ _ tms f1t A drc_out var i _ rlm _ md1 k _ xpt j _all nsm_ tms f 1t ncks A drc_out var i _ rlm _ md1 k _ xpt j _all nsm_ tms f1t drc_out var i _ rlm _ md1 0 _ xpt j _all nsm_ tms f1t done Loop done models ncks A drc_out var i _ rlm _ md1 0 _ xpt j _all nsm_ tms f1t drc_out var i _ rlm _ 4 md1 0 _ xpt 0 _all nsm_ tms f1t done Loop done experiments ncks A drc_out var i _ rlm _ md1 0 _ xpt 0 _all nsm_ tms f1t drc_out var 0 _ rlm _ md1 0 _ xpt 0 _all nsm_ tms f 1t done Loop done variables Rename output file mv drc_out var 0 _ rlm _ md1 0 _ xpt 0 _all nsm_ tms f1t drc_out sn_ rlm _al1l md1l_all xpt_all nsm_ tms f1t Remove temporary files rm drc_out sn _ rlm nc Rename Group E g file snc_LImon_CESM1 CAM5_historical_rii1p1_199001 200512 nc is now group historical CESM1 CAM5 CESM1 CAM5_00 You can rename it to historical CESM1 CAM5 riilp1 to make more sense Note You don t need to write the full path of the new name ncrename g xpt md1 md1l _00 r1ilp1 drc_out var _ rlm _ md1 _all nsm_ tms f1t esmHistorical CESM1 BGC CESM1 BGC_00
68. Mc gsl sf mathieu Ms gsl sf mathieu Mc array gsl sf mathieu Ms array gsl sf pow int gsl sf psi int int expr gsl_sf_psi dbl_expr gsl sf psi 1piy dbl expr gsl sf complex psi gsl sf psi 1 int int expr gsl sf psi 1 dbl expr gsl sf psi n int expr dbl expr gsl sf synchrotron 1 dbl expr gsl_sf_synchrotron_2 dbl_expr gsl_sf_transport_2 dbl_expr gsl_sf_transport_3 dbl_expr gsl_sf_transport_4 dbl_expr gsl_sf_transport_5 dbl_expr gsl_sf_sin gsl_sf_cos esl_sf_hypot esl_sf_complex_sin Chapter 4 Reference Manual 169 gsl sf complex cos e N gslsf complex cos gsl sf complex logsin e N gslsf complex logsin gsl sf sinc e N gslsf sinc gsl_sf_Insinh_e N gslsflnsinh gsl_sf_Incosh_e N gslsflncosh gsl_sf_polar_to_rect N gslsf polar to rect gsl sf rect to polar N gslsf rect to polar gsl sf sin err e N gslsf sin err gsl sf cos err e N gslsf cos err gsl sf angle restrict symm e N gslsf angle restrict symm gsl sf angle restrict pos e N gslsf angle restrict pos gsl sf angle restrict symm err e N gsl sf angle restrict symm err gsl sf angle restrict pos err e N gslsf angle restrict pos err gsl sf zeta int e Y gslsfzeta int int expr gsl sf zeta e Y gslsfzeta dbl expr gsl_sf_zetam1_e Y gslsf zetaml dbl expr gsl_sf_zetam1_int_e Y gsl_sf_zetaml int int_expr gsl_sf_hzeta_e Y gslsf hzeta dbl expr dbl expr gsl sf eta int e Y gslsf eta int int expr
69. NC_USHORT NC_CHAR NC_BYTE NC_UBYTE pck_map key values flt_byt pck map flt byt Pack Higher Precisions to NC SHORT Definition Pack higher precision types to NC SHORT Map Pack NC DOUBLE NC FLOAT NC INT64 NC UINT64 NC INT NC UINT to NC SHORT Types copied instead of packed NC_SHORT NC_USHORT NC_CHAR NC_BYTE NC_ UBYTE pck_map key values hgh_sht pck map hgh sht Pack Higher Precisions to NC BYTE Definition Pack higher precision types to NC BYTE Map Pack NC_DOUBLE NC_FLOAT NC_INT64 NC_UINT64 NC_INT NC_UINT NC_ SHORT NC_USHORT to NC BYTE Types copied instead of packed NC_CHAR NC_BYTE NC_UBYTE pck_map key values hgh_byt pck_map_hgh_byt Pack to Next Lesser Precision Definition Pack each type to type of next lesser size Map Pack NC_DOUBLE NC_INT64 NC_UINT64 to NC_INT Pack NC_FLOAT NC_ INT NC_UINT to NC_SHORT Pack NC_SHORT NC_USHORT to NC_BYTE Types copied instead of packed NC_CHAR NC_BYTE NC_UBYTE pck_map key values nxt_lsr pck map nxt lsr The default all_new packing policy with the default flt_sht packing map reduces the typical NC FLOAT dominated file size by about 50 f1t_byt packing reduces an NC_ DOUBLE dominated file by about 8796 The netCDF packing algorithm see Section 4 1 11 Methods and functions page 145 is lossy once packed the exact original data cannot be recovered without a full backup Hence users should be aware of some p
70. O FL OUT goss on ogo 9e EUIRU EIE UR 22 32 BOs oor da 18 121 lt p iu put path s0i e ir il 30 34 PA 209 LAA D sse PE P D S hM 217 A QA 209 TQ lvDebPHPRSRIB REED oe ME 209 215 FE Sipu uu eb INI RYAN M VET 15 16 127 O ech MEISSRLRUDA ERR IRMAESR PR 3T as 209 214 St GRP Nbr eos 653 2 ida idas 25 27 Moria da 209 E AA bPSCOUCLEI PR AS 197 Uria oda ide aes 218 SW co Rhee hE Ae Ae a DU Bo is 221 EM VAL i A A cea ads 43 213 SV ICnELLEOUr yee hah betas Ged OM DN hate 209 215 SWiWELGHEs ils psa age nad lese LE 235 weti scum eet tti alae teenies 200 TE Pots Cua eh ete cast etal 43 213 215 Xos gnteeru reve AA E 67 X lon min lon max lat min lat max 67 SV OP CYP sive s A ere p eu v as Re 107 189 m C 230 wildcard character slesssessese 46 A 4x4 A RERPRIUREQSMESS STA 211 netrc lob las wilede levied seg lie eee g lads 33 THOStS dy id Mee ees ae 33 bru ELE teen tenes he We Sales eae NE E LS ee 189 A diVISION EE rm 178 comment ssssessseseseee 131 288 coment 0v AA 131 separator character 0 0 020005 49 end of statement 0 0 0008 131 lt arpa nameser h sees 10 amp KreSolvil izln live eg deba vent eee Debra 10 filename expansion ssseesessssse 46 question mark 0 0 02 005 186 wildcard character 0 0 0 0000 46 O attribute dots a asis 131 separator c
71. Section 3 5 Specifying Input Files page 30 for an explanation of the distinctions between these methods The output file 85 nc is five times the size as a single input file It contains 60 months of data One often prefers that the new record dimension have a more descriptive context based name than simply record This is easily accomplished with the u ulm_nm switch ncecat u realization 85_0 1 5 nc 85 nc Users are more likely to understand the data processing history when such descriptive coordinates are used Consider a file with an existing record dimension named time and suppose the user wishes to convert time from a record dimension to a non record dimension This may be useful for example when the user has another use for the record variable The simplest method is to use ncks fix rec dmn but another possibility is to use ncecat followed by ncwa ncecat in nc out nc Convert time to non record dimension ncwa a record in nc out nc Remove new degenerate record dimension The second step removes the degenerate record dimension See Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 and Section 4 7 ncks netCDF Kitchen Sink page 203 for other methods of of changing variable dimensionality including the record dimension 200 NCO 4 5 3 alpha03 User Guide 4 6 ncflint netCDF File Interpolator SYNTAX ncflint 8 4 6 7 I A Cl c cnk dmn nm sz cnk map map cnk mi
72. Signell Gary Strand George White Andrew Wittenberg Remik Ziemlinski Excellent bug reports and feature requests Daniel Baumann Nick Bower Luk Claebs Bas Couwenberg Barry deFreese Aleksandar Jelenak Francesco Lovergine Matej Vela Debian packaging Patrice Dumas Ed Hill Orion Poplawski RedHat packaging George Shapavalov Patrick Kursawe Gentoo packaging Filipe Fernandes OpenSuse packaging Takeshi Enomoto Alexander Hansen Ian Lancaster Alejandro Soto Mac OS packaging Eric Blake Autoconf M4 help Gavin Burris Kyle Wilcox RHEL and CentOS build scripts and bug reports Chapter 5 Contributing 243 Andrea Cimatoribus NCO Spiral Logo Martin Otte Etienne Tourigny Single bug reports and fixes Wenshan Wang CMIP5 and MODIS processing documentation Please let me know if your name was omitted 5 2 Citation The recommended citations for NCO software are Zender C S 2008 Analysis of Self describing Gridded Geoscience Data with netCDF Operators NCO Environ Modell Softw 23 10 1338 1342 doi 10 1016 j envsoft 2008 03 004 Zender C S 2014 netCDF Operator NCO User Guide Version 4 4 3 http nco sf net nco pdf Use the former when referring to overall design purpose and optimization of NCO and use the latter when referring to specific features and or the User Guide itself Additional information on citing NCO is in the User Guide at http nco sf net ctt A complete list of NCO public
73. The loss scales with the squareroot of n For larger n users should promote floats with the dbl option if they want to preserve more than four significant digits in their results The db1 and f1t switches are only available with the NCO arithmetic operators that could potentially perform more than a few single precision floating point operations per result These are nces ncra and ncwa Each is capable of thousands to millions or more operations per result By contrast the arithmetic operators ncbo and ncflint perform at most one floating point operation per result Providing the dbl option for such trivial operations makes little sense so the option is not currently made available We are interested in users opinions on these matters The default behavior was changed from f1t to db1 with the release of NCO version 4 3 6 October 2013 We will change the default back to 1t if users prefer Or we could set a threshold e g n gt 10000 after which single to double precision promotion is automatically invoked Or we could make the default promotion convention settable via an environment variable GSL does this a lot Please let us know what you think of the selected defaults and options 3 36 3 Manual type conversion ncap2 provides intrinsic functions for performing manual type conversions This for exam ple converts variable tpt to external type NC SHORT a C type short and variable
74. User Guide dimension as must be done in netCDF3 files Also when ncpdq reorders dimensions it can preserve the record property of record variables ncpdq tries to define as a record dimension whichever dimension ends up first in a record variable and in netCDF4 files this becomes an additional record dimension unless the original record dimension is changed to a fixed dimension as must be done in netCDF3 files It it easier if ncpdq and ncecat do not increase the number of record dimensions in a variable so that is the default Use mrd to override this 3 27 Missing values Availability ncap2 ncbo nces ncflint ncpdq ncra ncwa Short options None The phrase missing data refers to data points that are missing invalid or for any reason not intended to be arithmetically processed in the same fashion as valid data The NCO arithmetic operators attempt to handle missing data in an intelligent fashion There are four steps in the NCO treatment of missing data 1 Identifying variables that may contain missing data NCO follows the convention that missing data should be stored with the FillValue specified in the variable s FillValue attributes The only way NCO recognizes that a variable may contain missing data is if the variable has a _FillValue attribute In this case any elements of the variable which are numerically equal to the _FillValue are treated as missing data NCO adopted the behavior that the defaul
75. VAs and GAs ncks A v one in nc out nc Append VAs not GAs ncks A M v one in nc out nc Append VAs and GAs ncks A m v one in nc out nc Append only data no atts ncks A M m v one in nc out nc Append GAs not VAs where VAs and GAs denote variable and group global attributes respectively Turn on printing to screen or turn off copying variable metadata Using m will print variable metadata to screen similar to ncdump h This displays all metadata pertaining to each variable one variable at a time This includes information on the storage properties of the variable such as whether it em ploys chunking compression or packing Also activated using mtd and metadata The ncks default behavior is to print variable metadata to screen if no netCDF output file is specified Use m to print variable metadata to screen if a netCDF output is specified Also use m to turn off copying of variable metadata to an output file Chapter 4 Reference Manual 209 no blank Print numeric representation of missing values As of NCO version 4 2 2 Octo ber 2012 NCO prints missing values as blanks i e the underscore character gt by default To enable the old behavior of printing the numeric repre sentation of missing values e g 1 0e36 use the no_blank switch Also activated using noblank or no blank TP Print data metadata and units to screen The P switch i
76. appoximates all months as being equal so e g February weighs slightly too much in the climatological mean This approximation can be removed by weighting months appropriately We must add the number of days per month to the monthly mean files First create a shell variable dpm unset dpm Days per month declare a dpm dpm 0 31 28 25 31 30 31 30 31 31 30 31 30 31 Allows 1 based indexing Method 1 Create dpm directly in climatological monthly means for mth in 1 12 do mm printf 02d mth ncap2 0 s dpm 0 0 date dpm mth drc_out caseid _clm mm nc drc_out caseid _clm mm nc done end loop over mth 278 NCO 4 5 3 alpha03 User Guide Method 2 Create dpm by aggregating small files for mth in 1 12 do mm printf 4O2d mth ncap2 0 v s dpm dpm mth nco data in nc drc_out foo_ mm nc done end loop over mth ncecat 0 D 1 p drc_out n 12 2 2 foo_ mm nc foo nc ncrename 0 D 1 d record time drc_out foo nc ncatted 0 h a long name dpm o c Days per month a units dpm o c days drc_out caseid _clm_0112 nc ncks A v dpm drc_out foo nc drc_out caseid _clm_0112 nc Method 3 Create small netCDF file using ncgen cat gt foo cdl lt lt EOF netcdf foo dimensions time unlimited variables float dpm time dpm long_name Days per month dpm units days data dpm 31 28 25 31 30 31 30 31 31 30 31 30 31 EOF ncgen b o
77. available with your NCO build use ncap2 f grep gsl The function names used by ncap2 mirror their GSL names The NCO wrappers for GSL functions automatically call the error handling version of the GSL function when available This allows NCO to return a missing value when the GSL library encounters a domain error or a floating point exception The slow down due to calling the error handling version of the GSL numerical functions was found to be negligible please let us know if you find otherwise Consider the gamma function The GSL function prototype is int gsl sf gamma e const double x gsl sf result result The ncap2 script would be lon in 10on 21 1 0 1 0 2 0 3 lon out gsl sf gamma lon in lon out _ 9 5135 4 5908 2 9915 These are the GSL standard function names postfixed with _e NCO calls these functions automatically without the NCO command having to specifically indicate the _e function suffix 162 NCO 4 5 3 alpha03 User Guide The first value is set to _FillValue since the gamma function is undefined for negative integers If the input variable has a missing value then this value is used Otherwise the default double fill value is used defined in the netCDF header netcdf h as NC FILL DOUBLE 9 969e 36 Consider a call to a Bessel function with GSL prototype int gsl_sf_bessel_Jn_e int n double x gsl_sf_result result An ncap2 script would be lon out gsl sf bessel Jn 2 1on in lon ou
78. avg 1 0 var nbr apx varsz stl 2D Block size lmn nbr wgt dmnsz stl lat nbr Weight size nco op typ avg fl_typ Conditional Operator netCDF4 needed for this example th nw three dmn var sht gt O three dmn var sht uint three dmn var sht int 4 1 9 print statement print variable name attribute name string format string The print function takes a variable name or attribute name or a quoted string and prints the contents in a in a similar fashion to ncks H There is also an optional C language style format string argument Currently the print function cannot print RAM variables or expressions such as print var_msk 3 4 To print an expression first evaluate it as a non RAM variable so it will be saved and can be printed and then print the variable examples print lon lon 0 0 144 NCO 4 5 3 alpha03 User Guide lon 1 90 lon 2 180 lon 3 270 print lon_2D_rrg 3 2f 0 00 0 00 180 00 0 00 180 00 0 00 180 00 0 00 print mss_val_fst _FillValue mss_val_fst _FillValue size 1 NC_FLOAT value 999 print This function Nt is monotonic n This function is monotonic 4 1 10 Missing values ncap2 Missing values operate slightly differently in ncap2 Consider the expression where op is any of the following operators excluding Arithmetic operators 4 Binary Operators gt gt lt lt amp amp gt gt lt lt
79. be a step in a procedure whose goal is to compute anomalies from a mean state In any case when one desires to generate a file whose statistical properties are equally influenced by all the inputs then nces is the operator to use Variables in the output file are the same size as the variable hyperslab in each input file or group and each input file or group must be the same size after hyperslabbing nces does allow files to differ in the record dimension size if the requested record hyperslab see Section 3 15 Hyperslabs page 58 resolves to the same size for all files nces recomputes the record dimension hyperslab limits for each input file so that coordinate limits may be used to select equal length timeseries from unequal length files This simplifies analysis of unequal length timeseries from simulation ensembles e g the CMIP3 IPCC AR4 archive nces works in one of two modes file ensembles or group ensembles File ensembles are the default equivalent to the old ncea and may also be explicitly specified by the nsm_fl or ensemble_file switches To perform statistics on ensembles of groups a 1 The old ncea command was deprecated in NCO version 4 3 9 released December 2013 NCO will attempt to maintain back compatibility and work as expected with invocations of ncea for as long as possible Please replace ncea by nces in all future work N As of NCO version 4 4 2 released February 2014 nces allows hypersla
80. be considered mandatory although NCO provides defaults for any arguments omitted The remaining arguments depend on the whether the grid is global or regional For global grids one should specify only two more arguments the latitude lat_typ and longitude lon_typ grid types These types are chosen as described below from a small selection of options that together define the most common rectangular global grids For regional grids one must specify the bounding box i e the edges of the rectangular grid on the North lat_nrt South lat_sth East lat_est and West lat_nrt sides Specifying a bounding box for global grids is redundant and will cause an error to ensure the user intends a global grid NCO assumes that regional grids are uniform though it will attempt to produce regional grids of other types if the user specifies other latitude lat typ and longitude lon_typ grid types e g Gaussian or Cap Edges of a regional bounding box may be specified individually or in the single argument forms The full description of grid generation arguments and their corresponding keys is Grid Title grd ttl It is surprisingly difficult to discern the geometric configuration of a grid from the coordinates of a SCRIP format gridfile A human readable grid description should be placed in grd ttl Examples include CAM FV scalar grid 129x256 and T42 Gaussian grid Grid File grid The grid generation API was bolted on to NCO an
81. binds together the input file data The new record dimension is defined in the root group of the output file so it is visible to all sub groups Its name is by default record This default name can be overridden with the u ulm nm short option or the ulm_nm or rcd nm long options Each extracted variable must be constant in size and rank across all input files The only exception is that ncecat allows files to differ in the record dimension size if the requested record hyperslab see Section 3 15 Hyperslabs page 58 resolves to the same size for all files This allows easier gluing averaging of unequal length timeseries from simulation ensembles e g the CMIP rchive Classic i e all net CDF3 and NETCDFA CLASSIC output files can contain only one record dimension ncecat makes room for the new glue record dimension by changing the pre existing record dimension if any in the input files into a fixed dimension in the output file netCDF4 output files may contain any number of record dimensions so ncecat need not and does not alter the record dimensions if any of the input files as it copies them to the output file Group AGgregation GAG stores input files as top level groups in the output file No assumption is made about the size or shape or type of a given object variable or dimension or group in the input file The entire contents of the extracted portion of each input 198 NCO 4 5 3 alpha03 User Guide fi
82. build and execute production data analysis scripts that are repeated frequently on large datasets Obviating an extra file write can then conserve significant disk space and time For this purpose NCO has since version 4 2 1 in August 2012 made configurable the controls over temporary file creation The wrt_tmp_fl and equivalent write_tmp_fl switches ensure NCO writes output to an intermediate temporary file This is and has always been the default behavior so there is currently no need to specify these switches However the default may change some day especially since writing to RAM disks see Section 3 33 RAM disks page 104 may some day become the default The no_tmp_fl switch causes NCO to write directly to the final output file instead of to an intermediate temporary file Power users may wish to invoke this switch to increase performance i e reduce wallclock time when manipulating large files When eschewing temporary files users may forsake the ability to have the same name for both output file and input file since as described above the temporary file prevented overlap issues However if the user creates the output file in RAM see Section 3 33 RAM disks page 104 then it is still possible to have the same name for both output file and input file ncks in nc out nc Default create out pid tmp nc then move to out nc ncks wrt tmp fl in nc out nc Same as default ncks no tmp fl in nc out nc
83. call by reference variables If any of the output variables don t exist prior to the call then they are created on the fly as scalar variables of type double If they already exist then their existing value is overwritten If the function call is successful then status 0 status gsl fit wlinear data x stride x data w stride w data y stride y n amp co amp c1 amp cov00 amp cov01 amp covi1 amp chisq Similar to the above call except it creates an additional weighting vector from the variables data_w stride_w n data_y_out gsl_fit_linear_est data_x c0 c1 cov00 cov01 cov11 This function calculates y values along the line Y c0 c1 X Section B status gsl fit mul data x stride x data y stride y n amp ci amp covii amp sumsq Input variables data x stride x data y stride y n From the above variables an X and Y vector both of length n are derived If data_x or data y is less than type double then it is converted to type double Output variables c1 cov11 sumsq status gsl fit wmul data x stride x data w stride w data y stride y n amp ci amp covii amp sumsq Similar to the above call except it creates an additional weighting vector from the variables data_w stride_w n data_y_out gsl_fit_mul est data_x c0 c1 cov11 This function calculates y values along the line Y c1 X The below example shows gsl_fit_linear in action defdim d1 10 xin di 1 2 3 4 5 6 7 8 9 10 0 yin d1 3 1 6 2 9 1 12 2 1
84. cesm grid MCD43C3 2000049 regrid nc H Output files modis mcd43c3 cesm grid MCD43C3 2000049 regrid nc HH Online http nco sourceforge net nco html Permute MODIS Coordinates HH Execute this script bash pmt crd sh HESS SSS SSS SS SS SS SS SS SS SS SSS SS SSS SSS SS SS SSS SSS SS SS SSS SS SSS SS SSS SS SS SSSSSSSssa Tt Se o ic eee tees Permute coordinates HH Inverse lat from 90 90 to 90 90 Permute lon from 180 180 to 0 360 for fn in 1s MCD43C3 nc do Loop over files sfx echo fn cut d f 1 3 Part of file names echo sfx Lat ncpdq 0 a lat fn fn Inverse latitude NB there is before lat Lon ncks 0 msa d lon 0 0 180 0 d lon 180 0 1 25 fn fn Add new longitude coordinates ncap2 0 s lon array 0 0 1 25 lon fn fn done Chapter 8 Parallel 273 8 Parallel This section will describe NCO scripting strategies Many techniques can be used to exploit script level parallelism including GNU Parallel and Swift ls historical nc parallel ncks 0 d time 1950 01 01 2000 01 01 50y Chapter 9 CCSM Example 275 9 CCSM Example This chapter illustrates how to use NCO to process and analyze the results of a CCSM climate simulation FEA 3K 3K 3K 3K K K aK K K aK aK 2K 2K 2K 2K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K K aK AA 2K 2K aK 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K I A I I K K K 2k 2k 2k 2k
85. command line 1 1 nc is the last file named because it is the pre existing output file to which we will append all the other input files listed in this case only f1 2 nc The contents of 1 1 nc are completely preserved and only values in 1 2 nc and any other input files are copied This switch avoids the necessity of copying all of 1 1 nc through the netCDF interface to a new output file The rec_apn switch automatically puts NCO into append mode see Section 2 4 Appending Variables page 19 so specifying A is redundant and simultaneously specifying overwrite mode with 0 causes an error By default NCO works in an intermediate temporary file Power users may combine rec_apn with the no_tmp_f1 switch see Section 2 3 Temporary Output Files page 17 ncrcat rec apn no tmp fl fl 2 nc fl 1 nc This avoids creating an intermediate file and copies only the minimal amount of data i e all of 1 2 nc Hence it is fast We recommend users try to understand the safety trade offs involved 3 18 Subcycle Availability ncra ncrcat Short options d dim min max stride subcycle Long options mro dimension dim min max stride subcycle dmn dim min max stride subcycle As of version 4 2 1 August 2012 NCO allows both Multi File Multi Record operators ncra and ncrcat to extract and operate on multiple gro
86. coordinate that has a one to one map ping to both latitude and longitude coordinates Rectangular in longitude and latitude horizontal hyperslabs cannot be selected using the typical procedure see Section 3 15 Hy perslabs page 58 of separately specifying d arguments for longitude and latitude In stead when the X is used NCO learns the names of the latitude and longitude coordinates by searching the standard_name attribute of all variables until it finds the two variables whose standard_name s are latitude and longitude respectively This standard_name attribute for latitude and longitude coordinates follows the CF convention see Section 3 41 CF Conventions page 124 Putting it all together consider a variable gds_3dvar output from simulations on a cell based geodesic grid Although the variable contains three dimensions of data time latitude and longitude it is stored in the netCDF file with only two dimensions time and gds_crd ncks m C v gds_3dvar nco data in nc gds 3dvar type NC FLOAT 2 dimensions 4 attributes chunked no compressed no packed no ID 41 gds 3dvar RAM size is 10 8 sizeof NC FLOAT 80 4 320 bytes Eds 3dvar dimension 0 time size 10 NC DOUBLE dim ID 20 CRD REC gds 3dvar dimension 1 gds crd size 8 NC FLOAT dim ID gds 3dvar attribute 0 long_name size 17 NC CHAR value Geodesic variable gds_3dvar attribute 1 units size 5 NC CHAR
87. dbl_expr esl_sf_fermi_dirac_m1 dbl_expr esl_sf_fermi_dirac_0 dbl_expr esl_sf_fermi_dirac_1 dbl_expr gsl_sf_fermi_dirac_2 dbl_expr esl_sf_fermi_dirac_int int_expr dbl_expr esl_sf_fermi_dirac_mhalf dbl_expr esl_sf_fermi_dirac_half dbl_expr gsl_sf_fermi_dirac_3half dbl_expr gsl sf fermi dirac inc O dbl expr dbl expr gsl sf Ingamma dbl expr gsl sf Ingamma sgn gsl sf gamma dbl expr gsl_sf_gammastar dbl_expr gsl_sf_gammainv dbl_expr gsl_sf_Ingamma_complex gsl_sf_taylorcoeff int_expr dbl_expr gsl_sf_fact uint_expr gsl_sf_doublefact uint_expr esl_sf_Infact uint_expr gsl_sf_Indoublefact uint_expr gsl_sf_Inchoose gsl_sf_choose gsl_sf_Inpoch dbl_expr dbl_expr gsl sf Inpoch sgn gsl_sf_poch dbl_expr dbl_expr gsl_sf_pochrel dbl_expr dbl_expr Chapter 4 Reference Manual gsl sf gamma inc Q e gsl sf gamma inc P e gsl sf gamma inc e gsl_sf_Inbeta_e gsl sf Inbeta sgn e gsl sf beta e gsl sf beta inc e gsl sf gegenpoly 1 e gsl sf gegenpoly 2 e gsl sf gegenpoly 3 e gsl_sf_gegenpoly_n_e gsl_sf_gegenpoly_array esl_sf_hyperg 0F1_e esl_sf_hyperg 1F1_int_e esl_sf_hyperg 1F1_e gsl_sf_hyperg_U_int_e gsl sf hyperg U int el0 e gsl sf hyperg U e esl_sf_hyperg U_el0_e gsl sf hyperg 2F1 e gsl sf hyperg 2F1 conj e esl_sf_hyperg 2F1_renorm_e esl_sf_hyperg 2F1_conj_renorm_e esl_sf_hyperg 2F0_e esl_sf_laguerre_1_e esl_sf_laguerre_2_e gsl sf laguerre 3 e esl_sf_laguerre_n_e esl_sf_la
88. decent hte me 43 e A its QD LIIS na UIS RA eld peas 43 eshdf unpack Muria deen eee eh ales 105 T hdf upkE AAA RU RDLEEEUR qe desde 105 a e A eig x eei 13 SENG cnt lS gest Nuesezczies Eie bu SenbiqM 205 hdr pad hdr_pad 0 cece 27 header_pad hdr_pad 0000 27 oS Add si thin ata T a ee 205 TOO EOD MUSA tels Boe EEA 207 SHHITSt OP Yuet hid oes atin Nie Biss alk Ga ne PN eis 122 a Eee srr aw eee bcd ners aa EE rdi i 122 cu ntersection ii 2 A ee 45 Ss reloutput patlis A Se ew 33 Local output path cc cei ele eee bees 33 EMP nU pe 69 73 map cnkvmap a nbsp Re leat care POPE 88 semap pck map n0ltb bein tes hes 218 mapshile us suc td 73 mask value mask val 236 mask variable mask var 235 mask comparator mask comp 236 mask condition mask cond 235 236 mask value mask val 236 mask variable mask var 235 z2mdb dgs nenebllw P ePEULLEME S 102 mdbidigestozo eue fe exer writ deren 102 md5b write attribute Ls 102 mdb wft att iososvesewee5ebela Wu Wwesers 102 Metadatasc o dle olan du dl 207 Metadata weeded A CAG WO Sa 208 mk rec dmn dim s 199 206 ASM a iX O CERES 83 SMO ns via a bande bit aie ks nV S ERE 62 SSMS actos aces cutem ber CO Eon da Ee RR 64 msa user order eese 64 ssmsa usrJrdr i ves B elc 64
89. default 3 N4 1 2 15 1 51 2 ppc default 2 N4 1 les 9 9 33 6 ppc default 1 A selective per variable approach to PPC yields the best balance of precision and com pression yet requires the dataset producer to understand the intrinsic precision of each variable Such a specification for a GCM dataset might look like this using names for the NCAR CAM model Be conservative on non explicit quantities so default 5 Some quantities deserve four significant digits Many quantities such as aerosol optical depths and burdens are highly uncertain and only useful to three significant digits ncks 7 0 ppc default 5 ppc AN AQ 4 ppc AER A0D ARE AW BURDEN 3 ncar_cam nc foo nc 3 30 Deflation Availability ncap2 ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options L Long options df1_lvl deflate All NCO operators that define variables support the netCDF4 feature of storing variables compressed with the lossless DEFLATE compression algorithm DEFLATE combines the Lempel Ziv encoding with Huffman coding The specific version used by netCDF4 HDF5 is that implemented in the zlib library used by gzip Activate deflation with the L df1_lv1 short option or with the same argument to the df1 1v1 or deflate long options Specify the deflation level df_Ivl on a scale from no deflation df lvl 0 to maximum deflation dfl_lvl 9 Under the hood
90. dimension that span more than one file are handled correctly ncra sup ports the stride argument to the d hyperslab option for the record dimension only stride is not supported for non record dimensions Concatenating a variable packed with different scales multiple datasets is beyond the capabilities of ncrcat and ncecat the other concatenator Section 2 6 1 Concatenation page 20 ncrcat does not unpack data it simply copies the data from the input files and the metadata from the first input file to the output file This means that data compressed with a packing convention must use the identical packing parameters e g scale_factor and add_offset for a given variable across all input files Otherwise the concatenated dataset will not unpack correctly The workaround for cases where the packing parameters differ across input files requires three steps First unpack the data using ncpdq Second concatenate the unpacked data using ncrcat Third re pack the result with ncpdq ncrcat applies special rules to ARM convention time fields e g time offset See Section 3 42 ARM Conventions page 127 for a complete description EXAMPLES Concatenate files 85 nc 86 nc 89 nc along the record dimension and store the results in 8589 nc ncrcat 85 nc 86 nc 87 nc 88 nc 89 nc 8589 nc ncrcat 8 56789 nc 8589 nc ncrcat n 5 2 1 85 nc 8589 nc These three methods produce identical answers See Section 3 5 Specifying Input Files
91. dt dt dt dt db dt dt dt OF OF d d V I X f f NCO 4 5 3 alpha03 User Guide After cmb fl sh Example Long term average of each model globally Input files like data cmip5 snc LImon bcc csm 1 historical riiipi 185001 200512 nc Output files like data cmipb5 output snc snc LImon all mdl historical all nsm clm nc Online http nco sourceforge net nco html Global Distribution of Long_002dterm Average Execute this script bash glb_avg sh Parameters rc_in home wenshanw data cmip5 Directory of input files rc_out home wenshanw data cmip5 output Directory of output files ar snc snd Variables lm LImon Realm pt historical Experiment could be more ld out snc snd Folders of output files or var_id in 0 1 do Loop over two variables Names of all models 1s get file names cut get the part for model names sort uniq remove duplicates awk print mdl set 1s drc_in var var_id _ rlm _ _ xpt 0 _ nc cut d _ f 3 sort uniq c awk print 2 Number of models echo print contents wc count mdl num echo mdl set wc w for mdl in mdl set do Loop over models Average all the ensemble members of each model Use nces file ensembles mode nsm fl nces nsm fl 0 4 d time 1956 01 01 00 00 0 0 2005 12 31 23 59 9 9 drc inj var var id r1m md1 xpt O T nc
92. equal to input path stripped of any machine names If 1 is not specified and the remote file resides on the NCAR HPSS system then the leading character of input path is also stripped from output path Specifying output path as 1 tells NCO to store the remotely retrieved file and the output file in the current directory Note that 1 is equivalent to 1 though the latter is syntactically more clear 3 7 1 OPeNDAP The Distributed Oceanographic Data System DODS provides useful replacements for com mon data interface libraries like netCDF The DODS versions of these libraries implement network transparent access to data via a client server data access protocol that uses the HTTP protocol for communication Although DODS technology originated with oceanogra phy data it applyies to virtually all scientific data In recognition of this the data access protocol underlying DODS which is what NCO cares about has been renamed the Open source Project for a Network Data Access Protocol OPeNDAP We use the terms DODS and OPeNDAP interchangeably and often write OPeNDAP DODS for now In the future we will deprecate DODS in favor of DAP or OPeNDAP as appropriate NCO may be DAP enabled by linking NCO to the OPeNDAP libraries This is described in the OPeNDAP documentation and automagically implemented in NCO build mechanisms The configure mechanism automatically enables NCO as OPeNDAP clients if it can find the
93. foo nc foo cdl ncks A v dpm drc_out foo nc drc_out caseid _clm_0112 nc Another way to get correct monthly weighting is to average daily output files if available FEA AA K K K K K aK aK aK 2K 2k 2K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K K aK 2K 2K 2K 2K K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K A I A K K K K K K 2k 2k 2k 2k 2k Task 3 Regional processing FEA OOOO AAAI 2K 2K 2k 2K 3K 3K 3K CCI K K K K K K 1 K 2k 2k 2k 2k 2k 2k Let s say you are interested in examining the California region Hyperslab your dataset to isolate the appropriate latitude longitudes ncks 0 D 1 d 1at 30 0 37 0 d lon 240 0 270 0 drc_out caseid _clm_0112 nc drc_out caseid _clm_0112_Cal nc The dataset is now much smaller To examine particular metrics Chapter 9 CCSM Example 279 FEA AA K K aK K aK aK aK aK ak ak 2K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K K aK K 2K K 2K 2K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K I K I K K K 2k 2k 2k 2k 2k 2k 2k Task 4 Accessing data stored remotely FEA 3K 3k 3K 3K K K K K aK aK aK 2K AAA IIIA IRR 3K 3K 3K 3K 3K 3K 3K K K K K K K K 1 2k 2k 2k 2k 2k 2k 2k OPeNDAP server examples UCI DAP servers ncks M p http dust ess uci edu cgi bin dods nph dods dodsdata in nc ncrcat 0 C D 3 p http dust ess uci edu cgi bin dods nph dods dodsdata 1l tmp in nc in nc foo nc Unidata DAP servers ncks M p http thredds test ucar edu thredds dodsC testdods in nc ncrcat 0
94. g1 is not a complete component of the path g12 fg1 and gigi The second command above shows how a terminating slash character cancels the recursive copying of groups An argument to g which terminates with a slash character extracts the group and its direct ancestors but none of its children The third command above shows how an initial slash character anchors the argument to the root group The third command would not extract the group g2 g1 because the g1 group is not at the root level but it would extract any group g1 at the root level and all its children recursively The fourth command is the non recursive version of the third command The fifth command is a special case of the fourth command As mentioned above both v and g options may be specified simultaneously and NCO will by default extract the intersection of the lists i e the specified variables found in the specified groups 4 The unn option causes NCO to extract the union rather than the intersection of the specified groups and variables Consider the following commands which may be assumed to end with in nc out nc Intersection mode subsetting default ncks g gl v vi Yes gl v1 g2 g1 v1 No vi g2 v1 ncks g gl v vi Yes gi v1 g1 g2 v1 No v1 g2 v1 g2 g1 v1 ncks g gl v vi Yes gi v1 g2 g1 v1 No v1 g2 v1 g1 g2 v1 necks v gi v1 Yes g1 v1 g2 g1 v1 No vi g2 v1 g1 g2 v1 ncks g g1
95. get miss Returns the missing value of a variable in Input or Output delete miss Deletes the missing value associated with a variable ram write Writes a RAM variable to disk i e converts it to a regular disk type variable ram delete Deletes a RAM variable or an attribute PDQ Methods See see Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 reverse dim args Reverses the dimension ordering of elements in a variable 148 NCO 4 5 3 alpha03 User Guide permute dim args Re shapes variables by re ordering the dimensions All the dims of the variable must be specified in the arguments A limitation of this permute unlike ncpdq is that the record dimension cannot be re assigned Swap dimensions about and reorder along lon lat_2D_rrg_new lat_2D_rrg permute lon lat reverse lon lat 2D rrg new 0 90 30 30 30 30 90 0 Type Conversion Methods These methods allow ncap2 to convert variables and attributes to the different netcdf types For more details on automatic and manual type conversion see see Section 3 36 Type Conversion page 113 You may only use the new netCDF4 types if you have compiled links NCO with the netCDF4 library and the Output file is HDF5 netCDF3 4 Types byte convert to NC_BYTE a signed 1 byte integer char convert to NC CHAR an ISO ASCII character short convert to NC_SHORT a signed 2 byte integer int convert to NC INT a signed 4 byte integer float con
96. h switch does this too and turns off the history attribute as well see Section 3 39 History Attribute page 122 Hence both switches allows space conscious users to avoid storing what may amount to many thousands of filenames in a metadata attribute 3 41 CF Conventions Availability ncbo nces ncecat ncflint ncpdq ncra ncwa Short options None NCO recognizes some Climate and Forecast CF metadata conventions and applies special rules to such data NCO was contemporaneous with COARDS and still contains some rules to handle older model datasets that pre date CF such as NCAR CCM and early CCSM datasets Such datasets may not contain an explicit Conventions attribute e g CF 1 0 Nevertheless we refer to all such metadata collectively as CF metadata Skip this section if you never work with CF metadata The latest CF netCDF conventions are described here http cfconventions org 1 6 html Most CF netCDF conventions are transparent to NCO There are no known pitfalls associated with using any NCO operator on files adhering to these conventions To increase user friendliness NCO applies special rules to certain variables in CF files Rules not required by CF are because experience shows that they simplify data analysis Here is a general sense of NCO s CF support Understanding and implementing NUG rec ommendations such as the history attribute packing conventions and attention to units Special handling of v
97. in 5 away from zero This rounds for example 1 5d and 2 5d to 2s and 3s respectively Since April 2013 version 4 3 0 NCO has employed the other C library family of round ing functions lrint O This algorithm rounds x to the nearest integer using the current rounding direction Halfway cases are rounded to the nearest even integer This rounds for example both 1 5d and 2 5d to the same value 2s as recommended by the IEEE This rounding is symmetric up half the time down half the time This is the current and hopefully final demotion algorithm employed by NCO Hence because of automatic conversion NCO will compute the average of 2s and 3s in double precision arithmetic as 2 0d 3 0d 2 0d 2 5d It then demotes this intermediate result back to NC SHORT and stores it on disk as trunc 2 5d 2s versions up to 4 0 8 lround 2 5d 3s versions 4 0 9 4 2 6 and 1rint 2 5d 2s versions 4 3 0 and later 3 36 2 Promoting Single precision to Double Promotion of real numbers from single to double precision is fundamental to scientific computing When it should occur depends on the precision of the inputs and the number of operations Single precision four byte numbers contain about seven significant figures while double precision contain about sixteen More err precisely the IEEE single precision representation gives from 6 to 9 significant decimal digits precision And the IEEE double precision representation gives from
98. includes the starting date yyyymm_str ls drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ nc sed n 1p cut d _ f 6 cut d f 1 Ending date of data sed the name of the last file includes the ending date yyyymm_end 1s drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ nc sed n fl_nbr p cut d _ f 6 cut d f 2 Concatenate one ensemble member files into one along the record dimension now is time ncrcat 0 drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ nc drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ yyyymm_str yyyymm_end Remove useless files rm drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ yyyymm_str yyyymm_end done done done CMIP5 model data downloaded from the Earth System Grid Federation ESGF http pcmdi9 11n1 gov esgf web fe does not contain group features yet Therefore users must aggregate flat files into hierarchical ones themselves The following script shows how Each dataset becomes a group in the output file There can be several levels of groups In this example we employ two experiments scenarios as the top level The second level comprises different models e g CCSM4 CESM1 BGC Many models are run multiple times with slight perturbed initial conditions to produce an ensemble of realizations These ensemble members comprise the third level of the hiera
99. increasingly complex as the netCDF4 features in the input file become more elaborate Convert file with netCDF4 atomic types ncks 3 in nc4 out nc3 Convert file with multiple record dimensions netCDF4 atomic types ncks 3 fix rec dmn all in nc4 out nc3 Convert file with groups multiple record dimensions netCDF4 atomic types ncks 3 G fix rec dmn all in nc4 out nc3 Future versions of NCO may automatically invoke the record dimension fixation and group flattening when converting to netCDF3 rather than requiring it be specified manu ally If this feature would interest you please let us know 3 10 Large File Support Availability All operators Short options none Long options none NCO has Large File Support LFS meaning that NCO can write files larger than 2 GB on some 32 bit operating systems with netCDF libraries earlier than version 3 6 If desired LFS Chapter 3 Shared Features 43 support must be configured when both netCDF and NCO are installed netCDF versions 3 6 and higher support 64 bit file addresses as part of the netCDF standard We recommend that users ignore LFS support which is difficult to configure and is implemented in NCO only to support netCDF versions prior to 3 6 This obviates the need for configuring explicit LFS support in applications such as NCO that now support 64 bit files directly through the netCDF interface See Section 3 9 File Formats and Conversion page 38 for instru
100. it can be shown that the expected rounding error after n inexact operations is J 2n 7 for large n The expected i e mean absolute rounding error in our example with 13 824 additions is about 4 2 x 13824 m 91 96 Hence addition alone of about fifteen thousand single precision floats is expected to consume about two significant digits of precision This neglects the error due to the inner product weights times values and normalization division by tally aspects of a weighted average The ratio of two numbers each containing a numerical bias can magnify the size of the bias In summary a global mean number computed from about 15 000 gridpoints each with weights can be expected to lose up to three significant digits Since single precision starts with about seven significant digits we should not expect to retain more than four significant digits after computing weighted averages in single precision The above example with TREFHT shows the expected four digits of agreement The NCO results have been independently validated to the extent possible in three other languages C Matlab and NCL C and NCO are the only languages that permit single precision numbers to be treated with single precision arithmetic Double precision weight gw single precision arithmetic C ncwa 3528514 exe TREFHT 289 240112 Double precision weight gw double precision arithmetic C TREFHT 289 239964 Single precision weight area double precision
101. like Data or SLD To find the average temperature in a region bounded by lat min lat max and lon_min lon_max temperature msk south north east west 0 0 where lat gt lat min amp amp lat lt lat max amp amp lon gt lon min amp amp lon lt lon max temperature msk temperature elsewhere temperature msk temperatureO FillValue temp avg temperature msk avg temp max temperature max For North American Regional Reanalysis NARR data example dataset http dust ess uci edu diwg narr uwnd 199605 nc the procedure looks like this ncap2 0 v S narr nco DATA hdf narr_uwnd 199605 nc foo nc where narr nco is an ncap2 script like this North American Regional Reanalysis NARR Statistics NARR stores grids with 2 D latitude and longitude aka Swath like Data SLD Here we work with three variables lat y x lon y x and uwnd time level y x To study sub regions of SLD we use masking techniques 1 Define mask as zero times variable to be masked Chapter 4 Reference Manual 151 Then mask automatically inherits variable attributes And average below will inherit mask attributes 2 Optionally create mask as RAM variable as below with asterisk NCO does not write RAM variable to output Masks are often unwanted and can be big so this speeds execution 3 Example could be extended to preserve mean lat and lon of sub region Follow uwnd example to do this lat msk 0 0 1at lat a
102. many users prefer to continue invoking ncdiff rather than memorizing a new command ncbo y sbt which behaves identically to the original ncdiff command However from a software maintenance standpoint maintaining a distinct executable for each binary operation e g ncadd is untenable and a single executable ncbo is desirable To maintain backward compatibility therefore NCO automatically creates a symbolic link from ncbo to ncdiff Thus ncdiff is called an alternate invocation of ncbo ncbo supports many additional alternate invocations which must be manually activated Should users or system adminitrators decide to activate them the procedure is simple For example to use ncadd instead of ncbo op_typ add simply create a symbolic link from ncbo to ncadd The alternatate invocations supported for each operation type are listed above Alternatively users may always define ncadd as an alias to ncbo op_typ add It is important to maintain portability in NCO scripts Therefore we recommend that site specfic invocations e g ncadd be used only in interactive sessions from the command line For scripts we recommend using the full invocation e g ncbo op_typ add This ensures portability of scripts between users and sites ncbo operates e g adds variables in file 2 with the corresponding variables those with the same name in file 1 and stores the results in file 3 Variables in file 1
103. mask comp mask val The preferred method to specify the mask condition is in one string with the B or mask condition switches The older method is to use the three switches m T and M to specify the mask var mask comp and mask_val respectively The mask condition string is auto matically parsed into its three constituents mask var mask comp and mask val Here mask var is the name of the masking variable specified with m mask variable mask_variable msk_nm or msk var The truth mask comp argument specified with T mask_comparator msk cmp typ or op rlt may be any one of the six arithmetic comparators eq ne gt lt ge le These are the Fortran style character abbreviations for the logical comparisons Z gt lt gt The mask comparator defaults to eq equality The mask val argument to M or mask value or msk_val is the right hand side of the mask condition Thus for 1 The default behavior of I changed on 19981201 before this date the default was not to weight or mask coordinate variables 2 if lat wgt contains Gaussian weights then the value of latitude in the output file will be the area weighted centroid of the hyperslab For the example given this is about 30 degrees 3 The three switches m T and M are maintained for backward compatibility and may be deprecated in the future
104. max were swapped If this seems confusing test your command on just the coordinate variables with ncks and then examine the output to ensure NCO selected the hyperslab you expected coordinate wrapping is currently only supported by ncks Because of the way wrapped coordinates are interpreted it is very important to make sure you always specify hyperslabs in the monotonically increasing sense i e min max even if the underlying coordinate variable is monotonically decreasing The only exception to this is when you are indeed specifying a wrapped coordinate The distinction is crucial to understand because the points selected by e g d longitude 50 340 are exactly the complement of the points selected by d longitude 340 50 Not specifying any hyperslab option is equivalent to specifying full ranges of all dimen sions This option may be specified more than once in a single command each hyperslabbed dimension requires its own d option 3 16 Stride Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options d dim min max stride Long options dimension dim min max stride dmn dim min max stride All data operators support specifying a stride for any and all dimensions at the same time The stride is the spacing between consecutive points in a hyperslab A stride of 1 picks all the elements of the hyperslab and a stride of
105. may be chunked with their own cnk_dmn options that will apply to all variables cnk_ple key values rid cnk_rid plc_rid Mnemonic Record 1 D variables Chunk Variables Containing Explicitly Chunked Dimensions Definition Chunk all variables possible that contain at least one dimension whose chunksize was explicitly set with the cnk_dmn option Alternate invocation none cnk_plc key values xp1 cnk_xp1 plc xpl Mnemonic EXPLicitly specified dimensions Chunk Variables that are already Chunked Definition Chunk only variables that are already chunked in the input file When used in conjunction with cnk_map xst this option preserves and copies the chunking parameters from the input to the output file Alternate invocation none cnk_plc key values xst cnk_xst plc_xst Mnemonic EXiSTing chunked variables 88 NCO 4 5 3 alpha03 User Guide Chunk Variables with NCO recommendations Definition Chunk all variables according to NCO best practices This is a virtual option that ensures the chunking policy is in the subjective opinion of the authors the best policy for typical usage As of NCO version 4 4 8 February 2015 this virtual policy implements map_rew for 3 D variables and map 1fp for all other variables Alternate invocation none cnk plc key values nco cnk_nco plc_nco Mnemonic Net CDF Operator Unchunking Definition Unchunk all var
106. memory requirements due its ability to process arbitrarily long scripts of any complexity All scripts acceptable to ncap2 are ultimately processed as a sequence of binary or unary operations ncap2 requires MS lt 2VT under most conditions An exception to this is when left hand casting see Section 4 1 4 Left hand casting page 136 is used to stretch the size of derived variables beyond the size of any input variables Let VC be the memory required to store the largest variable defined by left hand casting In this case MS lt 2VC 26 NCO 4 5 3 alpha03 User Guide ncap2 scripts are complete dynamic and may be of arbitrary length A script that contains many thousands of operations may uncover a slow memory leak even though each single operation consumes little additional memory Memory leaks are usually identifiable by their memory usage signature Leaks cause peak memory usage to increase monotonically with time regardless of script complexity Slow leaks are very difficult to find Sometimes a malloc or new failure is the only noticeable clue to their existance If you have good reasons to believe that a memory allocation failure is ultimately due to an NCO memory leak rather than inadequate RAM on your system then we would be very interested in receiving a detailed bug report 2 10 Performance An overview of NCO capabilities as of about 2006 is in Zender C S 2008 Analysis of Self describing Gridded Geoscience Data wi
107. metadata expansions will not incur the netCDF3 performance penalty of copying the entire output file unless the expansion exceeds the amount of header padding exceeded This can be beneficial when it is known that some metadata will be added at a future date The operators which benefit most from judicious use of header padding are ncatted and ncrename since they only alter metadata This optimization exploits the netCDF library nc__enddef O function which behaves differently with different versions of netCDF It will improve speed of future metadata expansion with CLASSIC and 64bit netCDF files though not necessarily with NETCDF4 files i e those created by the netCDF interface to the HDF5 library see Section 3 9 File Formats and Conversion page 38 3 3 OpenMP Threading 28 NCO 4 5 3 alpha03 User Guide Availability ncap2 ncbo nces ncecat ncflint ncpdq ncra ncrcat ncwa Short options t Long options thr_nbr threads omp_num_threads NCO supports shared memory parallelism SMP when compiled with an OpenMP enabled compiler Threads requests and allocations occur in two stages First users may request a specific number of threads thr_nbr with the t switch or its long option equiva lents thr_nbr threads and omp_num_threads If not user specified OpenMP obtains thr_nbr from the OMP_NUM_THREADS environment variable if present or from the OS if not C N
108. most conservative and least likely to lead to misinterpretation Option two is to output the weighted valid data point i e val3 wgt2 x val2 The rationale for this behavior is that interpolation is really a weighted average of known points so ncflint should weight the valid point Option three is to return the unweighted valid point i e val3 val2 This behavior would appeal to those who use ncflint to estimate data using the closest available data When a point is not bracketed by valid data on both sides it is better to return the known datum than no datum at all The current implementation uses the first approach Option one If you have strong opinions on this matter let us know since we are willing to implement the other approaches as options if there is enough interest EXAMPLES Although it has other uses the interpolation feature was designed to interpolate file_3 to a time between existing files Consider input files 85 nc and 87 nc containing variables describing the state of a physical system at times time 85 and time 87 Assume each file contains its timestamp in the scalar variable time Then to linearly interpolate to a file 86 nc which describes the state of the system at time at time 86 we would use ncflint i time 86 85 nc 87 nc 86 nc Say you have observational data covering January and April 1985 in two files named 85 01 nc and 85 04 nc respectively Then you can estimate the values for Februa
109. must separate them If only grp pth is specifed the separator character may be omitted e g G g1 If only lvl nbr is specifed the separator character is still required to indicate it is a Ivl nbr arugment and not a grp pth e g G 1 or G 01 If the at sign separator character is used instead of the colon separator character then the following lvl nbr arugment must be positive and it will be assumed to refer to Truncation Mode Hence G 1 is the same as G 1 This is simply a way of making the Ivl_nbr argument positive definite 3 13 1 Deletion Truncation and Flattening of Groups GPE has three editing modes Delete Truncate and Flatten Select one of GPE s three editing modes by supplying a Ivl nbr that is positive negative or zero for Delete Truncate and Flatten mode respectively In Delete mode lvl nbr is positive integer which specifies the maximum number of group path components i e groups that GPE will try to delete from the head of grp pth For example lvl nbr 3 changes the input path g1 g2 g3 g4 g5 to the output path g4 g5 Input paths with Ivl_nbr or fewer components groups are completely erased and the output path commences from the root level In other words GPE is tolerant of specifying too many group components to delete It deletes as many as possible without complaint and then begins to flatten the file which fails if namespace conflicts arise In Truncate mod
110. nc 17 When originally released in 2012 this was called the duration feature and was abbreviated DRN 64 NCO 4 5 3 alpha03 User Guide These operations generate and require no intermediate files This contrasts to previous NCO methods which require generating averaging then catenating 150 files The mro option has no effect on or rather is redundant for ncrcat since ncrcat always outputs all selected records 3 19 Multislabs Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat Short options d dim min max stride Long options dimension dim min max stride dmn dim min max stride msa_usr_rdr msa_user_order A multislab is a union of one or more hyperslabs One defines multislabs by chaining together hyperslab commands i e d options see Section 3 15 Hyperslabs page 58 Support for specifying a multi hyperslab or multislab for any variable was first added to ncks in late 2002 The other operators received these capabilities in April 2008 Multi slabbing is often referred to by the acronym MSA which stands for Multi Slabbing Algorithm As explained below the user may additionally request that the multislabs be returned in the user specified order rather than the on disk storage order Although MSA user ordering has been available in all operators since 2008 most users were unaware of it since the documentation
111. nc tmp ncdismember http thredds test ucar edu thredds dodsC testdods foo nc tmp ncdismember nco data mdl 1 nc data zender nco tmp cf ncdismember nco data mdl 1 nc data zender nco tmp nerc ncdismember nco data mdl 1 nc data zender nco tmp cf 1 3 ncdismember nco data mdl 1 nc data zender nco tmp cf 1 5 fix rec dmn all dk dk dk dt dt dt dt dt dt H dt dt dt Gto OF Command line argument defaults fl in HOME nco data mdl 1 nc sng Input file to dismember check drc_out DATA nco tmp sng Output directory cf_chk 0 flg Perform CF compliance check Which checker cf_vrs 1 5 sng Compliance check this CF version e g 1 5 opt flg Additional ncks options e g fix rec dmn all Use single quotes to pass multiple arguments to opt 5 Otherwise arguments would be seen as 15 16 17 Command line argument option parsing if n 1 then fl_in 1 fi if n 2 then drc out 2 fi if n 3 then cf chk 13 fi if n 4 then cf_vrs 4 fi if n 5 then opt 5 fi Prepare output directory echo NCO dismembering file fl_in fl stb basename 1f1_in drc out drc out fl stb mkdir p drc_out cd drc out chk_dck n chk nrc n if cf_chk nerc then chk nrc y fi chk nrc if cf chk 0 amp amp cf_chk nerc then chk dck y hash cfchecker 2
112. nc time units hours since 1990 1 1 file02 nc time units hours since 1990 1 2 file31 nc time units hours since 1990 1 31 Mean noontime temperature in January ncra v tpt d time 1990 1 1 12 00 00 1990 1 31 23 59 59 24 N file nc noon nc Chapter 3 Shared Features 83 Concatenate day2 noon through day3 noon records ncrcat v tpt d time 1990 1 2 12 00 00 1990 1 3 11 59 59 N file01 nc file02 nc file03 nc noon nc Results time is re based to the time units in file01 nc time 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 If we repeat the above command but with only two input files ncrcat v tpt d time 1990 1 2 12 00 00 1990 1 3 11 59 59 file02 nc file03 noon nc then output time coordinate is based on time units in file02 nc time 12 13 14 15 16 17 18 19 20 21 22 23 24 25 A 26 27 28 29 30 31 32 33 34 35 As of NCO version 4 2 1 August 2012 NCO automatically rebases not only the record coordinate time here but also any cell boundaries associated with the record coordi nate e g time bnds see Section 3 41 CF Conventions page 124 As of NCO version 4 4 9 May 2015 NCO also rebases any climatology boundaries associated with the record coordinate e g climatology bounds see Section 3 41 CF Conventions page 124 3 26 Multiple Record Dimensions Availability ncecat ncpdq Short options None
113. nco sf net The HTML version of the User Guide is also available online through the World Wide Web at URL http nco sf net nco html To build and use NCO you must have netCDF installed The netCDF homepage is http www unidata ucar edu packages netcdf New NCO releases are announced on the netCDF list and on the nco announce mailing list http lists sf net mailman listinfo nco announce 1 2 How to Use This Guide Detailed instructions about how to download the newest version http nco sf net Source and how to complie source code http nco sf net bld as well as a FAQ http nco sf net FAQ and descriptions of Known Problems http nco sf net t bug etc are on our homepage http nco sf net There are twelve operators in the current version 4 5 3 alpha03 The function of each is explained in Chapter 4 Reference Manual page 129 Many of the tasks that NCO can accomplish are described during the explanation of common NCO Features see Chapter 3 Shared features page 27 More specific use examples for each operator can be seen by visiting the operator specific examples in the Chapter 4 Reference Manual page 129 1 To produce these formats nco texi was simply run through the freely available programs texi2dvi dvips texi2html and makeinfo Due to a bug in TEX the resulting Postscript file nco ps contains the Table of Contents as the final pages Thus if you print nco ps remember to insert the Table of C
114. nro iii estat ove wens dogs oe ome dees 34 H EME neuer one a s pl aos T HTTP protocol Lire la es Goths Leite be 35 Huftman codmg burrito 91 hyal cctieishesdut ee ede ELE 124 DM rr ES Paro fedus 124 Pi rs 124 DM iii ies ela 124 hybrid coordinate SysteM oooocooccccoco 136 hyperbolic arccosine function 178 hyperbolic arcsine function 178 hyperbolic arctangent function 178 hyperbolic cosine function 178 hyperbolic sine function 2 178 hyperbolic tangent 0 00048 178 hyperslab 58 86 194 197 225 228 235 hyperslab8 z ii RR beeen tes MER 137 I WO a2 DE 37 57 65 I O block sme ic ee e e eee 103 LIISN YR ate capes RR RD eR YA ERES 27 Tain Lancaster PR 242 A O 8 DOG A A OIT 8 ID Quotige i 4 RSS RRISISRIeGEReRA 181 DA oe canst hea ea face nthe ii 17 IEEB 1 5 p oelHReepPQPG Re DC whee dad hoe 94 115 IEEE 154 id Gace PUR RED HEURE 93 IEEE NaN es netic ered chee TREE Senedd helices 186 TEC ce ceste es dee uo E EM UT T do 142 AMD cipue a e KE Piden 24 illegal names cece eee eee eee eee 14 implicit conversion eee e eee eee ee 115 include Paired er sc de dave weds NER e E cem 152 including files ee re peed Dosa riut 131 index convention 0 0 e eee eee 57 General Index indgen equivalent 00 2 eee ee 136 inexact conversion eese eene
115. obtain descending in time order loops could accomplish the same thing exercise left for reader However loops are slow for large datasets NO oF WN Ee Chapter 4 Reference Manual 157 Following index manipulation requires understanding correspondence between 1 d unrolled memory order of storage and access into that memory as a multidimensional 3 d in this case rectangular array Key idea to understand is how dimensionality affects offsets Copy 1 d sort map into 3 d sort map srt map 3d O int three dmn rec var srt map Multiply base offset by factorial of lesser dimensions srt map 3d 1at size 10on size lon idx array 0 1 10n lat idx array 0 1 1at 10n size lat lon idx 1at 10n 1at idx lon idx srt map 3dt lat lon idx print sort map 3d WMn print srt map 3d Use remap to re map the data three dmn rec var remap three dmn rec var srt map 3d Finally reverse data so time coordinate is descending time time reverse time print sorted time descending order n print time three dmn rec var three dmn rec var reverse time Method two Key difference is srt map time size srt map 1 EOF ncap2 0 v S ncap2_foo nco nco data in nc foo nc 4 1 17 Irregular Grids NCO is capable of analyzing datasets for many different underlying coordinate grid types netCDF was developed for and initially used with grids comprised of orthogonal dimensions forming a
116. of an existing history attribute are copied from the first input file to the output file The timestamps appear in reverse chronological order with the most recent timestamp appearing first in the history attribute Since NCO adheres to the history convention the entire data processing path of a given netCDF file may often be deduced from examination of its history attribute As of May 2002 NCO is case insensitive to the spelling of the history attribute name Thus attributes named History or HISTORY which are non standard and not recommended will be treated as valid history attributes When more than one global attribute fits the case insensitive search for history the first one found is used To avoid information overkill all operators have an optional switch h hst or Chapter 3 Shared Features 123 history to override automatically appending the history attribute see Section 4 2 ncatted netCDF Attribute Editor page 183 Note that the h switch also turns off writing the nco input file list attribute for multi file operators see Section 3 40 File List Attributes page 123 As of NCO version 4 5 0 June 2015 NCO supports its own convention to retain the history attribute contents of all files that were appended to a file This convention stores those contents in the history of appended files attribute which complements the history attribute to provide a more complete provenance These att
117. on the native unstructured grid Here are some quick examples of regridding from common models All examples require in nc out nc at the end Identity re map ACME CAM SE Cubed Sphere output to improve metadata ncks map DATA maps map_ne30np4_to_ne30np4_aave 20150603 nc Convert ACME CAM SE Cubed Sphere output to rectangular lat lon ncks map DATA maps map_ne30np4_to_fv129x256_aave 150418 nc Convert CAM3 T42 output to Cubed Sphere grid ncks map DATA maps map_ne30np4_to_t42_aave 20150601 nc 3 24 UDUnits Support Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options d dim min max stride Long options dimension dim min max stride dmn dim min max stride There is more than one way to hyperskin a cat The UDUnits http www unidata ucar edu packages udunits package provides a library which if present NCO uses to translate user specified physical dimensions into the physical dimensions of data stored in netCDF files Unidata provides UDUnits under the same terms as netCDF so sites should install both Compiling NCO with UDUnits support is currently optional but may become required in a future version of NCO Two examples suffice to demonstrate the power and convenience of UDUnits support First consider extraction of a variable containing non record coordinates with physical dimensions stored in MKS
118. or file 2 1 A naked i e unprotected or unquoted is a wildcard character A naked may confuse the com mand line parser A naked and are relatively harmless The widely used shell Bash correctly interprets all these special characters even when they are not quoted That is Bash does not prevent NCO from correctly interpreting the intended arithmetic operation when the following arguments are given without quotes to ncbo op_typ op_typ op_typ and op typ 3 The command to do this is In s f ncbo ncadd 4 The command to do this is alias ncadd ncbo op_typ add Chapter 4 Reference Manual 191 are broadcast to conform to the corresponding variable in the other input file if necessary Now ncbo is completely symmetric with respect to file 1 and file 2 i e file file files filei Broadcasting a variable means creating data in non existing dimensions by copying data in existing dimensions For example a two dimensional variable in file 2 can be subtracted from a four three or two not one or zero dimensional variable of the same name in file 1 This functionality allows the user to compute anomalies from the mean In the future we will broadcast variables in file 1 if necessary to conform to their counterparts in file 2 Thus presently the number of dimensions or rank of any processed variable in file 1 must be greater th
119. order of its arguments Differing dimensions are said to be orthogonal to one another and sets of dimensions which are mutually exclusive are or thogonal as a set and any arithmetic operation between variables in orthogonal dimensional spaces is ambiguous without further information The ambiguity may be resolved by enumerating the desired dimension ordering of the output expression inside square brackets on the left hand side LHS of the equals sign This is called left hand casting because the user resolves the dimensional ordering of the RHS of the expression by specifying the desired ordering on the LHS ncap2 s lat lon lat lon lat lon in nc out nc ncap2 s lon lat lon lat lat lon in nc out nc The explicit list of dimensions on the LHS lat lon resolves the otherwise ambiguous ordering of dimensions in lat lon In effect the LHS casts its rank properties onto the RHS Without LHS casting the dimensional ordering of lat lon would be undefined and hopefully ncap2 would print an error message Consider now a slightly more complex example In geophysical models a coordinate system based on a blend of terrain following and density following surfaces is called a hybrid coordinate system In this coordinate system four variables must be manipulated to obtain the pressure of the vertical coordinate PO is the domain mean surface pressure offset a scalar PS is the local time varying surface pressure usually two horizontal spatial
120. pfx _gw_rgn4 nc Remove temporary files optional rm drc_out fld_out var_id pfx _gw_ nc drc_out fld_out var_id pfx _gw nc Annual average use the feature of Duration ncra mro d time 1956 01 01 00 00 0 0 2005 12 31 23 59 9 9 12 12 drc_out fld_out var_id pfx _gw_rgn4 nc drc_out fld_out var_id pfx _yrly nc Anomaly Long term average ncwa 0 a time drc_out fld_out var_id pfx _yrly nc drc_out fld_out var_id pfx _clm nc Subtract long term average ncbo 0 op typ drc_out fld_out var_id pfx _yrly nc A drc_out fld_out var_id pfx _clm nc drc_out fld_out var_id pfx _anm nc done rm drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _ _yrly nc Average over all the ensemble members Chapter 7 CMIP5 Example 259 ncea 0 4 drc_out fld_out var_id var var_id _ r1lm _ md1 _ xpt 0 _ _anm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_anm nc Standard deviation for fn in ls drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _ _anm nc do pfx echo fn cut d f8 cut d _ f1 5 Difference between each ensemble member and the average of all members ncbo 0 op_typ fn drc_out fld_out var_id var var_id _ r1m _ md1 _ xpt 0 _all nsm_anm nc drc_out fl
121. provide an extended set of mathematical functions that apply to single float and quadruple long double precision variables Using these functions e g sinf for float sinl for long double when available is presumably more efficient than casting variables to type double performing the operation and then re casting NCO uses the faster intrinsic functions when they are available and uses the casting method when they are not 4 Linux supports more of these intrinsic functions than other OSs Chapter 4 Reference Manual 181 Assignment Right to Left pe Addition subtraction assignment Multiplication division assignment 4 1 27 ID Quoting In this section when I refer to a name I mean a variable name attribute name or a dimension name The allowed characters in a valid netCDF name vary from release to release See end section If you want to use metacharacters in a name or use a method name as a variable name then the name has to be quoted wherever it occurs The default NCO name is specified by the regular expressions DGT 0 79 LPH E AAA RIE AA ES ae ys name LPH LPH DGT The first character of a valid name must be alphabetic or the underscore Any subsequent characters must be alphanumeric or underscore e g al _23 hell_is_666 The valid characters in a quoted name are specified by the regular expressions LPHDGT a z A 2Z _ 0 9 name LPHDGT je
122. prs to external type NC DOUBLE a C type double ncap2 s tpt short tpt prs double prs in nc out nc See Section 4 1 ncap2 netCDF Arithmetic Processor page 130 for more details 3 37 Batch Mode Availability All operators Short options 0 A gt 6 6 Long options ovr overwrite apn append If the output file specified for a command is a pre existing file then the operator will prompt the user whether to overwrite erase the existing output file attempt to append to it or abort the operation However interactive questions reduce productivity when processing large amounts of data Therefore NCO also implements two ways to override its own safety features the 0 and A switches Specifying 0 tells the operator to overwrite any existing output file without prompting the user interactively Specifying A tells the operator to attempt to append to any existing output file without prompting the user interactively These switches are useful in batch environments because they suppress 122 NCO 4 5 3 alpha03 User Guide interactive keyboard input NB As of 20120515 ncap2 is unable to append to files that already contain the appended dimensions 3 38 Global Attribute Addition Availability All operators Short options None Long options glb gaa glb att add glb att nm att val multiple invocations allowed All o
123. rectangular coordinate system We call such grids standard grids It is increas ingly common for datasets to use metadata to describe much more complex grids Let us first define three important coordinate grid properties rectangularity regularity and fxm Grids are regular if the spacing between adjacent is constant For example a 4 by 5 degree latitude longitude grid is regular because the spacings between adjacent latitudes 4 degrees are constant as are the 5 degrees spacings between adjacent longitudes Spac ing in irregular grids depends on the location along the coordinate Grids such as Gaussian grids have uneven spacing in latitude points cluster near the equator and so are irregular Grids are rectangular if the number of elements in any dimension is not a function of any other dimension For example a T42 Gaussian latitude longitude grid is rectangular because there are the same number of longitudes 128 for each of the 64 latitudes Grids are non rectangular if the elements in any dimension depend on another dimension Non rectangular grids present many special challenges to analysis software like NCO 158 NCO 4 5 3 alpha03 User Guide Wrapped coordinates see Section 3 20 Wrapped Coordinates page 66 such as longi tude are independent of these grid properties regularity rectangularity The preferred NCO technique to analyze data on non standard coordinate grids is to create a region mask with ncap2 and then to
124. required OPeNDAP libraries in the usual locations The DODS ROOT environment variable may be used to override the default OPeNDAP library location at NCO compile time Building NCO with bld Makefile and the command make DODS Y adds the non intuitive commands to link to the OPeNDAP libraries installed in the DODS_ROOT directory The file 6 DODS is being deprecated because it is ambiguous referring both to a protocol and to a collection of oceanography data It is superceded by two terms DAP is the discipline neutral Data Access Protocol at the heart of DODS The National Virtual Ocean Data System NVODS refers to the collection of oceanography data and oceanographic extensions to DAP In other words NVODS is implemented with OPeNDAP OPeNDAP is also the open source project which maintains develops and promulgates the DAP standard OPeNDAP and DAP really are interchangeable Got it yet Automagic support for DODS version 3 2 x was deprecated in December 2003 after NCO version 2 8 4 NCO support for OPeNDAP versions 3 4 x commenced in December 2003 with NCO version 2 8 5 NCO support for OPeNDAP versions 3 5 x commenced in June 2005 with NCO version 3 0 1 NCO support for OPeNDAP versions 3 6 x commenced in June 2006 with NCO version 3 1 3 NCO support for OPeNDAP versions 3 7 x commenced in January 2007 with NCO version 3 1 9 8 The minimal set of libraries required to build NCO as OPeNDAP clients where OPeNDAP is supplied as a
125. separate library apart from libnetcdf a are in link order libnc dap a libdap a and libxm12 and libcurl a x 36 NCO 4 5 3 alpha03 User Guide doc opendap sh contains a generic script intended to help users install OPeNDAP before building NCO The documentation at the OPeNDAP Homepage http www opendap org is voluminous Check there and on the DODS mail lists http www unidata ucar edu packages dods home mailLists to learn more about the extensive capabilities of OPeNDAP Once NCO is DAP enabled the operators are OPeNDAP clients All OPeNDAP clients have network transparent access to any files controlled by a OPeNDAP server Simply specify the input file path s in URL notation and all NCO operations may be performed on remote files made accessible by a OPeNDAP server This command tests the basic functionality of OPeNDAP enabled NCO clients 4 ncks 0 o foo nc C H v one 1 tmp p http thredds test ucar edu thredds dodsC testdods in nc 4 ncks H v one foo nc one 1 The one 1 outputs confirm first that ncks correctly retrieved data via the OPeNDAP protocol and second that ncks created a valid local copy of the subsetted remote file With minor changes to the above command netCDF4 can be used as both the input and output file format 4 ncks 4 0 o foo nc C H v one 1 tmp p http thredds test ucar edu thredds dodsC testdods in 4 nc 4 ncks H v one foo nc one 1 And of course O
126. similarity between group hierarchies and directory structures As of NCO 4 4 4 June 2014 ncks alone supports an option to include specified groups yet exclude specified variables The grp_xtr_var_xcl switch with long option equivalent gxvx extracts all contents of groups given as arguments to g grp except for variables given as arguments to v var Use this when one or a few variables in hierarchical files are not to be extracted and all other variables are This is useful when coercing netCDF4 files into netCDF3 files such as with converting flattening or dismembering files see Section 3 13 1 Flattening Groups page 49 ncks grp xtr var xcl g gl v vi Extract all of group gl except v1 Two properties of subsetting recursion and anchoring are best illustrated by reminding the user of their UNIX equivalents The UNIX command mv src dst moves src and all its subdirectories and all their subdirectories etc to dst In other words mv is by default recursive In contrast the UNIX command cp src dst moves src and only src to dst If src is a directory not a file then that command fails One must explicitly request to copy directories recursively i e with cp r src dst In NCO recursive extraction and copying of groups is the default like with mv not with cp Recursion is turned off by appending a trailing slash to the path These UNIX commands also illustrate a property we call anch
127. sish 5 n Ad etes ee uses fen ets c 151 whitespaeenec dessen Wendt AE 80 wildcards eere 30 46 184 187 MINS se racial afore ahs E hal eect REO NUTS ud 10 MandOoWws usara Lune 8 10 wrapped coordinates 60 66 158 216 wrapped filenamesS ooooooocccccoonccccn oc 31 WRE a eii aiies hes tan GA e RISUS OMS 159 WRF Weather and Research Forecast Model e a Pm 150 WWW documentation selseeeeeeeeeee T X RSIS tae EE TE E EEE 75 KATE Suey 8s unia e Sesh Se Tl Ga ee SE SU 22 33 Xl uunsuctectem A ae tecti 8 ZIG So tein ai ogee dei br rr bred 8 RME se by cud RBRUM P IB AM ES 210 XP Microsoft operating system 10 PL wlohe he A Ara 86 EA a a b Mta 86 88 Y A epp I vq wee ee tbe cag 75 bod c EE 17 26 Z ZlXb fied sale RM eV OLET Ern 91
128. the UDUNITS PATH environment variable With UDUnits version 2 users may specify the UDUnits database file itself udunits2 xml via the UDUNITS2 XML PATH environment vari able UDUnitsi export UDUNITS_PATH unusual location share udunits UDUnits2 export UDUNITS2 XML PATH unusual location share udunits udunits2 xml This run time flexibility can enable the full functionality of pre built binaries on machines with libraries in different locations The UDUnits http www unidata ucar edu packages udunits package documen tation describes the supported formats of time dimensions Among the metadata conven tions that adhere to these formats are the Climate and Forecast CF Conventions http cf pcmdi 11n1 gov and the Cooperative Ocean Atmosphere Research Data Service COARDS Conventions http ferret wrc noaa gov noaa coop coop cdf profile html The following d arguments extract the same data using commonly encountered time dimension formats d time 1918 11 11 00 00 0 0 1939 09 09 00 00 0 0 d time 1918 11 11 00 00 0 0 1939 09 09 00 00 0 0 d time 1918 11 11T00 00 0 02Z 1939 09 09T00 00 0 02 d time 1918 11 117 1939 09 09 d time 1918 11 117 1939 9 9 All of these formats include at least one dash in a non leading character position a dash in a leading character position is a negative sign NCO assumes that a space colon or non leading dash in a limit string i
129. the unused points near the poles We demonstrate the ncap2 analysis technique for irregular regions by constructing a mask for an R 2 grid We wish to find say the mean temperature within lat_min lat_max and lon_min lon_max ncap2 s mask var lat gt lat min amp amp lat lt lat max amp amp lon gt lon min amp amp lon lt lon max in nc out nc Arbitrarily shaped regions can be defined by more complex conditional statements Once defined masks can be applied to specific variables and to entire files ncap2 s temperature_avg temperature mask_var avg in nc out nc ncwa a lat lon m mask var w area in nc out nc Crafting such commands on the command line is possible though unwieldy In such cases a script is often cleaner and allows you to document the procedure cat gt ncap2 in lt lt E0OF mask var lat gt lat min amp amp lat lt lat max amp amp lon gt lon min amp amp gt lon lt lon max if mask_var total gt 0 Check that mask contains some valid values temperature_avg temperature mask_var avg Average temperature temperature_max temperature mask_var max Maximum temperature Chapter 4 Reference Manual 159 EOF ncap2 S ncap2 in in nc out nc Grids like those produced by the WRF model are complex because one must use global metadata to determine the grid staggering and offsets to translate XLAT and XLONG into real latitudes longitudes
130. three dmn var prm three dmn var permute 10n 1at 1ev The permuted values are three dmn var prm 0 4 8 12 16 20 1 5 9 13 17 21 2 6 10 14 18 22 3 7 11 15 19 23 4 1 6 Attributes Attributes are referred to by var_nm att_nm All the following are valid statements global text Test Attributes Assign a global variable attribute al time timex 20 al long_name Kelvin al min ai min al max ai max al mint al max q 140 NCO 4 5 3 alpha03 User Guide a1 0 a1 min a1 time size 1 a1 max A value list can be used on the RHS of an assignment statement to instantiate a variable al trip1 1 2 3 al triplet al min a1 min a1 max 2 a1 max The netCDF specification allows all attribute types to have a size greater than one The maximum is defined by NC_MAX_ATTRS The following is an ncdump of the metadata for variable a1 double al time ai long name Kelvin al max 199 al min 21 al trip1 1 2 3 al triplet 1 2 110 199 The size method can be used with attributes For example to save an attribute text string in a variable defdim sng len al long_name size sng arr sng len aiQ0long name sng arr now contains Kelvin Attributes defined in a script are stored in memory and are written to Output after script completion To stop the attribute being written use the ram delete method or use a bogus variable name Attribute Propag
131. time dimension unless b is given These operations work as expected in multi file operators Suppose that prs sfc is stored in multiple timesteps per file across multiple files say jan nc feb nc march nc We can now find the three month maximum surface pressure at every point nces y max v prs sfc jan nc feb nc march nc out nc It is possible to use a combination of these operations to compute the variance and standard deviation of a field stored in a single file or across multiple files The procedure to compute the temporal standard deviation of the surface pressure at all points in a single file in nc involves three steps 112 NCO 4 5 3 alpha03 User Guide ncwa v prs sfc a time in nc out nc ncbo 0 v prs_sfc in nc out nc out nc ncra y rmssdn out nc out nc First construct the temporal mean of prs sfc in the file out nc Next overwrite out nc with the anomaly deviation from the mean Finally overwrite out nc with the root mean square of itself Note the use of y rmssdn rather than y rms in the final step This ensures the standard deviation is correctly normalized by one fewer than the number of time samples The procedure to compute the variance is identical except for the use of y var instead of y rmssdn in the final step ncap2 can also compute statistics like standard deviations Brute force implementation of formulae is one option e g ncap2 s prs_sfc_sdn sqrt prs_sfc prs_
132. time size 2 val abs gsl_cdf_tdist_P t_value time size 2 gsl_cdf_tdist_P t_value time size 2 uu else Write RAM variables to disk Usually NCO writes the outputs directly to disk Using RAM variables declared by will shorten running time Output the final results using ram write am write cO am write ci With the group feature all the loops over experiments models and ensemble members can be omitted As we are working on implementing group feature in all NCO operators some functions e g regression and standard deviation over ensemble members may have to wait dk HHH dt dt db db dt dt db OF d P t n until the new versions bin bash Group data output by cmb fl grp sh Annual trend of each model over Greenland and Tibet Time and spatial average standard deviation and anomaly No regression yet needs ncap2 Input files sn LImon all mdl all xpt all nsm 200001 200512 nc Online http nco sourceforge net nco html Annual Average over Regions Execute this script bash ann avg grp sh Input and output directory rc data grp Constants fx sn LImon all mdl all xpt all nsm ms 200001 200512 Time series Greenland cwa 0 w gw d lat 60 0 75 0 d lon 300 0 340 0 a lat lon drc pfx _ tms nc drc pfx _ tms _grl nc 262 NCO 4 5 3 alpha03 User Guide Tibet ncwa
133. to RAM when opened All further metadata and data access occurs in RAM and thus avoids access time delays caused by disk head movement Usually input data is read at most once so it is unlikely that requesting input files be stored in RAM will save much time The likeliest exceptions are files that are accessed numerous times such as those analyzed extensively analyzed by ncap2 Invoking open_ram ram all or diskless all uses much more system mem ory To copy the input file to RAM increases the sustained memory use by exactly the on disk filesize of the input file i e MS FT For large input files this can be a huge memory burden that starves the rest of the NCO analysis of sufficient RAM To be safe use open ram ram all or diskless all only on files that are much say at least a factor of four smaller than your available system RAM See Section 2 9 Memory Requirements page 24 for further details The create_ram switch causes output files to be created in RAM rather than on disk These files are copied to disk only when closed i e when the operator completes Creating files in RAM may save time especially with ncap2 computations that are iterative e g loops and for multi file operators that write output every record timestep or file RAM files provide many of the same benefits as RAM variables in such cases see Section 4 1 12 RAM variables page 149 Two switches ram_all and
134. to know in advance whether all observational data will be valid or not Treatment of any data identified as missing in arithmetic operators NCO averagers ncra nces ncwa do not count any element with the value _FillValue towards the average ncbo and ncflint define a _FillValue result when either of the input values is a _FillValue Sometimes the FillValue may change from file to file in a multi file operator e g ncra NCO is written to account for this it always compares a variable to the _FillValue assigned to that variable in the current file Suffice it to say that in all known cases NCO does the right thing It is impossible to determine and store the correct result of a binary operation in a single variable One such corner case occurs when both operands have differing FillValue attributes i e attributes with different numerical values Since the output result of the operation can only have one FillValue some information may be lost In this case NCO always defines the output variable to have the same FillValue as the first input variable Prior to performing the arithmetic operation all values of the second operand equal to the second FillValue are replaced with the first _FillValue Then the arithmetic operation proceeds as normal comparing each element of each operand to a single _FillValue Comparing each element to two distinct _FillValue s would be much slower and would be no likelier to yield a more satisf
135. two arguments The first argument is a one dimensional array The second argument optional is the sort map srt map below Pass the sort map by reference i e prefix with an ampersand If the sort map does not yet exist then it will be created and returned as an integer type the same shape as the input variable The output of sort on the LHS is a sorted version of the input time sort time amp srt_map print sorted time ascending order and associated sort map n print time print sort always sorts in ascending order The associated sort map therefore re arranges the original randomized time array into ascending order There are two methods to obtain the descending order the user wants 1 We could solve the problem in ascending order the default and then apply the reverse method to re arrange the results 2 We could change the sort map to return things in descending order of time and solve the problem directly in descending order Following shows how to do method one Expand the sort map to srt map 3d the size of the data array Use data array to provide right shape for the expanded sort map Coerce data array into an integer so srt map 3d is an integer Multiply data array by zero so 3 d map elements are all zero Add the 1 d sort map to the 3 d sort map NCO automatically resizes Add the spatial lat lon offsets to each time index de sort using the srt map 3d Use reverse to
136. uiuc edu 8080 opendap data NASAFILE Strided access to netCDF3 file ncks 0 D 1 d time 1 d lev O d lat 0 100 10 d lon 0 100 10 v u velocity http n These servers were operational at the time of writing March 2014 Unfortunately admin istrators often move or rename path directories Recommendations for additional public OPeNDAP servers on which to test NCO are welcome 3 8 Retaining Retrieved Files Availability All operators Short options R Long options 2 rtn retain In order to conserve local file system space files retrieved from remote locations are automatically deleted from the local file system once they have been processed Many NCO operators were constructed to work with numerous large e g 200 MB files Retrieval of multiple files from remote locations is done serially Each file is retrieved processed then deleted before the cycle repeats In cases where it is useful to keep the remotely retrieved files on the local file system after processing the automatic removal feature may be disabled by specifying R on the command line Invoking R disables the default printing behavior of ncks This allows ncks to retrieve remote files without automatically trying to print them See Section 4 7 ncks netCDF Kitchen Sink page 203 for more details 10 For example DAP servers do not like variables with periods in their names even though this is perfectly legal with netCDF S
137. variable it will also contain the time coordinate A short message is added to the history global attribute whenever these ARM specific procedures are executed 3 43 Operator Version Availability All operators Short options r Long options revision version or vrs All operators can be told to print their version information library version copyright notice and compile time configuration with the r switch or its long option equivalent revision The version or vrs switches print the operator version information only The internal version number varies between operators and indicates the most re cent change to a particular operator s source code This is useful in making sure you are working with the most recent operators The version of NCO you are using might be e g 3 9 5 Using r on say ncks produces something like NCO netCDF Operators version 3 9 5 last modified 2008 05 11 built May 12 2008 on neige by zender Copyright C 1995 2008 Charlie Zender ncks version 20090918 This tells you that ncks contains all patches up to version 3 9 5 which dates from May 11 2008 Chapter 4 Reference Manual 129 4 Reference Manual This chapter presents reference pages for each of the operators individually The operators are presented in alphabetical order All valid command line switches are included in the syntax statement Recall that descriptions of m
138. variable p to pressure and t to temperature in netCDF in nc In this case p must exist in the input file or ncrename will abort but the presence of t is optional ncrename v p pressure v t temperature in nc Rename the attribute long_name to largo nombre in the variable u and no other vari ables in netCDF in nc ncrename a uOlong name largo nombre in nc Rename the group g8 to g20 in netCDF4 file in grp nc ncrename g g8 g20 in grp nc Rename the variable g1 lon to longitude in netCDF4 in grp nc ncrename v gi lon longitude in grp nc ncrename v gi lon gi longitude in grp nc amp Alternate ncrename does not automatically attach dimensions to variables of the same name This is done to make renaming an easy way to change whether a variable is a coordinate If you want to rename a coordinate variable so that it remains a coordinate variable you must separately rename both the dimension and the variable ncrename d lon longitude v lon longitude in nc Unfortunately the netCDF4 library had a longstanding bug all versions until 4 3 1 rc5 released in December 2013 that crashed NCO when performing this operation Simultane ously renaming variables and dimensions in netCDF4 files with earlier versions of netCDF is impossible it must instead be done in two separate ncrename invocations e g first rename the variable then rename the dimension to avoid triggering the libary bug A related bug causes unintended side effects
139. wildcard character 0 0 02008 46 da di casters cad que fas 189 5s subkraction etapa nic 178 A a dra 38 I EE camel 38 O dried meu mui cu M Peau e RD E LAE 38 r2 e Desde cO rece Mo rie D Leurs s dale 38 SHO4D ithe wate ds a a e 38 A a ls dle 38 AO a a ido 204 ca lphabetize upeittildsii ieu p a diea ai 204 E E E E A CONAN 18 121 213 Xxappend ipiri ryd onn ipa nien SNES 18 121 213 o A A e este ced en 67 auxiliary lon min lon max lat min lat max dd ee bed ot iet si hc e 67 bfriszhntorieicpeetbnvUieiemileenbnet 103 CEbIBarygsdlbe syl tieu EM 205 A RRA 205 AH a 206 285 cell_methods 0 cc cece cece eee eee 126 chunk byte asi wa eub vn entree aes 85 chunk_dimension 0 0c cece cence 85 Chunk Map jos a seta eR Lada ween as 85 ch nk min eso ed ag E PE CX PR YR 85 CHUN policy 6B where ser se Dei CU epos 85 csch nk scalar AA AN 85 Cll ME co 126 TICA dui A A TNI es 85 cik di ia da IS y 85 z cnk MPa a an ls 85 ccnkmap cnk Bap io seg Mv ei ee ERES 88 EM wv was ed EE eg A DA 85 SR A 4a gy el eared Pelee Macs 85 S cOnkSClwe vo llli wel i S4 nU 85 TSCOOTS A ec A E aE 4T 126 SH CEO i hee Fi Linas NE E 47 126 create ram 2 62h nas e 18 104 HS Gabe n ae Pei EN 207 dbg_lvl debug level 16 24 29 sO Ds Sry ia desee di cave ante lchenece E devas 115 debug level debug level 16 24 cdeflatelsislide efi is EET and sets 101 A AA 4 S
140. with a value equal to the location of the input dataset e g location home zender in nc The xml spr chr and xml spr nmr options customize the strings used as NcML separators for attributes and variables of character type and numeric type respectively Their default separators are and a space zender roulee ncks xml d time 0 3 v two dmn rec var sng in nc Chapter 4 Reference Manual 211 values separator gt abc bcd cde def lt values gt nender rodlsoi ncks xml spr chr v two dmn rec var sng in nc dius separator gt abc bcd cde def efg fgh ghi hij jkl klm lt val oder ncks xml v one dmn rec var in nc svatussid 234567 89 10 lt values gt zender ccuiger d ncks xml spr nmr v one dmn rec var in nc values separator 51 2 3 4 5 6 7 8 9 10 lt values gt Separator elements for strings are a thorny issue One must be sure that the separator element is not mistaken as a portion of the string NCO attempts to produce valid NcML and supplies the xml_spr_chr option to work around any difficulties NCO performs precautionary checks with strstr val spr to identify presence of the separator string spr in data val and when it detects a match automatically switches to a backup separator string However limitations of strstr may lead to false negatives when the separator string occurs in data beyond the first string in multi dimensional NC CHAR arra
141. 1 g1 v vi g1 g1 v1 gt vl ncks G g2 v vi tt g vl gt g2 v1 ncks G g2 g v gt g2 v ncks G g2 g g1 g1 v vl g1 g1 v1 gt g2 v1 Truncate mode Truncate from and Append to path tail Syntax G apn lvl_nbr of levels to truncate NB G apn 1vl nbr is equivalent to G apn lvl_nbr ncks G 1 g g1 v vi g1 v1 gt vi ncks G 1 g g1 g2 v v1 g1 g2 v1 gt g1 v1 ncks G 2 g g1 g2 v vi g1 g2 v1 gt vl ncks G 2 g g1 v vi g1 v1 gt vi ncks G g2 1 v vi g v1 gt g2 v1 ncks G g2 1 g g1 v vi g1 v1 gt g2 v1 ncks G g1 1 g g1 g2 v v1 g1 g2 v1 gt g1 g1 v1 3 13 2 Moving Groups Until fall 2013 netCDF version 4 3 1 pre1 netCDF contained no library function for re naming groups and therefore ncrename cannot rename groups However NCO built on earlier versions of netCDF than 4 3 1 can use a GPE based workaround mechanism to re name groups The GPE mechanism actually moves i e copies to a new location groups Chapter 3 Shared Features 51 a more arduous procedure than simply renaming them GPE applies to all selected groups so in the general case one must move only the desired group to a new file and then merge that new file with the original to obtain a file where the desired group has been renamed and all else is unchanged Here is how to rename group g4 to group f4 with GPE instead of ncrename ncks 0 G f
142. 105 and see Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 pack amp pack short The default packing algorithm is applied and variable is packed to NC SHORT pack byte Variable is packed to NC BYTE pack short Variable is packed to NC SHORT Chapter 4 Reference Manual 147 pack int Variable is packed to NC INT unpack The standard unpacking algorithm is applied NCO automatically unpacks packed data before arithmetically modifying it After modi fication NCO stores the unpacked data To store it as packed data again repack it with e g the pack O function To ensure that temperature is packed in the output file regardless of whether it is packed in the input file one uses e g ncap2 s temperature pack temperature 273 15 in nc out nc Basic Methods These methods work with variables and attributes They have no arguments size Total number of elements ndims Number of dimensions in variable type O Returns the netcdf type see previous section Utility Methods These functions are used to manipulate missing values and RAM variables see Section 4 1 10 Missing values ncap2 page 144 set_miss expr Takes one argument the missing value Sets or overwrites the existing missing value The argument given is converted if necessary to the variable type change miss expr Changes the missing value elements of the variable to the new missing value n b an expensive function
143. 110 2 3 4 6 5 7 3 4 1 ali sort sort a1 print ai sort 1 2 3 3 4 4 5 6 7 10 a2 lon 2 1 4 3 a2 sort sort a2 amp a2 map print a2 1 2 3 4 print a2_map 1 0 3 2 If the map variable does not exist prior to the sort call then it will be created with the same shape as the input variable and be of type NC_INT If the map variable already exists then the only restriction is that it be of at least the same size as the input variable To apply a map use remap var_in srt_map defdim nlat 5 a3 lon 2 5 3 7 a4 nlat lon 1 2 3 4 5 6 n 8 9 10 11 12 13 14 15 16 17 18 19 20 a3 sort sort a3 amp a3 map print a3 map 1504 2y 2 95 a4 sort remap a4 a3 map print a4 sort 1 3 2 4 5 7 6 8 9 11 10 12 13 15 14 16 17 19 18 20 a3_map2 nlat 4 3 0 2 1 154 NCO 4 5 3 alpha03 User Guide a4 sort2 remap a4 a3 map2 print a4 sort2 JS Be Ae Dd 3 10 9 7 6 13 15 14 12 11 18 20 19 17 16 As in the above example you may create your own sort map To sort in descending order apply the reverse method after the sort O Here is an extended example of how to use ncap2 features to hyperslab an irregular region based on the values of a variable not a coordinate The distinction is crucial hyperslabbing based on dimensional indices or coordinate values is straightforward Using the values of single or multi dimen
144. 112 nc 85 x nc tx anm 85 0112 nc This examples works assuming 85 0112 nc has dimensions time and lon and that 85 x nc has no time or lon dimension Group broadcasting simplifies evaluation of multiple models against observations Con sider the input file cmip5 nc which contains multiple top level groups cesm ecmwf and giss each of which contains the surface air temperature field tas We wish to compare these models to observations stored in obs nc which contains tas only in its top level i e root group It is often the case that many models and or model simulations exist whereas only one observational dataset does We evaluate the models and obtain the bias difference between models and observations by subtracting obs nc from cmip5 nc Then ncbo broadcasts i e replicates the observational data to match the group structure of cmip5 nc subtracts and then stores the results in the output file bias nc which has the same group structure as cmip5 nc ncbo 0 cmip5 nc obs nc bias nc 4 ncks H v tas d time 3 bias nc cesm tas Chapter 4 Reference Manual 193 time 3 tas 3 1 ecmwf tas time 3 tas 3 0 giss tas time 3 tas 3 1 As a final example say we have five years of monthly data i e 60 months stored in 8501 8912 nc and we wish to create a file which contains the twelve month seasonal cycle of the average monthly anomaly from the five year mean of this data The following method is just one permutation
145. 14 for NC DOUBLE Clearly values stored as NC DOUBLE i e eight bytes are susceptible to much greater compression than NC FLOAT for a given precision because their significands explicitly contain 53 bits rather than 23 bits Maintaining non biased statistical properties during lossy compression requires special attention The DSD algorithm uses rint OO which rounds toward the nearest even integer 22 Rounding is performed by the internal math library rint family of functions that were standardized in C99 The exact alorithm employed is val rint scale val scale where scale is the nearest power of 2 that exceeds 10 prc and the inverse of scale is used when prc lt 0 For ppc 3 or ppc 2 for example we have scale 1024 and scale 1 128 Chapter 3 Shared Features 95 Thus DSD has no systematic bias However the NSD algorithm uses a bitmask technique susceptible to statistical bias Zeroing all non significant bits is guaranteed to produce numbers quantized to the specified tolerance i e half of the decimal value of the position occupied by the LSD However always zeroing the non significant bits results in quantized numbers that never exceed the exact number This would produce a negative bias in statistical quantities e g the average subsequently derived from the quantized numbers To avoid this bias NSD rounds non significant bits down to zero or up to one in an alternating fashion when processing
146. 180 General Index appending data eee eee ee eee 178 203 appending to fileS o o o 18 121 213 appending variables 18 19 130 221 ARA ct de coo oot ees ete Aa be tet he tees 194 arccosine function 0 cece eee eee eee 178 aresine functioning yang eee hee PRE RE 178 arctangent function 0000 178 ANOS dois SA saci 76 124 area vera tadas E TUS 256 ALCA Ad Had SAE aa des 76 arithmetic OperatorS o oooooooooo 84 85 235 arithmetic processOr 0 0 eee eee eee eee 130 ARM conventions 0c eee ee eee 127 228 array e upDIg CR E OEE SE 136 array Uncle ios ass 136 array ndexing idad eed eRe 131 Array SOLAD ONES ois Bot ay nte 131 array SYNTAR urilxoedu BUSA usare Ru ARS us 131 ATAV ln dea RP eru dac ec ws dedo ut deas 136 arrival value 0 0 eect roo 200 ASI ado ts xu Pie ale date Meee 186 ISI liada e Insee tenga Ru A 178 asinh oss sedie da 178 AO dui cis eer e vene istae a beds 152 assignment statement 0008 131 asynchronous file accesS oooooocoooccoooo 33 AbAniiiexn e v eR p ERG E prr eed Re paved le wees 178 atanh A ER bbb SLM 4 178 attribute inheritance 00 ee eee 140 attribute names eee eee eee 183 230 attribute propagation eee ee 140 attribute syntax 00 131 attribute units cce 50 iccachblese e 79 attributes scier uiv eee vee need p
147. 19115 H20x d in nc The first example creates isotope attributes for all variables whose names contain H20 The second deletes all attributes whose names end in _iso19115 from all variables whose names contain H20 See Section 3 11 Subsetting Files page 43 for more details on using regular expressions As of NCO 4 3 8 November 2013 ncatted accepts full and partial group paths in names of attributes variables dimensions and groups 1 Arithmetic comparisons to NaN and NaN like numbers always return False contrary to the behavior of all other numbers This is difficult to inuit yet is the behavior defined for NaN by IEEE 754 Using NaN for the missing value in datasets is legal We strongly discourage it 188 NCO 4 5 3 alpha03 User Guide Overwrite units attribute of specific lon variable ncatted 0 a units g lon o c degrees west in grp nc Overwrite units attribute of all lon variables ncatted 0 a units lon o c degrees west in grp nc Delete units attribute of all lon variables ncatted 0 a units lon d in grp nc Overwrite units attribute with new type for specific lon variable ncatted 0 a units gi lon o sng degrees_west in grp nc Add new att attribute to all variables ncatted 0 a new att c sng new variable attribute in grp nc Add new grp att group attribute to all groups ncatted 0 a new grp att group c sng new group attribute in grp nc
148. 2 ncatted netCDF Attribute Editor page 183 Linear regression using ncap2 see Section 4 1 ncap2 netCDF Arithmetic Processor page 130 Chapter 7 CMIP5 Example 257 10 Use ncap2 see Section 4 1 ncap2 netCDF Arithmetic Processor page 130 with nco script file i e nco file Move variables around using ncks see Section 4 7 ncks netCDF Kitchen Sink page 203 Flat files example t bin bash Includes gsl rgr nco dj2 222 2 2 2 2222 222 2 2 2 2 2 22 22222222222222 2 22 2 22 22 2 2 2 2 2 2 2 2 2 2 2 22 22222222 2 2 2 2 2 2 After cmb fl sh Example Annual trend of each model over Greenland and Tibet time and spatial average standard deviation anomaly and linear regression Input files data cmip5 snc_LImon_bcc csm1 1_historical_riiip1_185001 200512 nc Output files data cmipb outout snc snc LImon all mdl historical all nsm annual nc Online http nco sourceforge net nco html Annual Average over Regions Execute this script bash ann_avg sh dj2 2222 2 2 2 22222 2 2 2 2 2 2 22 2222222222222 2 2 2 2 2 22 22 2 2 2 2 2 2 2 2 2 2 2 22 2222222 22 2 2 2 2 2 SSS A ee eS ee A A A SS SS Se Parameters drc_in home wenshanw data cmip5 Directory of input files drc out home wenshanw data cmipb output Directory of output files var snc snd Variables rlm LImon Realm xpt historical Experiment coul
149. 263 George Shapavalov 00 0 0005 242 George Shapovalov 0 0 000008 242 George White 0 2 eee eee eee 242 292 getanissQ So 145 geth stnate eon e ve e aa 10 getopt i ill qe hb NEE EE 29 A cogs star ver Ae cia a XGA ME 29 getopt long sie e as 29 getuid ztologewsane ta etat aceti 10 Glenn Davis scere eer deans Sone 241 global attribute ccs 187 231 global attributes 22 122 123 127 184 187 203 204 213 231 233 globbing 22 30 47 177 190 226 228 gutim Q s vec ql Ns Fernand ashe eae Uae winapi 125 GME Seen noi 125 gnu win32 s Sates adie er pe me aes A NM ae Bie 10 GNU ae uer d e oU por E ole neis 29 46 GNU Linux ada fees Ley bes E AA 23 GNUnakefile6 4 e habet Rr E ERE 10 Gods eee mer RE Ped eU ttal 82 GODA Dee euer eere o eer ERUPPEERY 247 BU thls ire o STORES 70 pos jor 46 BEI SV cnt Wr E E ERST RATES Mm 70 grid CAME Vicio dea oserei buit code dieu cee 69 grid Equi Angular eese 69 grid Fixed ertum er veys D PEERS 69 grid EV By ase ode vehe y RI TORNA 69 grid Gaussian Sorn teg rk CURE RETE EDS 69 grid Offset ninia 69 gridhle i nw inrer issuer ria 69 group aggregation 0 0000 197 249 group attributes lee enire ira E uo e na 233 BTOUP names eritar Seek eese n pE E RAN A S 230 Group path zzii Libris ains SER 48 group aggregation 0 cece eee eee 261 group anomaly lt ibis
150. 2k 2k Task 0 Finding input files KKK KK K K K K K K K I 3K 3K 3K 3K 3K 3K K K K K K aK 2K 2K 2K 2K 2k 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K K aK K K K K K 3K 2K 3K 3K 3K 3K 3K 2K 2K 3K 2K K The CCSM model outputs files to a local directory like ptmp zender archive T42x1_40 Each component model has its own subdirectory e g ptmp zender archive T42x1_40 atm ptmp zender archive T42x1_40 cpl ptmp zender archive T42x1_40 ice ptmp zender archive T42x1_40 1nd ptmp zender archive T42x1_40 ocn within which model output is tagged with the particular model name ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0001 01 nc ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0001 02 nc ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0001 03 nc ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0001 12 nc ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0002 01 nc ptmp zender archive T42x1_40 atm T42x1_40 cam2 h0 0002 02 nc or ptmp zender archive T42x1_40 1nd T42x1_40 c1m2 h0 0001 01 nc ptmp zender archive T42x1_40 1nd T42x1_40 c1m2 h0 0001 02 nc ptmp zender archive T42x1_40 1nd T42x1_40 c1m2 h0 0001 03 nc K k ak ak 3k 3K 3K 3K 3K K K K K aK aK aK 2K 2k 2K 2K 3K 3k 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K aK aK 2K 2K K 2K aK 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K 3K K K K K K K K K K K K K K K 2K Task 1 Regional processing BEGGAR GAIA IAI AIRE 3K 3K 3K 3K 3K 3K 3K K K K K K K K K 2k 2k 2k 2k 2k 2k 2k The first task
151. 3 2 The string of trailing zero bits in the rounded values facilitates byte stream compression Note that the NSD and DSD algorithms do not always produce results that are bit for bit identical although they do in this particular case Reducing the preserved precision of NSD rounding produces increasingly long strings of identical bits amenable to compression S Exponent Fraction Significand Decimal Notes O 100000001 0010010000111111011011 3 14159265 Exact O 100000001 0010010000111111011011 3 14159265 NSD 8 O 100000001 0010010000111111011010 3 14159262 NSD 7 O 100000001 0010010000111111011000 3 14159203 NSD 6 O 100000001 0010010000111111000000 3 14158630 NSD 5 O 100000001 0010010000111100000000 3 14154053 NSD 4 O 100000001 0010010000000000000000 3 14062500 NSD 3 O 100000001 0010010000000000000000 3 14062500 NSD 2 O 100000001 0010000000000000000000 3 12500000 NSD 1 The consumption of about 3 bits per digit of base 10 precision is evident as is the coincidence of a quantized value that greatly exceeds the mandated precision for NSD 2 Although the NSD algorithm generally masks some bits for all nsd lt 7 for NC_FLOAT compression algorithms like DEFLATE may need byte size or greater i e at least eight bit bit patterns before their algorithms can take advantage of of encoding such patterns for compression Do not expect significantly enhanced compression from nsd gt 5 for NC_ FLOAT or nsd gt
152. 3 classic ncks fl fmt 64bit in nc foo 364 nc netCDF3 64bit ncks fl fmt netcdf4 classic in nc foo 4c nc netCDF4 classic ncks fl fmt netcdf4 in nc foo 4 nc netCDF4 ncks 3 in nc foo 3c nc netCDF3 classic ncks 3 in nc foo 3c nc netCDF3 classic ncks 6 in nc foo 364 nc netCDF3 64bit ncks 64 in nc foo 364 nc netCDF3 64bit ncks 4 in nc foo 4 nc netCDF4 ncks 4 in nc foo 4 nc netCDF4 ncks 7 in nc foo 4c nc netCDF4 classic ncks 7 in nc foo 4c nc netCDF4 classic Of course since most operators support these switches the conversions can be done at the output stage of arithmetic or metadata processing rather than requiring a separate step Producing netCDF3 CLASSIC or 64BIT files from NETCDF4 CLASSIC files will always work 3 9 4 Autoconversion Because of the dearth of support for netCDF4 amongst tools and user communities includ ing the CF conventions it is often useful to convert netCDF4 to netCDF3 for certain appli cations Until NCO version 4 4 0 January 2014 producing netCDF3 files from netCDF4 files only worked if the input files contained no net CDF4 specific features e g atomic types multiple record dimensions or groups As of NCO version 4 4 0 ncks supports autoconver sion of many netCDF4 features to their closest netCDF3 compatible representations Since converting netCDF4 to netCDF3 results in loss of features automatic down conversion may be a more precise description
153. 4 1 g g4 nco data in grp nc tmp nc Move g4 to f4 ncks 0 x g g4 nco data in grp nc out nc Excise g4 ncks A tmp nc out nc Add f4 to new file If the original group g4 is not excised from out nc step two above then the final output file would contain both g4 and a copy named f4 Thus GPE can be used to both rename and copy groups The recommended way to rename groups when when netCDF version 4 3 1 is availale is to use ncrename see Section 4 11 ncrename netCDF Renamer page 230 One may wish to flatten hierarchical group files for many reasons These include 1 Obtaining flat netCDF3 files for use with tools that do not work with netCDF4 files 2 Splitting apart hierarchies to re assemble into different hierarchies and 3 Providing a subset of a hierarchical file with the simplest possible storage structure ncks 0 G g cesm 3 nco data cmip5 nc cesm nc Extract cesm to The 3 switch specifies the output dataset should be in netCDF3 format the G option flattens all extracted groups and the g cesm option extracts only the cesm group and leaves all other groups e g ecmwf giss 3 13 3 Dismembering Files Let us show how to completely disaggregate or more memorably dismember a hierarchical dataset For now we take this to mean store each group as a standalone flat dataset in netCDF3 format This can be accomplished by looping the previous example over all groups This script ncdism
154. 4 1 0 April 2012 NCO supports data integrity verification using the MD5 digest algorithm This support is currently implemented in ncks and in the multifile concantenators ncecat and ncrcat Activate it with the md5_dgs or md5_digest long options As of NCO version 4 3 3 July 2013 NCO will write the MD5 digest of each variable as an NC CHAR attribute named MD5 This support is currently implemented in ncks and in the multifile concantenators ncecat and ncrcat Activate it with the md5_wrt_att or md5 write attribute long options The behavior and verbosity of the MD5 digest is operator dependent When activating MD5 digests with ncks it is assumed that the user simply wishes to see the digest of every variable and this is done when the debugging level exceeds one This incurs only the minor overhead of performing the hash algorithm for each variable read MD5 digests may be activated in both the one and two filename argument forms of ncks which are used for printing and for sub setting respectively The MD5 digests are shown as a 32 character hexadecimal string in which each two characters represent one byte of the 16 byte digest gt ncks 0 D 2 C md5 v md5 a md5 abc nco data in nc ncks INFO MD5 md5 a Occ175b9c0f1b6a831c399e269772661 md5_a a ncks INFO MD5 md5_abc 900150983cd24fb0d6963 7d28e17f72 lev 0 100 md5_abc 0 2 abc gt ncks 0 D 2 C d lev O md5 v mdb_a mdb_abc
155. 4 5 3 alpha03 User Guide lat 0 90 lev 0 100 lon 1 90 three dmn var 1 1 lat 0 90 lev 0 100 lon 2 180 three dmn var 2 2 lat 1 90 lev 2 1000 lon 1 90 three dmn var 21 21 lat 1 90 lev 2 1000 lon 2 180 three dmn var 22 22 lat 1 90 lev 2 1000 lon 3 270 three dmn var 23 23 ncks C v three dmn var out nc lat 0 90 lev 0 1000 lon 0 270 three dmn var 0 23 lat 0 290 lev 0 1000 lon 1 180 three dmn var 1 22 lat 0 290 lev 0 1000 lon 2 90 three dmn var 2 21 lat 1 90 lev 2 100 10on 1 2180 three dmn var 21 2 lat 1 90 lev 2 100 lon 2 90 three dmn var 22 1 lat 1 90 lev 2 100 lon 3 0 three_dmn_var 23 0 Creating a record dimension named e g time in a file which has no existing record dimension is simple with ncecat ncecat 0 u time in nc out nc Create degenerate record dimension named time Now consider a file with all dimensions including time fixed non record Suppose the user wishes to convert time from a fixed dimension to a record dimension This may be useful for example when the user wishes to append additional time slices to the data As of NCO version 4 0 1 April 2010 the preferred method for doing this is with ncks ncks 0 mk rec dmn time in nc out nc Change time to record dimension Prior to 4 0 1 the procedure to change an existing fixed dimension into a record di mension required three separate commands ncecat followed by ncpdq and then ncwa The rec
156. 4 files that have multiple record dimensions per group into netCDF3 files which are limited to at most one record dimension see Section 3 13 Group Path Editing page 48 As of NCO version 4 4 0 January 2014 the hdn or hidden options print hidden aka special attributes This is equivalent to ncdump s Hid den attributes include Format _DeflateLevel Shuffle Storage ChunkSizes Endianness Fletcher32 and _NOFILL Previously ncks ig nored all these attributes in CDL XML modes Now it prints these attributes as appropriate As of NCO version 4 4 6 September 2014 hdn also prints the extended file format i e the format of the file or server supplying the 206 NCO 4 5 3 alpha03 User Guide data as SOURCE FORMAT Users are referred to the Unidata netCDF Docu mentation http www unidata ucar edu software netcdf docs or the man pages for ncgen or ncdump for detailed descriptions of the meanings of these attributes cdl As of NCO version 4 3 3 July 2013 ncks can print extracted data and meta data to screen i e stdout as valid CDL network Common data form De scription Language CDL is the human readable lingua franca of netCDF ingested by ncgen and excreted by ncdump Compare ncks traditional with CDL printing zender roulee ncks v one nco data in nc one one size RAM one attribute 0 long_name size one 1 type NC FLOAT O dimens
157. 5 1 18 2 21 3 24 0 27 0 30 0 Egsl fit linear xin 1 yin 1 d1 size amp cO amp ci amp covOO amp covO1 amp covi1 amp sumsq print c0 0 2 print c1 2 98545454545 172 NCO 4 5 3 alpha03 User Guide defdim e1 4 xout e1 1 0 3 0 4 0 11 yout e1 0 0 yout gsl_fit_linear_est xout c0 c1 cov00 cov01 covii sumsq print yout 3 18545454545 9 15636363636 12 1418181818 33 04 4 1 22 GSL statistics Wrappers for most of the GSL Statistical functions have been implemented The GSL function names include a type specifier except for type double functions To obtain the equivalent NCO name simply remove the type specifier then depending on the data type the appropriate GSL function is called The weighed statistical functions e g gsl stats wvariance are only defined in GSL for floating point types so your data must of type float or double otherwise ncap2 will emit an error message To view the implemented functions use the shell command ncap2 f grep stats GSL Functions Short gsl stats max short data size t stride size t n double gsl stats int mean int data size t stride size t n double gsl stats short sd with fixed mean short data size t stride size t n doub double gsl stats wmean double w size t wstride double data l size t stride size double gsl stats quantile from sorted data double sorted data l size t stride size Equivalent ncap2 wrapper functions Short gsl s
158. 5 Lon 3 270 LatLon 7 5 There are other workable valid methods to accomplish this rotation yet none are simpler nor more efficient than utilizing MSA user ordering Some final comments on applying this algorithm Be careful to specify hemispheres that do not overlap e g by inadvertently specifying coordinate ranges that both include Greenwich Some users will find using index based rather than coordinate based hyperslabs makes this clearer 3 20 Wrapped Coordinates Availability ncks Short options d dim min max stride Long options dimension dim min max stride dmn dim min max stride Chapter 3 Shared Features 67 A wrapped coordinate is a coordinate whose values increase or decrease monotonically nothing unusual so far but which represents a dimension that ends where it begins i e wraps around on itself Longitude i e degrees on a circle is a familiar example of a wrapped coordinate Longitude increases to the East of Greenwich England where it is defined to be zero Halfway around the globe the longitude is 180 degrees East or West Continuing eastward longitude increases to 360 degrees East at Greenwich The longitude values of most geophysical data are either in the range 0 360 or 180 180 In either case the Westernmost and Easternmost longitudes are numerically separated by 360 degrees but represent contiguous regions on the globe F
159. 54 double precision is converted to a decimal string with at least 17 significant digits and then converted back to double then the final number must match the original 116 NCO 4 5 3 alpha03 User Guide are computed in the type of higher precision NCO by default employs the Fortan Convention for promotion 2 Aggressive aka C Convention The C language is by default much more aggressive and thus wasteful than Fortran and will always implicitly convert single to double precision numbers for no good reason All real number standard C library functions are double precision and C programmers must take extra steps to only utilize single precision arithmetic The high level interpreted data analysis languages IDL Matlab and NCL all adopt the C Convention NCO does not automatically promote NC FLOAT because in our judgement the perfor mance penalty of always doing so would outweigh the potential benefits The now classic text Numerical Recipes in C discusses this point under the section Implicit Conversion of Float to Double That said such promotion is warranted in some circumstances For example rounding errors can accumulate to worrisome levels during arithmetic per formed on large arrays of single precision floats This use case occurs often in geoscientific studies of climate where thousands to millions of gridpoints may contribute to a single av erage If the inputs are all single precision then so should be the ou
160. 565 5 18325 4 24084 3 29843 2 35602 1 41361 0 471204 0 471204 N 1 41361 2 35602 3 29843 4 24084 5 18325 6 12565 7 06806 8 01047 8 95288 9 89529 10 8377 11 7801 12 7225 13 6649 14 6073 15 5497 16 4921 17 4346 18 377 19 3194 20 2618 21 2042 22 1466 23 089 24 0314 24 9738 25 9162 26 8586 27 8011 28 7435 29 6859 30 6283 31 5707 32 5131 33 4555 34 3979 35 3403 36 2827 37 2251 38 1675 39 11 40 0524 40 9948 41 9372 42 8796 43 822 44 7644 45 7068 46 6492 47 5916 48 534 49 4764 50 4188 51 3613 52 3037 53 2461 54 1885 55 1309 56 0733 57 0157 57 9581 58 9005 59 8429 60 7853 61 7277 62 6702 63 6126 64 555 65 4974 66 4398 N 67 3822 68 3246 69 267 70 2094 71 1518 72 0942 73 0367 73 9791 74 9215 75 8639 76 8063 77 7487 78 6911 79 6335 80 5759 81 5183 82 4607 83 4031 84 3456 85 288 86 2304 N 87 1728 88 1152 89 0576 90 Copy of CCSM4 latitude lonn lonn 178 75 177 5 176 25 175 173 75 172 5 171 25 170 168 75 167 5 166 25 165 163 75 162 5 161 25 160 158 75 157 5 156 25 155 153 75 152 5 151 25 150 148 75 147 5 146 25 145 143 75 142 5 141 25 140 138 75 137 5 136 25 135 133 75 132 5 131 25 130 128 75 127 5 126 25 125 123 75 122 5 121 25 120 118 75 117 5 116 25 115 113 75 112 5 111 25 110 108 75 107 5 106 25 105 103 75 102 5 101 25 100 98 75 97 5 96 25
161. 7 189 orphan dimensions esses 209 Output fl OU iets or eR RED Rer as Une 32 TOMO AO micos ci d ca Rex epa 18 121 SE OVE c DEW Hate Nace e ite rnt e Bed teet 18 121 pack policy pck plc esee 217 scpath input path 2 4 5 ib e 30 33 pck map pcK map i s t2er en ve gee 218 pck ple ipck plc ge 217 SE PP Oris gos nite tater Mees col Mal a er tuer Ai aan eke 91 precision_preserving_compression 91 A AN 209 PEO ss o NAS 209 prn fnc tbl 5502 0 0 dios drid os 180 e A RE 10 pth input patita tas 30 33 SQUaMtiZO iio e Oke 91 QU iii erties lie ue 209 TEQUILA IAE 209 215 ni E A al UH EG ta we 209 z rali AA O 24 104 rcd nm ulm_nmM cc 197 a A E O 61 zrecord append ieve see als eee ed Si 61 renormalize ii 3 10 tar mas 74 EXEC a yet AA 37 retain all dimensions 209 S revision gai os ods pal eee 16 127 rgr area area nm sse eee eee 76 turgrfcolzcol nm ici li 75 rgr grd_ttl grd_ttl ooooooocomcomm o 70 Serger gridsgrid AAA Ver X bk Rn TO A 0e t ieee 69 73 trey l tzlat Om onsa arta 75 rgr lat_bnd lat_bnd_nm 75 rgr lat grd lat grd suus 70 rgr lat nbr lat nbr o 72 rgr lat_nrt lat_nrt 00 72 rgr lat weight lat wgt nm 76 rdr Ton Lon M 0 tale te ese Ras Bae ee 75 rgr lon_bnd lon_bnd_nm
162. 8 Subcycl iaces sebbene ORDRE area ae 62 3 19 NIultislabS scs norarvreteeberietesteresent idus iei veis 64 3 20 Wrapped Coordinates sssssssseeeeee eee 66 3 21 Auxiliary Coordinates 0 0 00 cece rre 67 3 22 Grid Generation sieri an ccc ene nen 69 9 29 Reddit A S 73 3 24 UDUnits Support i i Rc a REIS 79 3 25 Rebasing Time Coordinate 0 ccc ene ees 82 3 26 Multiple Record Dimensions 0 00 e eee eee eee 83 9 2 Missing VAMOS sisi eset ea a REG EO eed wane a RR 84 3 28 Chunking senarna aa ee ean on oif eda 85 3 29 Gompr ssiOn arnasan n oe edly RE RU PE NEVER eee Ot eder 91 3 30 Deflation sce OR e SER Eua 101 3 31 MD5 dig sts iter RR pr ERR OE e rad 102 3 92 Butler sues p deta vended as E thee a wel pene ee 103 3 39 RAM CISKSS n od RR COE Reb astute ues e ERU 104 3 94 Packed d t enr ref a 105 Packing Algorithms eben sae ek SDN ICT 105 Unpacking Algorithm ocn ob ene pH SIR eH ee RR 106 Default Handling of Packed Data 0oooooooooommmmmrros 106 Default Handling of Packed Data 0 oooooooooommmmmrr o 107 3 95 Operations Types drek piee wae vez sd ye een bk bet dened ed 107 3 36 Type Conversion 000 c cece ence nee ee menn 113 3 36 1 Automatic type conversion 000 cee eee ee eee 113 3 96 2 Promoting Single precision to Double 115 3 36 3 Manual type conversion 0c cece eee eee es 121 3 37 Ba
163. 9 Assign Operators var op var2 If varl has a missing value then this is the value used in the operation otherwise the missing value for var2 is used If during the element by element operation an element from either operand is equal to the missing value then the missing value is carried through In this way missing values percolate or propagate through an expression Missing values associated with Output variables are stored in memory and are written to disk after the script finishes During script execution its possible and legal for the missing value of a variable to take on several different values Consider the variable int rec var int mss val int time 999 2 3 4 5 6 7 8 999 999 rec var int mss val int FillValue 999 n2 rec var int mss val int rec var int mss val int reverse time n2 999 999 11 11 11 11 11 11 999 999 The following methods manipulate missing value information associated with a variable They only work on variables in Output set miss expr The numeric argument expr becomes the new missing value overwriting the old missing value if any The argument given is converted if necessary to the variable type NB This only changes the missing value attribute Missing val ues in the original variable remain unchanged and thus are no long considered missing values They are orphaned Thus set miss O is normally used only Chapter 4 Reference Manual 145 w
164. 9 999 netCDF files present the links are named 000001 nc to 999999 nc Create enumerated symbolic links bin ls grep nc perl e gt idx 1 while lt STDIN gt chop symlink _ sprintf 06d nc idx ncecat n 999999 6 1 000001 nc foo nc Remove symbolic links when finished The n loop option tells the NCO operator to automatically generate the filnames of the symbolic links This circumvents any OS and shell limits on command line size The symbolic links are easily removed once NCO is finished One drawback to this method is that the history attribute see Section 3 39 History Attribute page 122 retains the filename list of the symbolic links rather than the data files themselves This makes it difficult to determine the data provenance at a later date 2 8 Large Datasets Large datasets are those files that are comparable in size to the amount of random access memory RAM in your computer Many users of NCO work with files larger than 100 MB Files this large not only push the current edge of storage technology they present special problems for programs which attempt to access the entire file at once such as nces and ncecat If you work with a 300 MB files on a machine with only 32 MB of memory then you will need large amounts of swap space virtual memory on disk and NCO will work slowly or even fail There is no easy solution for this The best strategy is to work on a machine with sufficient amounts of m
165. A Acc ee b a Pl TIT 75 PAGE Gs dum eds sert uu ce A 12 lAb SUl AS puce DIEN uA T2 tat vertices cias ee o AUR 75 LAGO ii HESS 76 LAWS a Pall etm Los 12 Latitude ini adds it 75 LD LIBRARY PATH ds lee scald as 11 293 Least Significant Digit 00 92 least_significant_digit 6 95 left hand casting lees 25 136 Legendre polynomial 005 163 Lempel Ziv deflation 0000008 101 Len Makin 22 122 lira ira 242 EA AS bhesdeseb ed hi rU NEC E iS 130 AED sore Antes oie eens gg id cen pe 88 DES 4 LER EIS EIS ens 23 42 LHSz2 sB2bhie Re ded beaded DERE 136 libucodm XIAMEN EDU ERR AE SUE 8 libraries eti iB v ilvirY bte de 11 linkers es seinne dae eee ate 3 ee eee ee tA 33 Lih xoesihrnise ki tR pRRURRRIBAR ie dees DURMPERS 180 LEV M a a ubt creda e tera 8 Ine aspswilewustuebsuravviewustseb au ur dees 178 In38 5c 4g ieri hist Mv rini s ied 190 218 Lts os ve ere Mes Ca He hates 178 login ventre A 178 logarithm base 10 0 005 178 logarithm natural 0 004 178 VON ac Nee VON DRE RU boven a ded eed Hace 75 l n p d NM na EHE Rin 75 LODOS aa o EM e vere 75 124 lon cetd vue evuvee RENE RAT 70 LONE AA 72 A E EU ae ee SURE aS 75 VON vertic s 4 4 en A de eS 75 long double tasado RD bt 180 long Optica casara 29 223 long term average 0 02 e eee eee eee 253 longitude cito 66 75 Lori Sentman
166. CO in January 1998 to aid in maintainance and debugging In March 2001 the final NCO conversion to netCDF3 was completed coincidentally on the same day netCDF 3 5 was released NCO versions 2 0 and higher are built with the DNO NETCDF 2 flag to ensure no netCDF2 interface calls are used However the ability to compile NCO with only netCDF2 calls is worth maintaining because HDF version 4 aka HDF4 or simply HDF available from HDF http hdfgroup org supports only the netCDF2 library calls see http hdfgroup org UG4ir3 html SDS_SD fm12 htm1 47784 There are two versions of HDF Currently HDF version 4 x 5 The 1dd command if it is available on your system will tell you where the executable is looking for each dynamically loaded library Use e g ldd which nces 6 The Hierarchical Data Format or HDF is another self describing data format similar to but more elaborate than netCDF HDF comes in two flavors HDF4 and HDF5 Often people use the shorthand HDF to refer to the older format HDF4 People almost always use HDF5 to refer to HDF5 12 NCO 4 5 3 alpha03 User Guide supports the full netCDF2 API and thus NCO version 1 2 x If NCO version 1 2 x or earlier is built with only netCDF2 calls then all NCO operators should work with HDF4 files as well as netCDF files The preprocessor token NETCDF2 ONLY exists in NCO version 1 2 x to eliminate all netCDF3 calls Only versions of NCO numbered 1 2 x and earlier hav
167. CZUINTG64 sie rit enm TRIES Ds 12 218 NCSUSHBORT erre p is 12 218 ncadd sa ues ae es Me d Re Xe ET E REC 189 DE APh cust lo 130 LE AE reete his 9 25 28 121 130 217 NEAR fc are gee ide 17 101 NCAR MS Sateet i ce a ee en cleat Regd 33 ncattedociu uw rex xa 46 84 85 122 183 ncattget lec dei e 183 211 NGAY oF i ae ee Ther Ch eee b nri e 211 DO oet pare a E NR UR ocd 85 189 cdi a nisse Hos arse nO ete drehen 189 NCOISMEMbEY scsi eges tale 51 54 ncdivide seu See ERI LG DEL CHR 189 ncdnnlist uos i ween tc uM Pu mr d 211 CAME Zidane it UU 211 e A p pes 40 206 208 210 COCA in a a pid CREE 20 197 e A NO 21 85 194 CONE MEE 203 ucflint ERE E NO 21 85 200 General Index ncgen oj 2 os eens ee D tene ET e 206 ncgen hdf xc hexsele exo vies AME th m 206 DES aaa 39 102 178 203 nel convertono s uno Negri Rede 14 222 nclisto eevee ile ie mue elle euis 211 ING Been tec e et te ret ee es et sie 17 222 ncmax Ae oy So bL rie es ee os i de 211 TCI oes t douter Ser lues Y obtestor 211 TOMAR eos hed bes See ee be a ee ee ld 211 INGMD S Aa A SRM LE Lcid A ate 210 TOM Ge ire heed AAA bee ig eae au 189 Memultiply s is esis RA ss es BAM hs 189 Nico script liura Tw 256 nco config log GNU_TRP foo 16 nco configure GNU_TRP foo 16 nco make GNU_TRP f00 0 0000 ee 16 nco input file list issira iss 22 123 nco input file number 22 123 n
168. Caveat Unfortunately threading does not improve NCO throughput i e wallclock time because nearly all NCO operations are I O bound This means that NCO spends negligible time doing anything compared to reading and writing We have seen some and can imagine other use cases where ncwa ncpdq and ncap2 with long scripts will complete faster due to threading The main benefits of threading so far have been to isolate the serial from parallel portions of code This parallelism is now exploited by OpenMP but then runs into the I O bottleneck during output The bottleneck will be ameliorated for large files by the use of MPI enabled calls in the netCDF4 library when the underlying filesystem is parallel e g PVFS or JFS Implementation of the parallel output calls in NCO is not a goal of our current funding and would require new volunteers or funding 7 NCO may modify thr nbr according to its own internal settings before it requests any threads from the system Certain operators contain hard code limits to the number of threads they request We base these limits on our experience and common sense and to reduce potentially wasteful system usage by inexperienced users For example ncrcat is extremely I O intensive so we restrict thr_nbr lt 2 for ncrcat This is based on the notion that the best performance that can be expected from an operator which does no arithmetic is to have one thread reading and one thread writing simultaneously
169. Chunksizes use NCO recommendations default for netCDF3 input Definition Chunksizes are determined by the currently recommended NCO map This is a virtual option that ensures the chunking map is in the subjective opinion of the authors the best map for typical usage As of NCO version 4 4 9 May 2015 this virtual map calls map 1fp cnk map key values nco cnk_nco map_nco Mnemonic Net CDF Operator It is possible to combine the above chunking map algorithms with user specified per dimension though not per variable chunksizes that override specific chunksizes determined by the maps above The user specifies the per dimension chunksizes with the equivalent long options cnk_dmn or chunk dimension The option takes two comma separated arguments dmn nm cnk sz which are the dimension name and its chunksize respectively The cnk_dmn option may be used as many times as necessary The default behavior of chunking depends on several factors As mentioned above when no chunking options are explicitly specified by the user then NCO preserves the chunking characteristics of the input file in the output file This is equivalent to specifying both cnk plc and cnk_map as existing i e cnk_plc xst cnk map xst If output netCDF4 files are chunked with the default behavior of the netCDF4 library When any chunking parameter except cnk_plc or cnk_map is specified such as cnk_dmn
170. Compare the results of default ordering to user ordering for longitude ncks 0 H v Lon d Lon 0 180 d Lon 180 1 0 in nc Lon 0 180 Lon 1 90 Lon 2 0 Lon 3 90 66 NCO 4 5 3 alpha03 User Guide 4 ncks 0 H msa v Lon d Lon 0 180 d Lon 180 1 0 in nc Lon 0 20 Lon 1 90 Lon 2 180 Lon 3 90 The two multi slabs are the same but they can be presented to screen or to an output file in either order The second example shows how to place the western hemisphere after the eastern hemisphere although they are stored in the opposite order in the input file With this background one sees that the following commands suffice to rotate the input file by 180 degrees longitude ncks 0 v LatLon msa d Lon 0 180 d Lon 180 1 0 in nc out nc 4 ncap2 0 s where Lon lt 0 Lon Lon 360 out nc out nc ncks C H v LatLon nco data in nc Lat 0 2 45 Lon 0 180 LatLon 0 0 Lat 0 45 Lon 1 90 LatLon 1 1 Lat 0 45 Lon 2 0 LatLon 2 2 Lat 0 45 Lon 3 90 LatLon 3 3 Lat 1 45 Lon 0 180 LatLon 4 4 Lat 1 45 Lon 1 90 LatLon 5 5 Lat 1 45 Lon 2 0 LatLon 6 6 Lat 1 45 Lon 3 90 LatLon 7 7 ncks C H v LatLon out nc Lat 0 2 45 Lon 0 0 LatLon 0 2 Lat 0 45 Lon 1 90 LatLon 1 3 Lat 0 45 Lon 2 180 LatLon 2 0 Lat 0 45 Lon 3 270 LatLon 3 1 Lat 1 45 Lon 0 0 LatLon 4 6 Lat 1 45 Lon 1 90 LatLon 5 7 Lat 1 45 Lon 2 180 LatLon 6 4 Lat 1 4
171. Create out nc directly on disk ncks no tmp fl in nc in nc ERROR prone Overwrite in nc with itself ncks create ram no tmp fl in nc in nc Create in RAM write to disk ncks open ram no tmp fl in nc in nc Read into RAM write to disk There is no reason to expect the fourth example to work The behavior of overwriting a file while reading from the same file is undefined much as is the shell command cat foo gt foo Although it may work in some cases it is unreliable One way around this is to use create_ram so that the output file is not written to disk until the input file is closed See Section 3 33 RAM disks page 104 However as of 20130328 the behavior of the create_ram and open_ram examples has not been thoroughly tested The NCO authors have seen compelling use cases for utilizing the RAM switches though not yet for combining them with no tmp 1 NCO implements both options because they are largely independent of eachother It is up to power users to discover which best fit their needs We welcome accounts of your experiences posted to the forums Other safeguards exist to protect the user from inadvertently overwriting data If the output file specified for a command is a pre existing file then the operator will prompt the user whether to overwrite erase the existing output file attempt to append to it or abort the operation However in processing large amounts of data
172. Ds PO t d 3 Quote a variable ave 10_ 10 set_miss 90field test 10 0d Quote a attribute three 10 set_mss 10 666Ghell t1 units kelvin Quote a dimension 10 t1 odd c1 10 t1 23 0d The following comments are from the netCDF library definitions and detail the naming conventions for each release netcdf 3 5 1 netcdf 3 6 0 p1 netcdf 3 6 1 netcdf 3 6 2 a zA Z 0 9 PE PEPP PPPE po por J Verify that name string is valid CDL syntax i e all characters are alphanumeric or Also permit or in names for chemists currently making use of these characters but don t document until ncgen and ncdump can also handle these characters in names 182 NCO 4 5 3 alpha03 User Guide netcdf 3 6 3 netcdf 4 0 Final 2008 08 28 XA X A X X kA X A LA A A KF X N Verify that a name string is valid syntax The allowed name syntax in RE form is a zA Z_ UTF8 C Nx00 Nx1FNx7F UTF8 where UTF8 represents a multibyte UTF 8 encoding Also no trailing spaces are permitted in names This definition must be consistent with the one in ncgen l We do not allow because HDF5 does not permit slashes in names as slash is used as a group separator If UTF 8 is supported then a multi byte UTF 8 character can occur anywhere within an identifier We later normal
173. F4 or external netCDF3 compression netCDF3 files compressed by an external utility such as gzip accrue approximately the same benefits shrinkage as netCDF4 although with netCDF3 the user or provider must uncompress e g gunzip the file before accessing the data There is no benefit to rounding numbers and storing them in netCDF3 files unless such custom compression decompression is employed Without that one may as well maintain the undesired precision The user accesses PPC through a single switch ppc repeated as many times as necessary To apply the NSD algorithm to variable u use e g ncks 7 ppc u 2 in nc out nc The output file will preserve only two significant digits of u The options 4 or 7 ensure a netCDF4 format output regardless of the input file format to support internal compression It is recommended though not required to write netCDF4 files after PPC For clarity the 4 7 switches are omitted in subsequent examples NCO attaches attributes that indicate the algorithm used and degree of precision retained for each variable affected by PPC The NSD and DSD algorithms store the attributes number of significant digits and least significant digit respectively 23 ec suggestion by Rich Signell and the nc3tonc4 tool by Jeff Whitaker inspired NCO to implement PPC NCO implements a different DSD algorithm than nc3tonc4 and produces slightly different not bit for bit though self consisten
174. HDF4 files It is simply too much trouble Chapter 1 Introduction 13 Support for netCDF4 in the first arithmetic operator ncbo was introduced in NCO version 4 3 0 March 2013 NCO version 4 3 1 May 2013 completed this support and introduced the first example of automatic group broadcasting See Section 4 3 ncbo netCDF Binary Operator page 189 for more details netCDF4 enabled NCO handles netCDF3 files without change In addition it automag ically handles netCDF4 HDF5 files If you feed NCO netCDF3 files it produces netCDF3 output If you feed NCO netCDF4 files it produces netCDF4 output Use the handy dandy 4 switch to request netCDF4 output from netCDF3 input i e to convert netCDF3 to netCDF4 See Section 3 9 File Formats and Conversion page 38 for more details When linked to a netCDF library that was built with HDF4 support NCO automatically supports reading HDF4 files and writing them as netCDF3 netCDF4 HDF5 files NCO can only write through the netCDF API which can only write netCDF3 netCDF4 HDF5 files So NCO can read HDF4 files perform manipulations and calculations and then it must write the results in netCDF format NCO support for HDF4 has been quite functional since December 2013 For best results install NCO versions 4 4 0 or later on top of netCDF versions 4 3 1 or later Getting to this point has been an iterative effort where Unidata improved netCDF library capabilities in response to our requests
175. IL 241 242 S safeguards ia do pu Eg PENES 17 230 scale facto irri caine cutee 105 198 217 228 scale formati viii a na Ben 26 scales 26 202 Scientific Data Operators 20005 243 SCT sues uber lool Wie dh ete oe i a 88 scott CAappsuicidnusewnsustiburusbe eet 242 SCD cose bas uet A e E dea qo ae tette 10 33 A ar ee OR e PR 130 SOR IP AS qusc edad acdatd NA ANTAN 69 73 SDO ccs uestis DE CHE orb eas 243 seasonal average 00 e EE eee eee 245 SECUTI edes Dates bd ere mee Herta 102 SELL viet tit oe esee A Sta oP Ute 243 Seco lO pity cocaine y WERE ake eae 131 SEPA ici ITUR dale 210 SOLVER i owns dtr ee mec CE RE LC REB 23 35 38 Server Side Distributed Data Reduction amp Analysis erp Woda Hep XR 243 server side processing 00 e ee eee 36 243 Set Missi vn cid 144 Sftp ote ASS Shes tains ESSE her 10 33 SGI eque eee QU d Eu pana 8 mn Shells 4 n her A Pessac dra Aeiceseiu 211 shared memory machines 04 24 shared memory parallelism 27 Shell cantado ds sanh 22 47 80 177 190 211 sig nedness A o aiken okt ph PEINT 26 significand cr adn 93 94 MM A O La ter a 178 Sin function A eae de ee RGA 178 single precision 02 e eee eee eee 180 GID cies ce etisset rwr p ais ERR UTERE 178 SIZE css DISSI it A 147 JM A a 76 SLD Swath like Data ooooooocccccocoo 150 MPa ld dia 27 SUWE rv AS A AERE RIS 72 solid angle 4 ice e e
176. IVES sean see ies ba iter Sa 44 reger is tier ete td aud E 46 TOTES OM des ds 256 regressions archive ooooooocccocccocccccccos 16 LO a A 266 Tegridding on n E XU RENS RETE 73 regular expressions 22 30 46 177 184 187 TOMA its reward ares 152 Remik Ziemlinski 0 0 e eee e ee 242 remote files oooooooooommmccrrmo o 10 33 rename groups wasn ei ie aoe ek i eue 50 renaming attributes 00 230 266 renaming dimensions 230 266 renaming QLOUPS 6 eee ee 230 renaming variables 230 256 266 renormalized regridding 0 74 reporting bugs aerate PERRO EUER 15 reshape variables 00 cece eee eens 217 restrictie toad es Ws Pee ae eas 9 R3 gee iss ees 9 reverse dabas css rip ea p ha MEUS 223 reverse dimensions s 217 222 223 LEVETSE ey enses A sealed 139 General Index Rich Signell io ten 95 242 TIU Lr ecnpE P PERRO RUE Pet ERR E 178 rint it RUF BRL ENPEHA 94 IDs ol hM een a ia ML RE Went 107 PMS SAM ve ora de mao ao RE 107 EMSS c iv Ebnehus EINE Sean thea ees 146 root mean square eee eee 107 Rorik Peterson sip co a Vea Ml i RR eR 241 IOUndo41u iik eda eee be Pn a EAD QE 178 ondine i ce RP ita rw 91 119 rounding functions 00 000 178 RPBNMIzuistiiergkeatbe eae ladies rad NINE 14 running average crecer bs etx t deir hes 225 Russ Rew i Aeon i bue LIE E
177. O forums described below you might first register https sf net account register php your name and email address with SourceForge net or else all of your postings will be attributed to nobody Once registered you may choose to monitor any forum and to receive or not email when there are any postings including responses to your questions We usually reply to the forum message not to the original poster If you want us to include a new feature in NCO check first to see if that feature is already on the TODO file TODO list If it is why not implement that feature yourself and send us the patch If the feature is not yet on the list then send a note to the NCO Discussion forum http sf net p nco discussion 9829 Read the manual before reporting a bug or posting a help request Sending questions whose answers are not in the manual is the best way to motivate us to write more docu mentation We would also like to accentuate the contrapositive of this statement If you think you have found a real bug the most helpful thing you can do is simplify the problem to a manageable size and then report it The first thing to do is to make sure you are running the latest publicly released version of NCO Once you have read the manual if you are still unable to get NCO to perform a doc umented function submit a help request Follow the same procedure as described below for reporting bugs after all it might be a bug That is describe what
178. P ica siia ena edible 24 25 27 73 operation typeS eee eee 107 225 236 operator speed 11 18 26 27 85 183 231 Operators onae iu is 5 OptlPuters lia a oa 243 Orion Powlawski 00 cece eee ee eee 242 A eas ia adda ds Santee ed das 124 238 orphan dimensions 2 00 e sees 209 OS a criss rst nb de a Ad erat rd times 8 output file uode Veta trate tastes 22 32 outp t palh rer b ecl eek hoe e Eres 34 OVeEVIOW i ecve word geo epe VERE SOL ER Nd 26 overwriting fileS oooooooooo o 18 121 P O prera hp net pai nb nets 146 pack ciar rides E 105 pack byte ooococcccccccccoccnccnccona eee 146 packant i bik eit ee ie etie 147 packsshort 22 pondre 146 packing ubxepEe 36 86 105 198 217 228 A A RASA 218 packing policy ocras Tusinan eii Oo TERR 217 296 papers aisre E dt eck dum Ns 26 Parallel aa envia RING ep SPINE RR 273 parallelescteii tit saat xw LEEREN 273 parallelism 2 eene stage rua 27 243 Parser sas AN 130 pasting variables 0 00 000008 19 path stell lebe MW heh os Seed db 8 path Can iced PERSA IE ee edad oes 8 Path Seale suse uss bie heeded I ete I etn 8 Patrice Dimas ovn re erum RE 242 Patrick Kursawe lees eese 242 pattern matching 22 46 184 187 PayPal eene id til v LE 241 pck mn ap oer eR eC Wee a eR 218 A Ri rAikerb md iemifrpnsee 217 peak memory usage 2 0008 24 104 Pedro Vic
179. PeNDAP enabled NCO clients continue to support orthogonal features such as UDUnits see Section 3 24 UDUnits Support page 79 4 ncks u C H v wvl d wvl 0 4 micron 0 7 micron p http thredds test ucar edu thredds dodsC testdods in 4 nc 4 wvl 0 25e 07 meter The next command is a more advanced example which demonstrates the real power of OPeNDAP enabled NCO clients The ncwa client requests an equatorial hyperslab from remotely stored NCEP reanalyses data of the year 1969 The NOAA OPeNDAP server hope fully serves these data The local ncwa client then computes and stores locally the regional mean surface pressure in Pa ncwa C a lat lon time d lon 10 10 d 1at 10 10 1 tmp p http www esrl noaa gov psd thredds dodsC Datasets ncep reanalysis dailyavgs surface pres sfc 1969 nc foo nc All with one command The data in this particular input file also happen to be packed see Section 4 1 11 Methods and functions page 145 although this complication is transparent to the user since NCO automatically unpacks data before attempting arithmetic We are most familiar with the OPeNDAP ability to enable network transparent data access OPeNDAP has many other features including sophisticated hyperslabbing and server side processing via constraint expressions If you know more about this please consider writing a section on OPeNDAP Capabilities of Interest to NCO Users for incorporation in the NCO User Guide
180. Permute Dimensions Quickly SYNTAX ncpdq 3 4 6 7 A a dim L c cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stride F G gpe dscl g grp glb h hdf hdr pad nbr L dfl lvl 1 path M pck map mrd no tmp f1 0 o output file P pck plcl p path ppc R 7r ram a11 t thr nbr U unn v var X x input file output file DESCRIPTION ncpdq performs one not both of two distinct functions packing or dimension permu tation ncpdq is optimized to perform these actions in a parallel fashion with a minimum of time and memory The pdq may stand for Permute Dimensions Quickly Pack Data Quietly Pillory Dan Quayle or other silly uses Packing and Unpacking Functions The ncpdq packing and unpacking algorithms are described in Section 4 1 11 Methods and functions page 145 and are also implemented in ncap2 ncpdq extends the func tionality of these algorithms by providing high level control of the packing policy so that users can consistently pack and unpack entire files with one command The user specifies the desired packing policy with the P switch or its long option equivalents pck_plc and pack policy and its pck_plc argument Four packing policies are currently im plemented Packing and Re Packing Variables de
181. R QT s dd heen datas 14 Nick Boweros dd Cid 242 NINTAP So state ale a da abe ees 30 226 228 NOCNETGCDE 25 2 IUe tios 11 non coordinate grid properties 125 295 non rectangular grids 00 157 non standard grids 000085 157 normalization lesse 237 North American Regional Reanalysis NARR SSS 150 Not a Number 0 sees eects 186 NRA Lud ure ete nutre yet ox thd accented ed 243 nrnet l4 A DAR 34 NSD 5ascoeebrie A qae obi CE 92 NSE Scania 243 NT Microsoft operating system 10 NUD i ere e a eR eg 186 219 NUL termination sees 186 nulloperation Rl eR E 202 number literals ncap2 seessessuse 140 Number of Significant DigitS 92 Mumber_misSS 0 ccc cee ee 145 number_of_significant_digits 95 n merator ziasob R4 vw pw moy reserve 109 237 DV spree A EE MEE eee aoe e GM 75 NVODS wa oen ipe ita Ren ta ie I eae a 35 xt IST Ae Cae abe eed ae eee eft n 218 O oceanography sse i xie Mesa hoe bee GPS PR 35 octal d mpeczciou 3 04 a AA EA 40 OQ s creat Veri aenea E pd Sarees 40 OMP NUM THREADS 2 eee eee eens 28 on line documentation 0 c ee eee eee 7 open SOUL CE Ars cerea eH die sup RR e Saas 1 35 Open source Project for a Network Data Access Protocols vai weds sed RE e 35 OPENDAP cascade ID LA 35 OpenM
182. Section 4 12 1 Mask condition page 236 is false Points x in either of these two categories do not contribute to x they are ignored We now define these criteria more rigorously Each x has an associated Boolean weight u whose value is 0 or 1 false or true The value of ju is 1 true unless x equals the missing value see Section 3 27 Missing Values page 84 for the variable Thus for a variable with no _FillValue attribute ju is always 1 All NCO arithmetic operators ncbo ncra nces ncflint ncwa treat missing values analogously Besides weighted averaging ncwa ncra and nces also compute some common non linear operations which may be specified with the y switch see Section 3 35 Operation Types page 107 The other rank reducing operations are simple variations of the generic weighted mean described above The total value of x y tt1 is i N l Note that the total is the same as the numerator of the mean of x and may also be obtained in ncwa by using the N switch see Section 4 12 ncwa netCDF Weighted Averager page 235 The minimum value of x y min is Tj min u mMm W121 l2M2W32 3 UNMNWNEN Analogously the maximum value of x y max is Tj max uymq121 l2M3W9223 UNMNWNEN Thus the minima and maxima are determined after any weights are applied The minimum absolute value of x y mibs is Tj min um w 11 q2m2w9 22 jv mv ww Tp Analogously the maximum abso
183. Summary of foo 4c nc filetype NC FORMAT NETCDF4 CLASSIC O groups 4 ncks M foo 4 nc Summary of foo 4 nc filetype NC FORMAT NETCDF4 O groups 11 The reason and mnemonic for 7 is that NETCDF4_CLASSIC files include great features of both netCDF3 compatibility and netCDF4 compression chunking and well 3 4 7 40 NCO 4 5 3 alpha03 User Guide This method requires a netCDF4 enabled NCO version 3 9 0 i e from 2007 or later As of NCO version 4 4 0 January 2014 ncks will also print the extended or underlying format of the input file The extended filetype will be one of the six underlying formats that are accessible through the netCDF API These formats are NC FORMAT NC3 clas sic and 64 bit versions of netCDF3 formats NC FORMAT NC HDF5 classic and extended versions of netCDF4 and pure HDF5 format NC FORMAT NC HDFA4 HDF4 format NC_ FORMAT PNETCDF PnetCDF format NC FORMAT DAP2 accessed via DAP2 protocol and NC FORMAT DAP4 accessed via DAP2 protocol For example 4 ncks D 2 M hdf hdf Summary of hdf hdf filetype NC FORMAT NETCDFA representation of Y extended underlying filetype NC FORMAT HDFA4 O groups ncks D 2 M http thredds test ucar edu thredds dodsC testdods in nc Summary of http thredds test ucar edu thredds dodsC testdods in nc filetype NC FORMAT CLASSIC representation of extended underlying filetype NC FORMAT DAP2 O groups nck
184. We eb EE AR 261 group dimension permutation 261 group spatial averaging 0 261 group standard deviation 261 group temporal averaging 261 BIOUDS i kh hus rs tae EXE 12 187 groups AVETAGING 6 6 eee 255 groups creating 249 groups MOVING ifs ete 4k i E GR ae 50 groups renaming clle 50 growing dimensions sees 176 gsl sf bessel NAAA s 162 gsl sf gammas c ies s EP pepe EIAS 161 gsl sf legendre Pl 0 tue 163 GS rra iaa zem eig 9 161 169 BUZZ Pica 95 Mia RAE wv 76 124 238 PP a ah et cu he 91 95 H HAVONCCL nse ver qe UEM 14 AGE i decre ON 14 NCO 4 5 3 alpha03 User Guide h sh conce ROC VE NOV 102 hdf name voeem oe vat e bs ou d ee Gs 14 HIDE cete ep PE RES 11 38 206 243 HDF unpacking 022 eee 106 HDE ssc acts do 13 206 HDF 4 UNKNOWN 2 ida ip 13 ADE on POUR 12 13 95 Hd sed yates E TU HEEHUEREUC I eV eS 206 Mel Dit tati 15 Henry Butowsky rece iiaa nri ronime eee eee 241 TEDDY ose persis kha LR RE MEER 218 heh sht ecl e tiw sa bane Let ark 218 hidden attributes 00 000 205 hidden features 0 00 0 cr 209 Hierarchical Data Format 11 history 23 33 122 127 183 198 213 history of appended files 123 HNC PONS sehen ae ota ires bc rUTI 206 HP aches da 8 OPS rap tees tee tek eek 34
185. a packed with one algorithm should never be unpacked with the other doing so will result in incorrect answers Unfortunately few users are aware that their datasets may be packed and fewer know the details of the packing algorithm employed This is what we in the bizness call an interoperability issue because it hampers data analysis performed on heterogeneous systems As described below NCO automatically unpacks data before performing arithmetic This automatic unpacking occurs silently since there is usually no reason to bother users with these details There is as yet no generic way for NCO to know which packing convention was used so NCO assumes the netCDF convention was used NCO uses the same convention for unpacking unless explicitly told otherwise with the hdf_upk also hdf_unpack switch Until and unless a method of automatically detecting the packing method is devised it must remain the user s responsibility to tell NCO when to use the HDF convention instead of the netCDF convention to unpack If your data originally came from an HDF file e g NASA EOS then it was likely packed with the HDF convention and must be unpacked with the same convention Our recom mendation is to only request HDF unpacking when you are certain Most packed datasets encountered by NCO will have used the netCDF convention Those that were not will hope fully produce noticeably weird values when unpacked by the wrong algorithm Before or
186. a variable name 4 1 2 Expressions Expressions are the fundamental building block of ncap2 Expressions are composed of variables numbers literals and attributes The following C operators are overloaded and work with scalars and multi dimensional arrays Arithmetic Operators Binary Operators gt gt lt lt I amp amp gt gt lt lt Unary Operators dome Conditional Operator expl exp2 exp3 Assign Operators In the following section a variable also refers to a number literal which is read in as a scalar variable Arithmetic and Binary Operators Consider var op var2 Precision e When both operands are variables the result has the precision of the higher precision operand e When one operand is a variable and the other an attribute the result has the precision of the variable 132 NCO 4 5 3 alpha03 User Guide e When both operands are attributes the result has the precision of the more precise attribute e The exponentiation operator is an exception to the above rules When both operands have type less than NC FLOAT the result is NC FLOAT When either type is NC DOUBLE the result is also NC DOUBLE QAM Rank e The Rank of the result is generally equal to Rank of the operand that has the greatest number of dimensions e If the dimensions in var2 are a subset of the dimensions in varl then its possible to make var2 conform to varl through broadcast
187. ab 37 40 40 42 43 45 46 49 52 58 60 62 62 65 67 The following all return variables of type NC DOUBLE defdim b1 1000 b1 b1 0 8 b2 gsl ran exponential b1 b2 avg b2 avgO print b2 avg b2 avg 0 756047976787 b3 gs1 ran gaussian b1 b3_avg b3 avg b3_rms b3 rms print b3_avg b3_avg 0 00903446534258 print b3_rms b3_rms 0 81162979889 176 NCO 4 5 3 alpha03 User Guide b4 b1 10 0 b5 b1 20 0 b6 gs1_ran_flat b4 b5 b6_avg b6 avg print b6_avg b6 avg 15 0588129413 4 1 24 Examples ncap2 See the ncap in and ncap2 in scripts released with NCO for more complete demonstrations of ncap2 functionality script available on line at http nco sf net ncap2 in Define new attribute new for existing variable one as twice the existing attribute dou ble_att of variable att var ncap2 s one new 2 att_var double_att in nc out nc Average variables of mixed types result is of type double ncap2 s average var_float var_double var_int 3 in nc out nc Multiple commands may be given to ncap2 in three ways First the commands may be placed in a script which is executed e g tst nco Second the commands may be individually specified with multiple s arguments to the same ncap2 invocation Third the commands may be chained into a single s argument to ncap2 Assuming the file tst nco contains the commands a23 b 4 c sqrt a 2 b 2 then
188. ability ncap2 ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options C c Long options no coords no crd crd coords By default coordinates variables associated with any variable appearing in the input file will be placed in the output file even if they are not explicitly specified e g with the v switch Thus variables with a latitude coordinate lat always carry the values of lat with them into the output file This feature can be disabled with C which causes NCO to not automatically add coordinates to the variables appearing in the output file However using C does not preclude the user from including some coordinates in the output files simply by explicitly selecting the coordinates with the v option The c option on the other hand is a shorthand way of automatically specifying that all coordinate variables in the input files should appear in the output file Thus c allows the user to select all the coordinate variables without having to know their names As of NCO version 4 4 5 July 2014 both c and C honor the CF ancillary variables convention described in Section 3 41 CF Conventions page 124 As of NCO version 4 0 8 April 2011 both c and C honor the CF bounds convention described in Section 3 41 CF Conventions page 124 As of NCO version 4 4 9 May 2015 both c and C honor the CF c
189. acking caveats First the interaction of packing and data equal to the _FillValue is complex Test the FillValue behavior by performing a pack unpack cycle to ensure data that are missing stay missing and data that are not 220 NCO 4 5 3 alpha03 User Guide misssing do not join the Air National Guard and go missing This may lead you to elect a new FillValue Second ncpdq actually allows packing into NC CHAR with e g 1t chr However the intrinsic conversion of signed char to higher precision types is tricky for values equal to zero i e for NUL Hence packing to NC CHAR is not documented or advertised Pack into NC BYTE with e g flt_byt instead Dimension Permutation ncpdq re shapes variables in input file by re ordering and or reversing dimensions specified in the dimension list The dimension list is a whitespace free comma separated list of dimension names optionally prefixed by negative signs that follows the a or long options arrange permute re order or rdr switch To re order variables by a subset of their dimensions specify these dimensions in a comma separated list following a e g a lon lat To reverse a dimension prefix its name with a negative sign in the dimension list e g a lat Re ordering and reversal may be performed simultaneously e g a lon lat time lev 6 Users may specify any permutation of dimensions including per
190. actory answer In practice judicious choice of _FillValue values prevents any important information from being lost 3 28 Chunking 20 For example the DOE ARM program often uses att type NC CHAR and _FillValue 99999 86 NCO 4 5 3 alpha03 User Guide Availability ncap2 ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options none Long options cnk_byt cnk sz chunk byte cnk sz cnk dmn dmn nm cnk sz chunk dimension dmn nm cnk sz cnk map cnk map chunk map cnk map cnk min var_sz chunk_min var sz cnk plc cnk plc chunk policy cnk plc cnk_scl cnk sz chunk scalar cnk sz All netCDF4 enabled NCO operators that define variables support a plethora of chunk size options Chunking can significantly accelerate or degrade read write access to large datasets Dataset chunking issues are described by THG and Unidata here http www hdfgroup org HDF5 doc H5 user Chunking html here http www unidata ucar edu blogs developer en entry chunking data why it matters and here http www unidata ucar edu blogs developer en entry chunking_data_choosing_shapes NCO authors are working on generalized algorithms and applications of chunking strategies stay tuned for more in 2015 The NCO chunking implementation is designed to be flexible Users control four aspects of the chunking implementation T
191. add offset stored at the original unpacked precision of the variable Let min and max be the minimum and maximum values of x scale factor max min ndrv add_offset min max 2 pck upk add offset scale factor ndry x upk min max 2 max min where ndrv is the number of discrete representable values for given type of packed variable The theoretical maximum value for ndrv is two raised to the number of bits used to store the packed variable Thus if the variable is packed into type NC SHORT a two byte datatype then there are at most 2 65536 distinct values representable In practice the number of discretely representible values is taken to be two less than the theoretical maximum This leaves space for a missing value and solves potential problems with rounding that may occur during the unpacking of the variable Thus for NC SHORT ndrv 65536 2 65534 Less often the variable may be packed into type NC CHAR where ndrv 25 2 256 2 254 or type NC INT where where ndrv 2 2 4294967295 2 4294967293 One useful feature of lossy netCDF packing algorithm is that additional loss less packing algorithms perform well on top of it Unpacking Algorithm Unpacking The unpacking algorithm depends on the presence of two attributes scale factor and add offset If scale factor is present for a variable the data are multiplied by the value scale factor after the d
192. age restrictions This refined map speci fies the final output dimensionality Third we translate the output dimensionality into one dimensional memory offsets for each datum according to the C language convention for multi dimensional array storage Dimension reversal changes the ordering of data though not the rank or dimensionality and so is part of the third step Dimensions R disjoint from D play no role in re ordering The first step taken to re order a variable is to determine S R is constant for all variables whereas D and hence is variable specific S is empty if R D This may be the case for some extracted variables The user may explicitly specify the one to one mapping of input to output dimension order by supplying with a a re order list R such that S N In this case D Sn The degenerate case occurs when D S This produces the identity mapping D Dn The mapping of input to output dimension order is more complex when S N In this case D D for the N S dimensions D S For the S dimensions D S Di Ss EXAMPLES Pack and unpack all variables in file in nc and store the results in out nc ncpdq in nc out nc Same as ncpack in nc out nc ncpdq P all new M flt sht in nc out nc Defaults ncpdq P all xst in nc out nc ncpdq P upk in nc out nc Same as ncunpack in nc out nc ncpdq U in nc out nc Same as ncunpack in nc out nc The first two commands pack any unpacked varia
193. ages and disadvantages The primary disadvantages are speed and size Double precision arithmetic is 10 60 slower than and requires twice the memory of single precision arithmetic The primary advantage is that rounding errors in double precision are much less likely to accumulate to values near the precision of the underlying geophysical variable For example if we know temperature to five significant digits then a rounding error of 1 bit could affect the least precise digit of temperature after 1 000 10 000 consecutive one sided rounding errors under the worst possible scenario Many geophysical grids have tens of thousands to millions of points that must be summed prior to normalization to compute an average It is possible for single precision rouding errors to accumulate and degrade the precision in such situtations Double precision arithmetic mititgates this problem so dbl would be warranted This can be seen with another example averaging a global surface temperature field with ncwa The input contains a single precision global temperature field stored in TREFHT produced by the CAM3 general circulation model GCM run and stored at 1 9 by 2 5 degrees resolution This requires 94 latitudes and 144 longitudes or 13 824 total surface gridpoints a typical GCM resolution in 2008 2013 These input characteristics are provided only to show the context to the interested reader equivalent results would be found in statistics of any data
194. amples may all be assumed to end with the standard options map map nc in nc out nc ncks rgr lat latitude rgr lon longitude ncks rgr col column rgr lat wgt lat wgt ncks rgr bnd bounds rgr lat bnd lat bounds rgr lon bnd lon bounds ncks rgr bnd vertices rgr lat bnd lat vrt rgr lon bnd 1lon vrt The first command causes the regridder to associate the latitude and longitude dimen sions with the dimension names latitude and longitude instead of the defaults lat and lon The second command causes the regridder to associate the independent columns in an unstructured grid with the dimension name column instead of the default ncol and the variable containing latitude weights to be named lat_wgt instead of the default gw The Chapter 3 Shared Features 77 third command associates the latitude and longitude bounds with the dimension bounds instead of the default nbnd and the variables lat_bounds and lon_bounds instead of the defaults lat_bnds and lon_bnds respectively The fourth command associates the lati tude and longitude bounds with the dimension vertices instead of the default nv and the variables lat_vrt and lon_vrt instead of the defaults lat_vertices and lon_vertices respectively When used with an identity remapping files regridding can signficantly enhance the metadata and therefore the dataset usability Consider these selected metadata those unchanged are not shown for brevity associated wi
195. an errors Emean are orders of magnitude smaller because quantization produces over and under estimated values in balance When prc 7 quantization of single precision values is ineffective because all available bits are used to represent the maximum precision of seven digits The maximum and mean absolute errors Emabs and Emebs are nearly identical across algorithms precisions and dataset types This is consistent with both the artificial data and empirical data being random and thus exercising equally strengths and weaknesses of the algorithms over the course of millions of input values We generated artificial arrays with many different starting values and interval spacing and all gave qualitatively similar results The results presented are the worst obtained The artificial data has much smaller mean error Emean than the observational analysis The reason why is unclear It may be because the temperature field is concentrated in particular ranges of values and associated quantization errors prevalent on Earth e g 200 T 320 It is worth noting that the mean error Emean lt 0 01 for 1 lt prc lt 6 and that Emean is typically at least two or more orders of magnitude less than Emabs Thus quantized values with precisions as low as prc 1 still yield highly significant statistics by contemporary scientific standards Testing shows that PPC quantization enhances compression of typical climate datasets The degree of enhancement dep
196. an or equal to the rank of the same variable in file 2 Of course the size of all dimensions common to both file 1 and file 2 must be equal When computing anomalies from the mean it is often the case that file 2 was created by applying an averaging operator to a file with initially the same dimensions as file 1 often file 1 itself In these cases creating file 2 with ncra rather than ncwa will cause the ncbo operation to fail For concreteness say the record dimension in file 1 is time If file 2 was created by averaging file 1 over the time dimension with the ncra operator rather than with the ncwa operator then file 2 will have a time dimension of size 1 rather than having no time dimension at all In this case the input files to ncbo file 1 and file 2 will have unequally sized time dimensions which causes ncbo to fail To prevent this from occuring use ncwa to remove the time dimension from file 2 See the example below ncbo never operates on coordinate variables or variables of type NC CHAR or NC STRING This ensures that coordinates like e g latitude and longitude are physically meaningful in the output file file 3 This behavior is hardcoded ncbo applies special rules to some CF defined and or NCAR CCSM or NCAR CCM fields such as ORO See Section 3 41 CF Conventions page 124 for a complete description Finally we note that ncflint see Section 4 6 ncflint netCDF File Interpolator page 200 is designed for file inte
197. ands work even when the HDF4 file contains netCDF4 atomic types e g unsigned bytes 64 bit integers because NCO can autoconvert everything to atomic types supported by netCDF3 As of NCO version 4 4 4 May 2014 both ncl convert2nc and NCO have built in automatic workarounds to handle element names that contain characters that are legal in HDF though are illegal in netCDF For example slashes and leading special characters are are legal in HDF and illegal in netCDF element i e group variable dimension and attribute names NCO converts these forbidden characters to underscores and retains the original names of variables in automatically produced attributes named hdf_name Finally in February 2014 we learned that the HDF group has a project called H4CF described here http hdfeos org software h4cflib php whose goal is to make HDF4 files accessible to CF tools and conventions Their project includes a tool named h4tonccf that converts HDF4 files to netCDF3 or netCDF4 files We are not yet sure what advantages or features h4tonccf has that are not in NCO though we suspect both methods have their own advantages Corrections welcome As of 2012 netCDF4 is relatively stable software Problems with netCDF4 and HDF libraries have mainly been fixed Binary NCO distributions shipped as RPMs and as debs have used the netCDF4 library since 2010 and 2011 respectively Prior to NCO version 4 4 0 January 2014 we recommended th
198. ans we cannot fix all build failures The information we need to diagnose and often fix build failures are the three files output by GNU build tools nco config log GNU_TRP foo nco configure GNU_ TRP foo and nco make GNU_TRP foo The file configure eg shows how to produce these files Here GNU_TRP is the GNU architecture triplet the chip vendor OS string returned by config guess Please send us your improvements to the examples supplied in configure eg The regressions archive at http dust ess uci edu nco rgr contains the build output from our standard test systems You may find you can solve the build problem yourself by examining the differences between these files and your own Chapter 2 Operator Strategies 17 2 Operator Strategies 2 1 Philosophy The main design goal is command line operators which perform useful scriptable operations on netCDF files Many scientists work with models and observations which produce too much data to analyze in tabular format Thus it is often natural to reduce and massage this raw or primary level data into summary or second level data e g temporal or spatial averages These second level data may become the inputs to graphical and statistical pack ages and are often more suitable for archival and dissemination to the scientific community NCO performs a suite of operations useful in manipulating data from the primary to the second level state Higher level interpretive languages e
199. any of these command line switches are provided only in Chapter 3 Shared features page 27 to avoid redundancy Only options specific to or most useful with a particular operator are described in any detail in the sections below 130 NCO 4 5 3 alpha03 User Guide 4 1 ncap2 netCDF Arithmetic Processor d m ncap2 understands a relatively full featured language of operations including loops con ditionals arrays and math functions ncap2 is the most rapidly changing NCO operator and its documentation is incomplete The distribution file data ncap2 tst nco contains an up to date overview of its syntax and capabilities The data nco distribution files especially bin cnt nco psd_wrf nco and rgr nco contain in depth examples of ncap2 solutions to complex problems SYNTAX ncap2 3 4 6 7 A C c D dbg F f glb h hdf hdr pad nbr L dfl 1vl 1 path no tmp 1 0 o output filel p path R r ram a11 s algebra S fl nco t thr_nbr v input file output file DESCRIPTION ncap2 arithmetically processes netCDF files The processing instructions are contained either in the NCO script file 1 nco or in a sequence of command line arguments The options s or long options spt or script are used for in line scripts and s or long options fl_spt or script file are used to provide the filename where us
200. ar expression pattern matches any part of a variable name then that variable is selected This capability is also called wildcarding and is very useful for sub setting large data files Extended regular expressions are defined by the POSIX grep E aka egrep command As of NCO 2 8 1 August 2003 variable name arguments to the v switch may contain extended regular expressions As of NCO 3 9 6 January 2009 variable names arguments to ncatted may contain extended regular expressions As of NCO 4 2 4 November 2012 group name arguments to the g switch may contain extended regular expressions Because of its wide availability NCO uses the POSIX regular expression library regex Regular expressions of arbitary complexity may be used Since netCDF variable names are relatively simple constructs only a few varieties of variable wildcards are likely to be useful For convenience we define the most useful pattern matching operators here t Matches the beginning of a string Matches the end of a string Matches any single character The most useful repetition and combination operators are q The preceding regular expression is optional and matched at most once Ld The preceding regular expression will be matched zero or more times up The preceding regular expression will be matched one or more times The preceding regular expression will be joined to the following regular ex pression The resulting regular expres
201. ar id md1 nc rm drc_out fld_out var_id var var_id _ historical echo Model md1 done done Store models as groups in the output file ncecat 0 gag drc_out fld_out var_id var var_id _ nc drc_out fld_out var_id var var_id _ rlm _all md1_ xpt 0 _all nsm_annual nc echo Var var var_id done done gsl rgr nco Linear Regression Called by ann avg sh Caution make sure the variable name is in agreement with the main script now is snd Online http nco sourceforge net nco htmlitAnnual Average over Regions Declare variables cO rgn 0 Intercept c1 rgn 0 Slope sdv rgn 0 Standard deviation covxy rgn 0 Covariance x double time for rgn_id 0 rgn_id lt rgn size rgn_id Loop over regions 1 gsl fit linear time 1 snd rgn id 1 time size amp tcO amp tci amp cov00 amp covOi amp covii amp sumsq Linear regression function cO rgn id tc0 Output results ci rgn id tcl covxy rgn id gsl stats covariance time 1 N time size double snd rgn id 1 time size Covariance function sdv rgn id gsl stats sd snd rgn id 1 time size Standard deviation function Chapte P r r r 7 CMIP5 Example 261 A en ag a a eine a te tne time sdv gsl stats sd time 1 time size r value covxy time_sdv sdv t value r_value sqrt 1 r_value 2
202. ar interp wrap bsa sw out latn lonn lat lon wsa sw less bilinear interp wrap wsa sw out latn lonn lat lon Add attributions latnOunits degree north lonn units degree east latnOlong name latitude lonnOlong name longitude bsa sw lessOhdf name Albedo_BSA_shortwave bsa sw lessOcalibrated nt 5 bsa sw lessOmissing value 32767 0 bsa sw lessOunits albedo no units bsa sw lessOlong name Global Albedo B8A shortwave wsa sw lessOhdf name Albedo WSA shortwave wsa sw lessOcalibrated nt 5 wsa sw lessOmissing value 32767 0 wsa sw lessOunits albedo no units wsa sw less long name Global Albedo WSA shortwave Add Coordinates to MODIS Data Script bin bash Example for in MODIS data We have to add it manually now Input files modis mcd43c3 cesm grid MCD43C3 2000049 regrid nc regrid using bi_interp nco the spatial resolution of MODIS data is much finer than those of CMIP5 models In order to compare the two we can regrid MODIS data to comform to CMIP5 add coordinates using coor nco there is no coordinate information 270 NCO 4 5 3 alpha03 User Guide Output files modis mcd43c3 cesm grid MCD43C3 2000049 regrid nc Online http nco sourceforge net nco html Add Coordinates to MODIS Data Execute this script bash add_crd sh s var MOD10CM 4 Variable fld in snc nc Folder of input files dr
203. are contiguous at the end of the array Use same srt map to hyperslab multiple variables of the same shape Remember to apply srt map to the coordinate variables crd in remap crd in srt map var in remap var in srt map Hyperslab last msk nbr values of variable s of interest crd out crd out crd in dmn in sz msk nbr dmn in sz 1 var out crd out var in dmn in sz msk nbr dmn in sz 1 NB Even though we created all variables possible as RAM variables the original coordinate of interest time is written to the ouput I m not exactly sure why For now delete it from the output with ncks 0 x v time foo nc foo nc EOF ncap2 0 v S ncap2_foo nco nco data in nc foo nc ncks 0 x v time foo nc foo nc 156 NCO 4 5 3 alpha03 User Guide ncks foo nc Here is an extended example of how to use ncap2 features to sort multi dimensional arrays based on the coordinate values along a single dimension cat gt ncap2 foo nco lt lt EOF di Purpose Sort multi dimensional array based on coordinate values This example sorts the variable three dmn rec var time lat lon based on the values of the time coordinate Included in NCO User Guide at http nco sf net nco html sort Randomize the time coordinate time 10 0 gsl rng uniform time print original randomized time n print time The sort routine is overloaded and takes one or
204. are free of polar singularities and thus are well suited to storage on Uniform grids Theoretically a Uniform latitude grid could have non uniform longitudes but NCO currently does not implement non uniform longitude grids Their mathematical properties convergence and excessive resolution at the poles which can appear as singularities make Uniform grids fraught for use in global models One purpose Uniform grids serve in modeling is as offset or staggered grids meaning grids whose centers are the interfaces of another grid The Finite Volume FV method is often used to represent and solve the equations of motion in climate related fields Many FV solutions includ ing the popular Lin Rood method as used in the CESM CAM FV atmospheric model evaluate scalar i e non vector fields e g temperature water vapor at gridcell centers of what is therefore called the scalar grid FV methods like Lin Rood that employ an Arakawa C grid or D grid formulation define veloc ities on the edges of the scalar grid This CAM FV velocity grid is therefore staggered or offset from the CAM FV scalar grid by one half gridcell The CAM FV scalar latitude grid has gridpoints the caps centered on each pole to avoid singularities The offset of a Cap grid is a Uniform grid so the Uni form grid is often called an FV offset or staggered grid Hence an NCO Uniform grid is equivalent to an NCL Fixed Offset grid For examp
205. ariable from the attribute name If the attribute is uniquely named no other variables contain the attribute then the old var nameQ old att name syntax is redundant The old var name variable names global and group have special significance They indicate that old_att_nm should only be renamed where it occurs as a global i e root group metadata attribute for global or for group as any group attribute and not where it occurs as a variable attribute The var nameQatt name syntax is accepted though not required for the new name d old name new name Dimension renaming The old and new names of the dimension are speci fied with d or dmn dimension by the associated old name and new name values This option may be specified more than once g old name new name Group renaming The old and new names of the group are specified with g or grp group by the associated old name and new name values This option may be specified more than once This functionality is only available in NCO version 4 3 7 October 2013 or later and only when built on netCDF library version 4 3 1 rc1 August 2013 or later v old name new name Variable renaming The old and new names of the variable are specified with y or variable by the associated old name and new name values This option may be specified more than once EXAMPLES 232 NCO 4 5 3 alpha03 User Guide Rename the
206. ariables ncks 0 v wsa sw less bsa sw less drc_in fld_out var sfx regrid nc drc_in fld_out var sfx regrid nc Rename the new variables dimensions and attributions ncrename 0 d latn lat d lonn lon v latn lat v lonn lon v wsa sw less wsa sw v bsa sw less bsa sw a missing value FillValue drc_in fld_out var sfx regrid nc echo sfx done done bi_interp nco Bilinear interpolation Included by rgr sh Online http nco sourceforge net nco html Regrid MODIS Data defdim latn 192 Define new dimension latitude 268 NCO 4 5 3 alpha03 User Guide defdim lonn 288 Define new dimension longitude latn 1atn 90 89 0576 88 1152 87 1728 86 2304 85 288 84 3456 83 4031 82 4607 81 5183 80 5759 79 6335 78 6911 77 7487 76 8063 75 8639 74 9215 73 9791 73 0367 72 0942 71 1518 70 2094 69 267 68 3246 67 3822 66 4398 65 4974 64 555 63 6126 62 6702 61 7277 60 7853 59 8429 58 9005 57 9581 57 0157 56 0733 55 1309 54 1885 53 2461 52 3037 51 3613 50 4188 49 4764 48 534 47 5916 46 6492 45 7068 44 7644 43 822 42 8796 41 9372 40 9948 40 0524 39 11 38 1675 37 2251 36 2827 35 3403 34 3979 33 4555 32 5131 31 5707 30 6283 29 6859 28 7435 27 8011 26 8586 25 9162 24 9738 24 0314 23 089 22 1466 21 2042 20 2618 19 3194 18 377 17 4346 16 4921 15 5497 14 6073 13 6649 12 7225 11 7801 10 8377 9 89529 8 95288 8 01047 7 06806 6 12
207. ariables designated as coordinates bounds or ancillary variables so that users subsetting a certain variable automatically obtain all related variables Special handling and prevention of meaningless operations e g the root mean square of latitude so that coordinates and bounds preserve meaningful information even as normal fields are statistically transformed Understanding units and certain calendars so that hyperslabs may be specified in physical units and so that user needs not manually decode per file time specifications Understanding auxiliary coordinates so that irregular hyperslabs may be specified on complex geometric grids Checking for CF compliance on netCDF3 and netCDF4 and HDF files Converting netCDF4 and HDF files to netCDF3 for strict CF compliance Finally a main use of NCO is to produce CF i e to improve CF compliance by annotating metadata renaming objects attributes variables and dimensions permut ing and inverting dimensions recomputing values and data compression Chapter 3 Shared Features 125 Currently NCO determines whether a datafile is a CF output datafile simply by checking case insensitively whether the value of the global attribute Conventions if any equals CF 1 0 or NCAR CSM Should Conventions equal either of these in the first input file NCO will apply special rules to certain variables because of their usual meaning in CF files NCO will not average the following variables ofte
208. ariance ncra v uprmvprm in nc foo nc Time mean covariance The mathematically inclined will note that the same covariance would be obtained by replacing the step involving ncap2 with ncap2 0 s uprmvprm u uavg v vavg foo nc foo nc Covariance As of NCO version 3 1 8 December 2006 ncap2 can compute averages and thus co variances by itself ncap2 s uavg u avg time vavg v avg time uprmvprm u v uavg vavg s uprmvrpmavg uprmvprm avg time in nc foo nc We have not seen a simpler method to script and execute powerful arithmetic than ncap2 ncap2 utilizes many meta characters e g O L1 that can confuse the command line shell if not quoted properly The issues are the same as those which arise in utilizing extended regular expressions to subset variables see Section 3 11 Subsetting Files page 43 The example above will fail with no quotes and with double quotes This is because shell globbing tries to interpolate the value of time from the shell environment unless it is quoted ncap2 s uavg u avg time in nc foo nc Correct recommended ncap2 s uavg u avg time in nc foo nc Correct and dangerous 178 NCO 4 5 3 alpha03 User Guide ncap2 s uavg u avg time in nc foo nc Wrong time ncap2 s uavg u avg time in nc foo nc Wrong time Without the single quotes the shell replaces time with an empty string The command ncap2 receives f
209. aris 2 x Cray UNICOS 8 x 10 x and Microsoft Windows 95 98 NT 2000 XP Vista 7 8 10 If you port the code to a new operating system please send me a note and any patches you required The major prerequisite for installing NCO on a particular platform is the successful prior installation of the netCDF library and as of 2003 the UDUnits library Unidata has shown a commitment to maintaining netCDF and UDUnits on all popular UNIX plat forms and is moving towards full support for the Microsoft Windows operating system OS Given this the only difficulty in implementing NCO on a particular platform is standard ization of various C language API system calls NCO code is tested for ANSI compliance by compiling with C99 compilers including those from GNU gcc std c99 pedantic D_BSD_SOURCE D_POSIX_SOURCE Wall Comeau Computing como c99 Cray cc HP Compaq DEC cc IBM xlc c qlanglvl extc99 Intel icc std c99 LLVM clang NEC cc PathScale QLogic pathcc std c99 PGI pgcc c9x SGI cc c99 and Sun cc NCO all commands and the libnco library and the C interface to netCDF called libnco_c comply with the ISO C stan dards as implemented by Comeau Computing como Cray CC GNU g Wall HP Compaq DEC cxx IBM x1C Intel icc Microsoft MVS NEC c Path Scale Qlogic p
210. arithmetic Matlab TREFHT 289 239964 Double precision weight gw double precision arithmetic Matlab TREFHT 289 239964 Single precision weight area double precision arithmetic NCL ncl lt ncwa_3528514 ncl TREFHT 289 239960 TREFHT_dbl 289 239964 Double precision weight gw double precision arithmetic NCL TREFHT 289 239960 TREFHT_dbl 289 239964 dk HH dt dbi dt db o Gt All languages tested C Matlab NCL and NCO agree to machine precision with double precision arithmetic Users are fortunate to have a variety of high quality software that liberates them from the drudgery of coding their own Many packages are free as in beer As shown above NCO permits one to shift to their float promotion preferences as desired No other language allows this with a simple switch Chapter 3 Shared Features 121 To summarize until version 4 3 6 September 2013 the default arithmetic convention of NCO adhered to Fortran behavior and automatically promoted single precision to double precision in all mixed precision expressions and left alone pure single precision expressions This is faster and more memory efficient than other conventions However pure single precision arithmetic can lose too much precision when used to condense e g average large arrays Statistics involving about n 10 000 single precision inputs will lose about 2 3 digits if not promoted to double precision prior to arithmetic
211. array data In general the first element is rounded down the second up and so on This results in a mean bias quite close to zero The only exception is that the floating point value of zero is never quantized upwards For simplicity NSD always rounds scalars downwards Although NSD or DSD are different algorithms under the hood they both replace the unwanted least siginificant bits of the IEEE significand with a string of consecutive zeroes Byte stream compression techniques such as the gzip DEFLATE algorithm compression available in HDF5 always compress zero strings more efficiently than random digits The net result is netCDF files that utilize compression can be significantly reduced in size This feature only works when the data are compressed either internally by netCDF or externally by another user supplied mechanism It is most straightfoward to compress data internally using the built in compression and decompression supported by netCDF4 For convenience NCO automatically activates file wide Lempel Ziv deflation see Section 3 30 Deflation page 101 level one i e L 1 when PPC is invoked on any variable in a netCDF4 output file This makes PPC easier to use effectively since the user need not explicitly specify deflation Any explicitly specified deflation including no deflation L 0 will override the PPC deflation default If the output file is a netCDF3 format NCO will emit a message suggesting internal netCD
212. as dimensioned lat lon it will have dimensions record lat lon in the output file A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file Normally this means all the input files are the same size and contain data on different realizations of the same variables See Section 4 5 ncecat netCDF Ensemble Concatenator page 197 for a complete description of ncecat ncpdq makes it possible to concatenate files along any dimension not just the record dimension First use ncpdq to convert the dimension to be concatenated i e extended 7 Currently nces and ncrcat are symbolically linked to the ncra executable which behaves slightly differ ently based on its invocation name i e argv 0 These three operators share the same source code and merely have different inner loops o The third averaging operator ncwa is the most sophisticated averager in NCO However ncwa is in a different class than ncra and nces because it operates on a single file per invocation as opposed to multiple files On that single file however ncwa provides a richer set of averaging options including weighting masking and broadcasting Chapter 2 Operator Strategies 21 with data from other files into the record dimension Second use ncrcat to concatenate these files Finally if desirable use ncpdq to revert to the original dimensionality As a concrete example say that files x O1 nc x 02
213. ata are read If add offset is present for a variable then the add offset value is added to the data after the data are read If both scale factor and add offset attributes are present the data are first scaled by scale factor before the offset add offset is added upk scale factor x pck add offset pek x max min min max ndrv 2 When scale factor and add offset are used for packing the associated variable con taining the packed data is typically of type byte or short whereas the unpacked values are intended to be of type int float or double An attribute s scale factor and add offset and _FillValue if any should all be of the type intended for the unpacked data i e int float or double Default Handling of Packed Data Most files originally written in HDF format use the HDF packing unpacking algorithm This algorithm is incompatible with the netCDF packing algorithm described above The 29 Although not a part of the standard NCO enforces the policy that the _FillValue attribute if any of a packed variable is also stored at the original precision Chapter 3 Shared Features 107 unpacking component of the HDF algorithm described here http www hdfgroup org HDF5 doc UG UG frameiODatasets html is upk scale factor x pck add offset Confusingly the incompatible netCDF and HDF algorithms both store their parameters in attributes with the same names scale factor and add offset Dat
214. ata sok uh OL ab V Eu qu 101 dimension dim nin max stride subcycle 62 dimension dim min max stride 60 dimension dim min max stride Ab obidadl IS R ELMS NL ilis 58 64 66 79 diskless all 0m E uA 24 104 dmn dim min max stride subcycle apiece abe AA Blow EEN AO 62 dmn dim min max stride 60 dmn dim min max stridelll 58 64 66 79 ensemble file eseess 194 ensemble_group cece eee eee 194 ensemble_suffix 0 cece cece ee 194 sze xcludeis sessi gemuche d xenie ary ces 43 213 file format inl sg Ves oe I vw 38 fil list 4 LaL EA D EY e 123 fix rec dmn all see 42 fix rec dun dim oves VI da 205 SSE MDI Boiss iid cceeraex Ww e kun e i V 205 eI TMT de E iai 38 f Tst iMr 2211 hae bee uh s 123 f l out fl out hod a AES oe LERRA 32 ELIAS orcs das 130 SEI A Son oe ents Pe es 115 SENG TDL amp ec 3 aes woe ewe aes eeu DS 180 SOfOrbtran scuicc loue Gut a MESA 57 SE Pac seag a ahead eae a eels 122 ad 22 ee ee E E eee ere eee eee ee 198 249 FHP LD tas towe lt sigws noi ePSHREQNUENE Mae ates 122 glb att nm att val sess 122 286 e att addins cnr wats mateo 122 gp ppe ASC hun haat berto n s i Pise 48 SSSTOUP LTD acti sch esce ta paths hades 43 CCOBEPORID iid halt names tise sais anaes tas 43 Hagrp xtr var xcl sos siii tha
215. ata time stm for time and time offsets If you do not work with ARM data then you may skip this section ARM data files store time information in two variables a scalar base time and a record variable time offset Subtle but serious problems can arise when these type of files are blindly concatenated without CF or ARM support NCO implements rebasing see Section 3 25 Rebasing Time Coordinate page 82 as necessary on both CF and ARM files Rebasing chains together consecutive input files and produces an output file which contains the correct time information For ARM files this is expecially complex because the time coordinates are often stored as type NC CHAR Currently ncrcat determines whether a datafile is an ARM datafile simply by testing for the existence of the variables base time time offset and the dimension time If these are found in the input file then ncrcat will automatically perform two non standard but hopefully useful procedures First ncrcat will ensure that values of time offset appearing in the output file are relative to the base time appearing in the first input file and presumably though not necessarily also appearing in the output file Second if a coordinate variable named time is not found in the input files then ncrcat automatically creates the time coordinate in the output file The values of time are defined by the ARM conventions time base_time time offset Thus if output file contains the time offset
216. ated by regular intervals see Section 3 18 Subcycle page 62 The option mro switches ncra to produce a Multi Record Output instead of a single record output For example assume snd is a 3D array with dimensions time latitude longitude and time includes every month from Jan 1990 to Dec 2005 192 months as total which are 16 years Let s look at the following two command lines ncra mro v snd d time 1990 12 01 12 3 in nc out mro nc ncra v snd d time 1990 12 01 12 3 in nc out sro nc In the first output file out mro nc snd is still a 3D array with dimensions time latitude longitude but the length of time now is 16 meaning 16 winters In the second output file out sro nc the length of time is only 1 It is now the average of all the 16 winters when using d dim min max to specify the hyperslabs you can leave it blank if you want to include the minimum or the maximum of the data like we did above 6 3 One time point one file This means if you have daily data of 30 days there will be 30 data files Or if you have monthly data of 12 months there will be 12 data files Dealing with this kind of files you need to specify the file names in shell scripts and pass them to NCO operators For example your daily data files may look like snd 19900101 nc snd_19900102 nc snd_19900103 nc If you want to know the monthly average of Jan 1990 you can write like ncra 0 snd_199001 nc out nc You might
217. athCC PGI pgCC SGI CC LANG std and Sun CC LANG std See nco bld Makefile and nco src nco_c Makefile old for more details and exact settings 2 The BSD SOURCE token is required on some Linux platforms where gcc dislikes the network header files like netinet in h Chapter 1 Introduction 9 Until recently and not even yet ANSI compliant has meant compliance with the 1989 ISO C standard usually called C89 with minor revisions made in 1994 and 1995 C89 lacks variable size arrays restricted pointers some useful printf formats and many mathemat ical special functions These are valuable features of C99 the 1999 ISO C standard NCO is C99 compliant where possible and C89 compliant where necessary Certain branches in the code are required to satisfy the native SGI and SunOS C compilers which are strictly ANSI C89 compliant and cannot benefit from C99 features However C99 features are fully supported by modern AIX GNU Intel NEC Solaris and UNICOS compilers NCO requires a C99 compliant compiler as of NCO version 2 9 8 released in August 2004 The most time intensive portion of NCO execution is spent in arithmetic operations e g multiplication averaging subtraction These operations were performed in Fortran by default until August 1999 This was a design decision based on the relative speed of Fortran based object code vs C based object code in late 1994 C compiler vectorization ca
218. ation 00 cece cece 27 3 3 OpenMP Threading 2 00 cece eee eens 27 3 4 Command Line Options 00 e eee eee eee 29 3 5 Specifying Input Files 0 ccc eee ene eee 30 3 6 Specifying Output Files 0 0 eens 32 3 7 Accessing Remote Files nana c cece cece eee eee ees 33 36 OREN DAP o 35 3 8 Retaining Retrieved Files 0 00 cece cee 37 NCO 4 5 3 alpha03 User Guide 3 9 File Formats and Conversion 00 cece eee e ees 38 3 9 1 File Formats c ise fecunoR iaa 38 3 9 2 Determining File Format 0 0 cee eee eee ee eee 39 3 9 8 File Conversion seseesseee ee 41 39 47 A utOCOTVerSlODL ses eerte ela 41 3 10 Large File Support 0 cece cece e 42 ILE Subsetting Piles et Ga decile abe Ri UE RD EROR I d peace 43 3 12 Subsetting Coordinate Variables 0 0 o ooooommomo 4T 3 13 Group Path Editing secsec en os eta pee a e i 48 3 13 1 Deletion Truncation and Flattening of Groups 49 3 13 2 Moving Groups secs sesoses cece cee eee mm 50 3 13 3 Dismembering Files 0000 51 3 13 4 Checking CF compliance 0 000 cece eee ee 54 3 14 C and Fortran Index conventions 0 00 e eee eee 57 3 15 HyperslabS i cde seis beh eee en aute Lea eee 58 9 16 Stride es caso een teecot ellen debeo eet ome eset 60 3 17 Record Appending 6 0 cece cece eee ees 61 9 1
219. ation and Inheritance e Attribute propagation occurs in a regular assign statement The variable being defined on the LHS gets copies of the attributes from the leftermost variable on the RHS e Attribute Inheritance The LHS variable inherits attributes from an Input variable with the same name e It is possible to have a regular assign statement for which both propagation and inher itance occur prs_mdp inherits attributes from PO prs mdp time lat lon lev PO hyam hybm PS th min inherits attributes from three dmn var dbl th min 1 0 2xthree dmn var dbl min time If the attribute name contains non regular characters use ID quoting See see Section 4 1 27 ID Quoting page 181 p miQc 10ost 23 4 1 7 Number literals The table below lists the postfix character s to add to a number literal aka a naked constant for explicit type specification The same type specification rules are used for Chapter 4 Reference Manual 141 variables and attributes A floating point number without a postfix defaults to NC DOUBLE while an integer without a postfix defaults to type NC INT var rlev 0 1 Variable will be type code NC_DOUBLE var lon_grd 2 0 Variable will be type code NC_DOUBLE var gds_crd 2e3 Variable will be type code NC_DOUBLE var gds_crd 2 0f Variable will be type code NC_FLOAT note f var gds_crd 2e3f Variable will be type code NC_FLOAT note f var gd
220. ation directory Do this with the NETCDF4_ROOT variable Then use your preferred NCO build mechanism e g export NETCDF4_ROOT usr local netcdf4 Set netCDF4 location cd nco configure enable netcdf4 Configure mechanism or cd nco bld make NETCDF4 Y allinone Old Makefile mechanism We carefully track the netCDF4 releases and keep the netCDF4 atomic type support and other features working Our long term goal is to utilize more of the extensive new netCDF4 feature set The next major netCDF4 feature we are likely to utilize is parallel I O We will enable this in the MPI netCDF operators 1 7 Help Requests and Bug Reports We generally receive three categories of mail from users help requests bug reports and feature requests Notes saying the equivalent of Hey NCO continues to work great and it saves me more time everyday than it took to write this note are a distant fourth There is a different protocol for each type of request The preferred etiquette for all communications is via NCO Project Forums Do not contact project members via personal e mail unless your request comes with money or you have damaging information about our personal lives Please use the Forums they preserve a record of the questions and answers so that others can learn from our exchange Also since NCO is government funded this record helps us provide program officers with information they need to evaluate our project Before posting to the NC
221. ations and presentations is at http nco sf netitpub This list links to the full papers and seminars themselves 5 3 Proposals for Institutional Funding NSF has funded a project http nco sf net prp_sei to improve Distributed Data Reduction amp Analysis DDRA by evolving NCO into a suite of Scientific Data Operators called SDO The two main components of this project are NCO parallelism OpenMP MPI and Server Side DDRA SSDDRA implemented through extensions to OPeNDAP and netCDF4 This project will dramatically reduce bandwidth usage for NCO DDRA With this first NCO proposal funded the content of the next NCO proposal is clear We are interested in obtaining NASA support for HDF specific enhancements to NCO We plan to submit a proposal to the next suitable NASA NRA or NSF opportunity We are considering other interesting ideas for still more proposals Please contact us if you wish to be involved with any future NCO related proposals Comments on the proposals and letters of support are also very welcome Chapter 6 Quick Start 245 6 Quick Start Simple examples in Bash shell scripts showing how to average data with different file struc tures Here we include monthly seasonal and annual average with daily or monthly data in either one file or multiple files 6 1 Daily data in one file Suppose we have daily data from Jan 1st 1990 to Dec 31 2005 in the file of in nc with the record dimension as time Monthly average
222. b ukmo hadcm3 runi 200101 209911 nc ncks A v tas sresa b ukmo hadcm3 run tas 200101 209911 nc sresaib ukmo hadcm3 runi 200101 209911 nc ncks A v pr Ssresai b ukmo hadcm3 runi pr 200101 209911 nc sresaib ukmo hadcm3 runi 200101 209911 nc If already have file then mv sresaib ukmo hadcm3 runi 200101 209911 nc foo nc bin cp area msk ukmo hadcm3 nc sresaib ukmo hadcm3 runi 200101 209911 nc Chapter 9 CCSM Example 281 ncks v tas pr foo nc sresalb ukmo hadcm3 runi 200101 209911 nc Level II Correct years months Template scn _ md1 _ run _ var _ yyyymm _ yyyymm nc ncks d time filei nc file2 nc ncrcat file2 nc file3 nc sresaib ukmo hadcm3 runi 200001 209912 nc Level III Many derived products from level II e g A Global mean timeseries ncwa w area a lat lon sresaib ukmo hadcm3 runi 200001 209912 nc sresaib ukmo hadcm3 runi 200001 209912 xy nc B Califoria average timeseries ncwa m msk rgn w area a lat lon sresaib ukmo hadcm3 runi 200001 209912 nc sresaib ukmo hadcm3 runi 200001 209912 xy Cal nc Chapter 10 References 283 10 References e ZeM07 Zender C S and H J Mangalam 2007 Scaling Properties of Common Statistical Operators for Gridded Datasets Int J High Perform Comput Appl 21 4 485 498 doi 10 1177 1094342007083802 e Zen08 Zender C S 2008 Analysis of Self describing Gridded Geoscience Data wi
223. ble in the input file They also unpack and then re pack every packed variable The third command only packs unpacked variables in the input file If a variable is already packed the third command copies it unchanged to the output file T he fourth and fifth commands unpack any packed variables If a variable is not packed the third command copies it unchanged The previous examples all utilized the default packing map Suppose you wish to archive all data that are currently unpacked into a form which only preserves 256 distinct values Then you could specify the packing map pck_ map as hgh_byt and the packing policy pck plc as all_xst ncpdq P all xst M hgh byt in nc out nc 222 NCO 4 5 3 alpha03 User Guide Many different packing maps may be used to construct a given file by performing the packing on subsets of variables e g with v and using the append feature with A see Section 2 4 Appending Variables page 19 Users may wish to unpack data packed with the HDF convention and then re pack it with the netCDF convention so that all their datasets use the same packing convention prior to intercomparison One step procedure For NCO 4 4 0 netCDF 4 3 1 1 Convert unpack and repack HDF file into netCDF file ncpdq hdf upk P xst new modis hdf modis nc HDF4 files ncpdq hdf upk P xst new modis h5 modis nc HDF5 files One step procedure For NCO 4 3 7 4 3 9 1 Convert unpack and repac
224. bodies the necessary algorithms to transition geoscience data analysis from relying solely on traditional or flat datasets to allowing newer hierarchical or nested datasets Hierarchical datasets support and enable combining all datastreams that meet user specified criteria into a single or small number of files that hold all the science relevant data NCO and no other software to our knowledge exploits this capability now Data and metadata may be aggregated into and analyzed in hierarchical structures We call the resulting data storage distribution and analysis paradigm Group Oriented Data Analysis and Distribution GODAD GODAD lets the scientific question organize the data not the ad hoc granularity of all relevant datasets This chapter illustrates GODAD techniques applied to analysis of the CMIP5 dataset To begin we document below a prototypical example of CMIP5 analysis and evaluation using traditional NCO commands on netCDF3 format model and HDF EOS format observa tional NASA MODIS satellite instrument datasets These examples complement the NCO User Guide by detailing in depth data analysis in a frequently encountered real world context Graphical representations of the results NCL scripts available upon request are provided to illustrate physical meaning of the analysis Since NCO can process hierarchical datasets i e datasets stored with netCDF4 groups we present sample scripts illustrating group base
225. bs in all dimensions so long as the hyperslabs resolve to the same size The fixed i e non record dimensions should be the same size in all ensemble members both before and after hyperslabbing although the hyperslabs may and usually do change the size of the dimensions from the input to the output files Prior to this nces was only guaranteed to work on hyperslabs in the record dimension that resolved to the same size Chapter 4 Reference Manual 195 newer feature use nsm_grp or ensemble group Members of a group ensemble are groups that share the same structure parent group and nesting level Members must be leaf groups i e not contain any sub groups Their contents usually have different values because they are realizations of replicated experiments In group ensemble mode nces computes the statistics across the ensemble which may span multiple input files Files may contain members of multiple distinct ensembles However all ensembles must have at least one member in the first input file Group ensembles behave as an unlimited dimension of datasets they may contain an arbitrary and extensible number of realizations in each file and may be composed from multiple files Output statistics in group ensemble mode are stored in the parent group by default If the ensemble members are cesm cesm_01 and cesm cesm 02 then the computed statistic will be in cesm in the output file The nsm_sfx option instructs nces to i
226. butes from the input file to output file if output file is a pre existing netCDF file whose relevant dimensions conform to dimension sizes of input file The append features of ncks are intended to provide a rudimentary means of adding data from one netCDF file to another conforming netCDF file If naming conflicts exist between the two files data in output file is usually overwritten by the corresponding data from input file Thus when appending the user should backup output file in case valuable data are inadvertantly overwritten If output file exists the user will be queried whether to overwrite append or exit the ncks call completely Choosing overwrite destroys the existing output file and create an entirely new one from the output of the ncks call Append has differing effects depending on the uniqueness of the variables and attributes output by ncks If a variable or attribute extracted from input file does not have a name conflict with the members of output file then it will be added to output file without overwriting any of the existing contents of output file In this case the relevant dimensions must agree conform between the two files new dimensions are created in output file as required When a name conflict occurs a global attribute from input file will overwrite the corresponding global attribute from output file If the name conflict occurs for a non record variable then the dimensions and type of the variable and of its coo
227. c 198612 nc 198712 nc 3 6 Specifying Output Files Availability All operators Short options o Long options fl_out output NCO commands produce no more than one output file fl out Traditionally users spec ify fl out as the final argument to the operator following all input file names This is the positional argument method of specifying input and ouput file names The positional ar Chapter 3 Shared Features 33 gument method works well in most applications NCO also supports specifying fl out using the command line switch argument method o f1 out Specifying fl out with a switch rather than as a positional argument allows fl out to precede input files in the argument list This is particularly useful with multi file operators for three reasons Multi file operators may be invoked with hundreds or more filenames Visual or automatic location of fl out in such a list is difficult when the only syntactic distinction between input and output files is their position Second specification of a long list of input files may be difficult see Section 2 7 Large Numbers of Files page 21 Making the input file list the final argument to an operator facilitates using xargs for this purpose Some alternatives to xargs are heinous and undesirable Finally many users are more comfortable specifying output files with o f1 out near the beginning of an argument list Compilers and linkers are usually in
228. c 85 0112 nc 86m85_0112 nc ncdiff 86 0112 nc 85 0112 nc 86m85_0112 nc ncbo op typ sub 86 0112 nc 85 0112 nc 86m85_0112 nc ncbo op typ 86 0112 nc 85 0112 nc 86m85_0112 nc These commands are all different ways of expressing the same thing The following examples demonstrate the broadcasting feature of ncbo Say we wish to compute the monthly anomalies of T from the yearly average of T for the year 1985 First we create the 1985 average from the monthly data which is stored with the record dimension time ncra 85 0112 nc 85 nc ncwa 0 a time 85 nc 85 nc The second command ncwa gets rid of the time dimension of size 1 that ncra left in 85 nc Now none of the variables in 85 nc has a time dimension A quicker way to accomplish this is to use ncwa from the beginning ncwa a time 85 0112 nc 85 nc We are now ready to use ncbo to compute the anomalies for 1985 ncdiff v T 85 0112 nc 85 nc t anm 85 0112 nc Each of the 12 records in t anm 85 0112 nc now contains the monthly deviation of T from the annual mean of T for each gridpoint Say we wish to compute the monthly gridpoint anomalies from the zonal annual mean A zonal mean is a quantity that has been averaged over the longitudinal or x direction First we use ncwa to average over longitudinal direction lon creating 85 x nc the zonal mean of 85 nc Then we use ncbo to subtract the zonal annual means from the monthly gridpoint data ncwa a lon 85 nc 85 x nc ncdiff 85 0
229. c in media grele data wenshan modis directory of input files for fn in 1s drc_in fld_in var nc do Loop over files sfx echo fn cut d f 8 cut d f 2 4 Part of file names echo sfx Rename dimension names ncrename d YDim_MOD_CMG_Snow_5km lat d XDim_MOD_CMG_Snow_5km lon 0 drc_in fld_in var sfx nce drc_in fld_in var sfx nc Add coordinates ncap2 0 S crd nco drc_in fld_in var sfx nc A drc_in fld_in var sfx nc done crd nco Add coordinates to MODIS HDF data Included by add_crd sh Online http nco sourceforge net nco html Add Coordinates to MODIS Data lon array 0 f 0 05 lon 180 lat 90 f array 0 f 0 05 lat 7 7 Permute MODIS Coordinates MODIS orders latitude data from 90 N to 90 N and longitude from 180 E to 180 E However CMIP5 orders latitude from 90 N to 90 N and longitude from 0 E to 360 E This script changes the MODIS coordinates to follow the CMIP5 convention bin bash Se SSS SSS SSS SS SS SS SSS SSS SSS SSS SSS SS SST SSS SSS SSSH Example for permute coordinates the grid of MODIS is it from 180 degE 90 degN the left up corner to 180 degE 90 degN the right low corner However CMIP5 is from 0 degE 90 degN to 360 degE 90 degN The script here changes the MODIS grid to CMIP5 grid Chapter 7 CMIP5 Example 271 H Input files modis mcd43c3
230. cWctee is O EARE or 243 provenance ssseeeeeeeeeee 22 23 123 198 pseudonym icieeig l9 eR edie ett BRE 10 publications asbendweert amp c e wx RT HIR E La eid 7 NCO 4 5 3 alpha03 User Guide Q OLOogle rees wise E id doen tae es 8 Qloster isti E bi bs 10 quadruple precision 0 cece eee eee 180 quantization ie eee bees pb PUT bees 91 92 Quench ii22ele izle pepe heey needed hia bese 209 Quick Startins ne rerea aede n De ROTE oe 245 QUOLES c eee aE aa 47 177 190 223 R A E A 86 RAG trizas Ey 197 ram delete uetinasia ir te pd 149 ram WIL sre bx E Lex bb 149 RAM Les ou e AQUI e 24 104 RAM disks cuca iii 18 104 RAMCRI6S zii aire atan dede Er ada Res 18 104 RAM variables om coges e dope 104 148 random walk dis enti terre eR xd 119 Tank see See once RES 132 190 191 192 235 OO roe ned he edet adele DERS 10 33 hio EE 127 di zcsiseccserc amp Patna Re REY UN drip 88 Te diMENSION sy valoda eder cu rane 217 re order dimensions eeeeee eens 217 record aggregation 0 ee ee eee 197 record append eee ui erene eee 61 record average eee eee eee eee 225 record concatenation eee eee eee 228 record dimension 19 58 91 194 197 199 205 206 220 224 225 228 record variable e essere eee 57 220 rectangular grids 00 0200 e ee 157 PECUTSION oie ac pve sind cutie ac lee eae Rete LAT 44 TECUTS
231. ce the command line is stored as metadata in the history attribute see Section 3 39 History Attribute page 122 However the n loop switch is useless when 22 NCO 4 5 3 alpha03 User Guide there is no simple arithmetic pattern to the input filenames e g h00001 nc h00002 nc h90210 nc Moreover filename globbing does not work when the input files are too numerous or their names are too lengthy when strung together as a single argument to be passed by the calling shell to the NCO operator When this occurs the ANSI C standard argc argv method of passing arguments from the calling shell to a C program i e an NCO operator breaks down There are at least three alternative methods of specifying the input filenames to NCO in environment limited situations The recommended method for sending very large numbers hundreds or more typically of input filenames to the multi file operators is to pass the filenames with the UNIX standard input feature aka stdin Pipe large numbers of filenames to stdin bin ls grep CASEID _ nc ncecat o foo nc This method avoids all constraints on command line size imposed by the operating system A drawback to this method is that the history attribute see Section 3 39 History Attribute page 122 does not record the name of any input files since the names were not passed on the command line This makes determining the data provenance at a later date difficult To remedy
232. ception of ncap2 2 9 1 Single and Multi file Operators The multi file operators currently comprise the record operators ncra and ncrcat and the ensemble operators nces and ncecat The record operators require much less memory than the ensemble operators This is because the record operators operate on one single record i e time slice at a time whereas the ensemble operators retrieve the entire variable into memory Let MS be the peak sustained memory demand of an operator FT be the memory required to store the entire contents of all the variables to be processed in an input file FR be the memory required to store the entire contents of a single record of each of the variables to be processed in an input file V R be the memory required to store a single record of the largest record variable to be processed in an input file VT be the memory required to store the largest variable to be processed in an input file VI be the memory required to store the largest variable which is not processed but is copied from the initial file to the output file All operators require MI VI during the initial copying of variables from the first input file to the output file This is the initial and transient memory demand The sustained memory demand is that memory required by the operators during the processing i e averaging concatenation phase which lasts until all the input files have been processed The operators have the following memory requirem
233. ch contains 60 consecutive months of data Say you wish to obtain just the March data from this file Using 0 based subscripts see Section 3 14 C and Fortran Index Conventions page 57 these data are stored in records 2 14 50 so the desired stride is 12 Without the stride option the procedure is very awkward One could use ncks five times and then use ncrcat to concatenate the resulting files together for idx in 02 14 26 38 50 do Bourne Shell ncks d time idx 8501_8912 nc foo idx done foreach idx 02 14 26 38 50 C Shell ncks d time idx 8501_8912 nc foo idx end ncrcat foo 8589_03 nc rm foo With the stride option ncks performs this hyperslab extraction in one operation ncks d time 2 12 8501 8912 nc 8589_03 nc See Section 4 7 ncks netCDF Kitchen Sink page 203 for more information on ncks Applying the stride option to the record dimension in ncra and ncrcat makes it possible for instance to average or concatenate regular intervals across multi file input data sets ncra F d time 3 12 85 nc 86 nc 87 nc 88 nc 89 nc 8589 03 nc ncrcat F d time 3 12 85 nc 86 nc 87 nc 88 nc 89 nc 8503 8903 nc 3 17 Record Appending Availability ncra ncrcat Short options None Long options rec_apn record append As of version 4 2 6 March 2013 NCO allows both Multi File Multi Record operators ncra and ncrcat to append their output directly to the end of an existing file Th
234. co openmp thread number 27 NCO availability ess cov Ret does eee 7 NCO homepage 0 0 e eee ne 7 NCO User Guide xii iieri 7 MCOd ness a veel MUERE ha 75 Mcpack 4 284 seat Mts dai dd oe es 217 DCPAG scene seal REIR 20 28 86 198 217 228 NCL A iiss oe tle eG uv LEV REUS 21 85 178 225 NCP CAU ie D ek oes genoa eder 20 28 228 licb CSZ Mica WAS Ru ERAI TS 211 ncrename 2 30 PRA MENS 84 230 CO o tive wha ee sobre a aaa eh ee tants 211 NGSA V heat DERE web AREE MU 12 DCSUD se eie te tee Ib Du trea t 189 nces btract vos ote v uere atin ae ne tend 189 ncuhlts eee hPa He YII UT Os eee 211 MCUNPACK erue epe ee T Ret eee eres 217 CWA cs sone aid tigen spies when Ped 21 28 85 178 235 DOS ica a poten el 147 nearbyint il A ae Sie gash Shee ens 178 nearest integer function exact 178 nearest integer function inexact 178 NEG III eve a ev E EDAX EG 8 Nel Dai oi ket IHRE IE 114 NESUING egos sie eles eee tlc bus 131 net C DP cs shtss A tre gee ste 7 NetCDF 2 vii ees medsler E Ib Rede r9 11 38 NETCDE2 0NLY 2 9 2525 03 da ww 12 netCDES8 hin ipae e eee et ede 11 38 netCDF3 classic file format 39 net CDE4 adi a adn 12 38 83 netCDF4 classic file format 39 netCDF4 file format 000 ee eee 39 NETCDF4 files ooo ccc woes bec baer wows 38 NETCDF4_CLASSIC files oooooooooo o 38 NETGDEA4Z
235. coeff gsl sf mathieu ce gsl sf mathieu se gsl sf mathieu ce array gsl sf mathieu se array gsl sf mathieu Mc gsl sf mathieu Ms gsl sf mathieu Mc array gsl sf mathieu Ms array gsl sf pow int e gsl sf psi int e gsl sf psi e gsl sf psi 1piy e gsl sf complex psi e gsl sf psi 1 int e gsl sf psi 1 e gsl sf psi n e gsl sf synchrotron 1 e gsl sf synchrotron 2 e gsl sf transport 2 e gsl sf transport 3 e gsl sf transport 4 e gsl sf transport 5 e gsl sf sin e gsl_sf_cos_e esl_sf_hypot_e gsl_sf_complex_sin_e AZAAAAZAAZZAZAZAANANANKKAKKAAKKKKKKKK ZAZZZ ZZZZKKKKKKKKKZKKK NCO 4 5 3 alpha03 User Guide gsl sf conicalP mhalf dbl expr dbl expr gsl_sf_conicalP_0 dbl_expr dbl_expr esl_sf_conicalP_1 dbl_expr dbl_expr gsl_sf_conicalP_sph_reg int_expr dbl_expr dbl_expr esl_sf_conicalP_cyl_reg int_expr dbl_expr dbl_expr gsl_sf_legendre_H3d_0 dbl_expr dbl_expr gsl_sf_legendre_H3d_1 dbl_expr dbl_expr gsl_sf_legendre_H3d int_expr dbl_expr dbl_expr gsl_sf_legendre_H3d_array esl_sf legendre_array_size gsl_sf_log dbl_expr gsl_sf_log_abs dbl_expr gsl_sf_complex_log gsl_sf_log_1plusx dbl_expr gsl_sf_log_1plusx_mx dbl_expr gsl_sf_mathieu_a_array gsl_sf_mathieu_b_array gsl sf mathieu a gsl sf mathieu b gsl sf mathieu a coeff gsl sf mathieu b coeff gsl sf mathieu ce gsl sf mathieu se gsl sf mathieu ce array gsl sf mathieu se array gsl sf mathieu
236. compatible with NCO 00000 8 1 3 1 Compiling NCO for Microsoft Windows OS 10 T4 Symbolic Links i2 o tei lat A ie ce ees 10 1 5 Libraries oe cdd Qk bea pete ou n REEL IIS 11 1 6 netCDF2 3 4 and HDP4 5 Supports 22 e rete xm eres 11 1 7 Help Requests and Bug Reports 0 0c eee eee eee 15 2 Operator Strategies 0 0 eee eee 17 2 1 Bhalosophiy 3i x ates oe hole ise Dec ees EDI eee a i 17 2 2 Climate Model ParadigM oocoooccccccnccccnnncccr rra 17 2 3 Temporary Output Files 0 00 2 eee 17 2 4 Appending Variables 0 00 uenn aeara 19 2 5 Simple Arithmetic and Interpolation 05 19 2 6 Statistics vs Concatenation 0 00 c cece eee eee eens 20 2 6 1 Concatenators ncrcat and ncecat eee ee eee 20 2 6 2 Averagers nces ncra and ncwa eee eee eee eee 21 2 6 3 Interpolator ncflint 0 cece cece eee eee eee 21 2 7 Large Numbers of Files 0 0 0 cece eee eee ee 21 2 8 Large Datasetg ernaar Hwee Rid ee eas 23 2 0 Memory Requirements 0 eee eee eee eee eee eens 24 2 9 1 Single and Multi file Operators 0 00 e ee eee 24 2 9 2 Memory for ncap2 cece eee ee n 25 2 10 Performanc cirerer ta ren ete oo kata Woe E kad 26 3 Shared Features 27 3 1 Internationalization sseseeeeesse nee eens 27 3 2 Metadata Optimiz
237. convention The first ACME rule implemented is that the contents of input file variables named date written and time written if any will be updated to the current system supplied with gntime O GMT time as the variables are copied to the output file You must spoof NCO if you would like any grid properties or other special CF fields processed normally For example rename the variables first with ncrename or alter the Conventions attribute As of NCO version 4 0 8 April 2011 NCO supports the CF bounds convention for cell boundaries described here http cfconventions org 1 6 htmlitcell boundaries This convention allows coordinate variables including multidimensional coordinates to describe the boundaries of their cells T his is done by naming the variable which contains the bounds in in the bounds attribute Note that coordinates of rank N have bounds of rank N 1 NCO generated subsets of CF compliant files with bounds attributes will include the coordinates specified by the bounds attribute if any Hence the subsets will themselves be CF compliant Bounds are subject to the user specified override switches including c and C described in Section 3 12 Subsetting Coordinate Variables page 47 As of NCO version 4 4 9 May 2015 NCO supports the CF climatology con vention for climatology statistics described here http cfconventions org Data cf conventions cf conventions 1 7 build cf conventions h
238. cript by Phil Rasch PJR echo Using remote CFchecker http puma nerc ac uk cgi bin cf checker pl cf lcn http puma nerc ac uk cgi bin cf checker pl for fl in drc_out nc do fl sml 1f1 cf out f14 nc html dmns ncdump h fl_in sed n e dimensions variables p grep se hyp sml for dmn in dmns do dmn lc echo dmn tr upper lower if dmn 1c lat dmn 1c latitude dmn_lc 1 hyp sml echo hyp_sml d dmn 0 fi dmn lc done Create small version of input file by sampling only first element of lat lo ncks 0 hyp_sm1 f1 fl_sm1 Send small file to NERC checker curl form cfversion 1 6 form upload 0 f1 sml form press Check 20file Strip most HTML to improve readability cat cf_out sed e s lt gt gt g e DOCTYPE d e s CF Conventi echo Full NERC compliance check log for f1 in cf_out 54 NCO 4 5 3 alpha03 User Guide done fi nerc EOF chmod 755 ncdismember Make command executable bin mv f ncdismember sh Store in location on PATH e g usr local bin zender roulee ncdismember nco data mdl 1 nc DATA nco tmp NCO dismembering file home zender nco data mdl 1 nc data zender nco tmp mdl 1 nc cesm cesm 01 nc data zender nco tmp mdl 1 nc cesm cesm 02 nc data zender nco tmp mdl 1 nc cesm nc data zender nco tmp mdl 1 nc ecmwf ecmwf 01 nc data zender nco tmp m
239. ctions on accessing the different file formats including 64 bit files supported by the modern netCDF interface If you are still interested in explicit LFS support for netCDF versions prior to 3 6 know that LFS support depends on a complex interlocking set of operating system and netCDF support issues The netCDF LFS FAQ http my unidata ucar edu content software netcdf faq lfs html describes the various file size limitations imposed by different versions of the netCDF standard NCO and netCDF automatically attempt to configure LFS at build time 3 11 Subsetting Files Options g grp Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options g Long options grp and group Options v var and x Availability ncap2 ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options v x Long options Options unn Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options Long options unn and union Options grp xtr var xcl Availability ncks Short options Long options variable exclude or xcl e gxvx and grp_xtr_var_xcl Subsetting variables refers to explicitly specifying variables and groups to be included or excluded from operator actions Subsetting is controlled by the v var and x options for directly specifying variables Spec
240. d are averaged by ncwa appear in the output file as scalars This allows the user to infer which dimensions may have been averaged Note that that it is impossible for ncwa to make make a weight or mask_var of rank W conform to a var of rank V if W gt V This situation often arises when coordinate variables which by definition are one dimensional are weighted and averaged ncwa assumes you know this is impossible and so ncwa does not attempt to broadcast weight or mask_var to conform to var in this case nor does ncwa print a warning message telling you this because it is so common Specifying dbg gt 2 does cause ncwa to emit warnings in these situations however 236 NCO 4 5 3 alpha03 User Guide Non coordinate variables are always masked and weighted if specified Coordinate vari ables however may be treated specially By default an averaged coordinate variable e g latitude appears in output file averaged the same way as any other variable containing an averaged dimension In other words by default ncwa weights and masks coordinate variables like all other variables This design decision was intended to be helpful but for some applications it may be preferable not to weight or mask coordinate variables just like all other variables Consider the following arguments to ncwa a latitude w lat wgt d latitude 0 90 where lat wgt is a weight in the latitude dimension Since by default ncwa weights coordinate variables the value of lati
241. d array variables T his bug corrupts affected variables by replacing their values with the default FillValue for that variable s type Demonstate bug in netCDF4 HDF5 library prior to netCDF 4 3 3 ncks 0 4 C M v lat nco data in nc bug nc ncrename 0 v lat tal bug nc foo nc Broken until netCDF 4 3 3 ncrename 0 d lat tal bug nc foo nc Broken until netCDF 4 3 3 ncrename d lat tal v lat tal bug nc foo nc Broken too ncks cdl foo nc To avoid this faulty behavior either build NCO with netCDF version 4 3 3 or later or convert the file to netCDF3 first then rename as intended then convert back This bug does not affect renaming of groups or of attributes The final netCDF bug that causes unintended side effects with ncrename affects only snapshots from January February 2015 and released version 4 3 3 February 2015 It was fixed in netCDF version 4 3 3 1 March 2015 This bug causes renamed attributes of coordinate variables in netCDF4 to files to disappear Demonstate bug in netCDF4 HDF5 library netCDF 4 3 3 ncrename 0 h a gi lonQOunits new units nco data in grp nc foo nc ncks v gi lon foo nc Shows units and new units are both gone As of netCDF version 4 3 3 1 March 2015 there are no known renaming bugs Create netCDF out nc identical to in nc except the attribute _FillValue is changed to missing value the attribute units is changed to CGS units but only in those variables
242. d be more fld out snc snd Folders of output files SSeS SSS So Se Se a eS SS a a ee ee i Se ee ee eS Se for var_id in 0 1 do Loop over two variables Names of all models 1s get file names cut get the part for model names sort uniq remove duplicates awk print mdl set 1s drc_in var var_id _ rlm _ _ xpt 0 _ nc cut d _ f 3 sort uniq c awk print 2 for mdl in mdl_set do Loop over models 258 NCO 4 5 3 alpha03 User Guide Loop over ensemble members for fn in 1s drc_in var var_id _ rlm _ md1 _ xpt 0 _ nc do pfx echo fn cut d f6 cut d f1 5 Two regions Geographical weight ncap2 0 s gw cos 1at 3 1415926 180 gwOlong name geographical weight gwOunits ratio fn drc_out fld_out var_id pfx _gw nc Greenland ncwa 0 w gw d lat 60 0 75 0 d lon 300 0 340 0 a lat lon drc_out fld_out var_id pfx _gw nc drc_out fld_out var_id pfx _gw_1 nc Tibet ncwa 0 w gw d 1at 30 0 40 0 d lon 80 0 100 0 a lat lon drc_out fld_out var_id pfx _gw nc drc_out fld_out var_id pfx _gw_2 nc Aggregate 2 regions together ncecat 0 u rgn drc_out fld_out var_id pfx _gw_ nc drc_out fld_out var_id pfx _gw_rgn4 nc Change dimensions order ncpdq 0 a time rgn drc_out fld_out var_id pfx _gw_rgn4 nc drc_out fld_out var_id
243. d in an external map file Use the map or rgr_map options to specify the map file and NCO will regrid the input file to a new or possibly the same aka an identity mapping horizontal grid in the output file using the input and output grids and mapping weights specified in the ESMF or SCRIP format map file Currently NCO understands only the mapfile format pioneered by SCRIP http oceansii lanl gov svn SCRIP trunk SCRIP and later extended by ESMF http www earthsystemcog org projects regridweightgen and adopted by Tempest https github com ClimateGlobalChange tempestremap git See those references for docu mentation on map formats grid specification and weight generation The regridding currently supported by NCO could equally well be called weight application NCO reads in pre stored weights from the map file and applies them to almost every variable thereby creating a regridded output file Specify regridding with a standard ncks command and options along with the additional specification of a map file Regrid entire file same output format as input ncks map map nc in nc out nc Entire file netCDF4 output ncks 4 map map nc in nc out nc Deflated netCDF4 output ncks 4 L 1 map map nc in nc out nc Selected variables ncks v FS T map map nc in nc out nc 74 NCO 4 5 3 alpha03 User Guide Threading ncks t 8 map map nc in nc out nc Deflated netCDF4 output threading selec
244. d is somewhat primitive e g having to repeat the rgr option Another aspect of this is that the output grid filename is distinct from the output filename of the host ncks command Specify the output gridfile name with grid e g t42_SCRIP 20150901 nc It is conventional to include a datestamp in the gridfile name This helps users identify up to date and out of date grids Any valid netCDF file may be named as the source e g in nc It will not be altered The destination file e g foo nc will be overwritten Its contents are immaterial Grid Types lat_grd lon_grd The keys that hold the longitude and latitude gridtypes which are by the way independent of eachother are lon grd and lat_grd The lat typ options for global grids are uni for Uniform cap or fv for Capped equivalent to FV and gss for Gaussian These values are all case independent so Gss and gss both work Chapter 3 Shared Features 71 As its name suggests the latitudes in a Uniform latitude grid are uniformly spaced The Uniform latitude grid may have any number of latitudes NCO can only generate longitude grids below that are uniformly spaced so the Uniform latitude grids we describe are also uniform in the 2D sense Uni form grids are intuitive easy to visualize and simple to program Hence their popularity in data exchange visualization and archives Moreover regional grids unless they include the poles
245. d processing as well 7 1 Combine Files Sometimes the data of one ensemble member will be stored in several files to reduce single file size It is more convenient to concatenate these files into a single timeseries and the following script illustrates how Key steps include 1 Obtain number and names or partial names of files in a directory 2 Concatenate files along the record dimension usually time using ncrcat see Section 4 10 ncrcat netCDF Record Concatenator page 228 bin bash shell type shopt s extglob enable extended globbing 248 NCO 4 5 3 alpha03 User Guide Some of the models cut one ensemble member into several files which include data of different time periods We d better concatenate them into one at the beginning so that we won t have to think about which files we need if we want to retrieve a specific time period later Method Make sure time is the record dimension i e left most ncrcat Input files like data cmip5 snc LImon bcc csm 1 historical riiipi 185001 190012 nc data cmip5 snc LImon bcc csm 1 historical riiipi 190101 200512 nc Output files like data cmip5 snc LImon bcc csm 1 historical riiipi 185001 200512 nc Online http nco sourceforge net nco html Combine Files dk dk dt dt HHH HH dbi dt dt dt dt dt dt dt dt dto OF Execute this script bash cmb fl sh drc_in home wenshanw data cmip5 Directory of input files var snc snd Var
246. d_out var_id pfx _dlt nc done RMS ncea 0 y rmssdn drc_out fld_out var_id var var_id _ rlm _ mdl xpt O dlt nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_sdv nc Rename variables ncrename v var var_id sdv drc_out f1ld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_sdv nc Edit attributions ncatted a standard_name sdv a c _standard_deviation_over_ensemble a long_name sdv a c Standard Deviation over Ensemble a original_name sdv a c sdv drc_out f1ld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_sdv nc Have to change the name of variable in the commands file of gsl_rgr nco manually gsl_rgr nco is listed below ncap2 0 S gsl rgr nco drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_anm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_anm_rgr nc Get rid of temporary variables ncks 0 v cO cl pval var var_id gw A drc_out f1ld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_anm_rgr nc A 260 NCO 4 5 3 alpha03 User Guide drc_out fld_out var_id var var_id _ mdl nc Move the variable sdv into the anomaly files i e anm nc files ncks A v sdv drc_out f ld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_sdv nc drc out fld out var id ivar v
247. data in one file with data or metadata from another Extract naked data only copies of two variables ncks h M m 0 C v one three dmn rec var nco data in nc out nc Change values to be sure original values are not copied in following step ncap2 0 v s one 2 three dmn rec var 0 nco data in nc in2 nc Append in2 nc metadata not data to out nc ncks A C H v one three dmn rec var in2 nc out nc Variables in out nc now contain data not metadata from in nc and metadata not data from in2 nc Print variable three dmn var from file in nc with default notations Next print three dmn var as an un annotated text column Then print three dmn var signed with very high precision Finally print three dmn var as a comma separated list 4 ncks C v three dmn var in nc lat 0 90 lev 0 100 lon 0 0 three dmn var 0 0 lat 0 90 lev 0 100 lon 1 90 three dmn var 1 1 lat 1 90 lev 2 1000 lon 3 270 three dmn var 23 23 4 ncks s 4f n C v three dmn var in nc 0 000000 1 000000 23 000000 4 ncks s 4 16 10fNn C v three dmn var in nc 0 0000000000 1 0000000000 23 0000000000 4 ncks s Af C v three dmn var in nc 0 000000 1 000000 23 000000 Programmers will recognize these as the venerable C language printf O formatting strings The second and third options are useful when pasting data into text files like reports or papers See Section 4 2 ncatted netCDF Attribu
248. del to all input files though not to output file J Thus using p the path to any number of input files need only be specified once Note input path need not end with the is automatically generated if necessary The last method passes with n syntax concisely describing the entire set of filenames This option is only available with the multi file operators ncra ncrcat nces and ncecat By definition multi file operators are able to process an arbitrary number of input files This option is very useful for abbreviating lists of filenames representable as alphanu meric prefix numeric suffix filetype where alphanumeric prefix is a string of arbitrary length and composition numeric_suffix is a fixed width field of digits and filetype is a standard filetype indicator For example in the file ccm3_h0001 nc we have alphanu meric prefix ccm3 h numeric suffix 0001 and filetype nc NCO decodes lists of such filenames encoded using the n syntax The simpler three argument n usage takes the form n file number digit number numeric increment where file number is the number of files digit number is the fixed number of numeric dig its comprising the numeric suffix and numeric increment is the constant integer valued difference between the numeric_suffix of any two consecutive files The value of alphanu meric prefix is taken from the input file which serves as a template for decoding the
249. dl 1 nc ecmwf ecmwf 02 nc data zender nco tmp mdl 1 nc ecmwf nc data zender nco tmp mdl 1 nc root nc A potentially more portable binary executable could be written to dismember all groups with a single invocation yet dismembering without loss of information is possible now with this simple script on all platforms with UNIXy utilities Note that all dimensions inherited by groups in the input file are correctly placed by ncdismember into the flat files Moreover each output file preserves the group metadata of all ancestor groups including the global metadata from the input file As written the script could fail on groups that contain advanced netCDF4 features because the user requests with the 3 switch that output be netCDF3 classic format However ncks detects many format incompatibilities in advance and works around them For example ncks autoconverts netCDF4 only atomic types such as NC STRING and NC UBYTE to corresponding netCDF3 atomic types NC CHAR and NC SHORT when the output format is netCDF3 3 13 4 Checking CF compliance One application of dismembering is to check the CF compliance of each group in a file When invoked with the optional third argumnt cf ncdismember passes each file it generates to the freely available cfchecker command zender roulee ncdismember nco data mdl 1 nc data zender nco tmp cf NCO dismembering file home zender nco data mdl 1 nc CFchecker reports CF compliance of each g
250. dled this demotion in three ways in its history Prior to October 2011 version 4 0 8 NCO employed the C library truncate function trunc O Truncation rounds x to the nearest integer not larger in absolute value For example truncation rounds 1 0d 1 5d and 1 8d to the same value 1s Clearly truncation does not round floating point numbers to the nearest integer Yet truncation is how the C language performs implicit conversion of real numbers to integers NCO stopped using truncation for demotion when an alert user Neil Davis informed us that this caused a small bias in the packing algorithm employed by ncpdq This led to 27 32767 2 1 5 Operators began performing automatic type conversions before arithmetic in NCO version 1 2 August 2000 Previous versions never performed unnecessary type conversion for arithmetic 22 The actual type conversions with trunction were handled by intrinsic type conversion so the trunc function was never explicitly called although the results would be the same if it were Chapter 3 Shared Features 115 NCO adopting rounding functions for demotion Rounding functions eliminated the small bias in the packing algorithm From February 2012 through March 2013 versions 4 0 9 4 2 6 NCO employed the C library family of rounding functions 1round O These functions round x to the nearest integer halfway cases away from zero The problem with 1round O is that it always rounds real values ending
251. duce correctly weighted win ter season mean statistics The second example weights the input per timestep to produce correctly weighted mean statistics 228 NCO 4 5 3 alpha03 User Guide 4 10 ncrcat netCDF Record Concatenator SYNTAX ncrcat 3 4 6 7 4 C c cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stride subcycle1111 F G gpe dsc g grp glb h hdr_pad nbr L dfl 1vl 1 path md5 digest n loop no tmp f1 0 o output file p path ppc R r ram all rec apn t thr nbr unn v var X x input files output file DESCRIPTION ncrcat concatenates record variables across an arbitrary number of input files The final record dimension is by default the sum of the lengths of the record dimensions in the input files See Section 2 6 Statistics vs Concatenation page 20 for a description of the distinctions between the various statistics tools and concatenators As a multi file operator ncrcat will read the list of input files from stdin if they are not specified as positional arguments on the command line see Section 2 7 Large Numbers of Files page 21 Input files may vary in size but each must have a record dimension l he record coordi nate if any should be monotonic or else non fatal warnings may be generated Hyperslabs along the record
252. e Ivl_nbr is a negative integer which specifies the maximum number of group path components i e groups that GPE will try to truncate from the tail of grp_pth For example Ivl_nbr 3 changes the input path g1 g2 g3 g4 g5 to the output path gi g2 Input paths with Ivl_nbr or fewer components groups are completely erased and the output path commences from the root level In Flatten mode indicated by the separator character alone or with lvl nbr 0 GPE removes the entire group path from the input file and constructs the output path beginning at the root level For example G 0 and G are identical and change the input path g1 g2 g3 g4 g5 to the output path whereas G g1 0 and G g1 are identical and result in the output path g1 for all variables Subsequent to the alteration of the input path by the specified editing mode if any GPE prepends in Delete Mode or Appends in Truncate mode any specifed grp_pth to the out put path For example G g2 changes the input paths and g1 to g2 and g1 g2 respec tively Likewise G g2 g3 changes the input paths and g1 to g2 g3 and g1 g2 g3 respectively When grp_pth and Ivl_nbr are both specified the editing actions are taken in sequence so that e g G g1 g2 2 changes the input paths and h1 h2 h3 h4 to g1 g2 and gi g2 h3 h4 respectively Likewise G g1 g2 2 changes the input paths and h1 h2 h3 h4 to g1 g2 and h1 h2 g1 g2 respectively Combining GPE with subsett
253. e links impute the passing of a command line argument to further modify the behavior of the underlying executable For example ncdivide is a pseudonym for ncbo op typ 1 5 Libraries Like all executables the NCO operators can be built using dynamic linking This reduces the size of the executable and can result in significant performance enhancements on mul tiuser systems Unfortunately if your library search path usually the LD_LIBRARY_PATH environment variable is not set correctly or if the system libraries have been moved re named or deleted since NCO was installed it is possible NCO operators will fail with a message that they cannot find a dynamically loaded aka shared object or so library This will produce a distinctive error message such as ld so 1 usr local bin nces fatal libsunmath so 1 can t open file errno 2 If you received an error message like this ask your system administrator to diagnose whether the library is truly missing or whether you simply need to alter your library search path As a final remedy you may re compile and install NCO with all operators statically linked 1 6 netCDF2 3 4 and HDF4 5 Support netCDF version 2 was released in 1993 NCO specifically ncks began soon after this in 1994 netCDF 3 0 was released in 1996 and we were not exactly eager to convert all code to the newer less tested netCDF implementation One netCDF3 interface call nc inq libvers was added to N
254. e ncl_convert2nc tool to convert HDF to netCDF3 when both these are true 1 You must have netCDF3 and 2 the HDF file contains netCDF4 atomic types More recent versions of NCO handle this problem fine and include other advantages so we no longer recommend ncl_convert2nc because ncks is faster and more space efficient Both automati cally convert netCDF4 types to netCDF3 types yet ncl_convert2nc cannot produce full netCDF4 files In contrast ncks will happily convert HDF straight to netCDF4 files with netCDF4 types Hence ncks can and does preserve the variable types Unsigned bytes stay unsigned bytes 64 bit integers stay 64 bit integers Strings stay strings Hence ncks conversions often result in smaller files than ncl_convert2nc conversions A tool useful for converting netCDF3 to netCDF4 files is the Python script nc3tonc4 by Jeff Whitaker 10 Two real world examples NCO translates the NASA CERES dimension FOV Footprints to _FOV_ Footprints and Cloud amp Aerosol Cloud Only Clear Sky w Aerosol and Clear Sky yes the dimen sion name includes whitespace and special characters to Cloud amp Aerosol Cloud Only Clear Sky w_ Aerosol and Clear Sky ncl convert2nc makes the element name netCDF safe in a slightly different manner and also stores the original name in the hdf name attribute Chapter 1 Introduction 15 One must often build NCO from source to obtain netCDF4 support Typically one specifies the root of the netCDF4 install
255. e oe 186 3 32 bit offset file format oooooooocoommm 39 6 64 bit offset file format oooooocoooommm 39 GABIT filesi ereo indie eee ROI eens bie pr nete 38 A A nes eUduEDLORU NuAS Od bPPSUEE 178 absolute value 00 eee cece eee eee eens 178 ACME conventions eene 124 BOOS A n ne pl Rar P es ees ye NA 178 ACOSlL qu V Lrel4r Pate E PE MPa T bU 178 BOG ss MICE 189 add_offset 50 26 105 198 217 228 adding data irte Ne 189 200 addition cep nia Bh SSE he 178 189 200 aggregation picks epbertemerii9 DP AT CAR ES 249 Alejandro Soto eese 242 Aleksandar Jelenak 0 200 eee 242 Alexander Hansen 0 020 eeeeeeee 242 alias seca Seen tek eget T SELLERS 190 211 218 ald Sealey tee es kak eA tae eae ete 86 alphabetization eese 204 alphabetize output 0 0 eee eee ee eee 213 alternate invocations eee eee ee eee 189 AUC ss caro nisi A Wada ear Ne 44 ANCHOTING whet ti koe het Ae Bn IS 44 ancillary variables convention 126 ancillary_variables attribute 126 215 Andrea Cimatoribus 200 200 ee 242 Andrew Wittenberg eese 242 annual average tdi ase dis bed 256 annual average from daily data 245 annual average from monthly data 245 anomalies 00 e cece eee eee 191 256 263 ANST A eee E SAM CRIT SU 8 ANST CE Pate omes Patent e wi
256. e r 280 NCO 4 5 3 alpha03 User Guide iap_fgoals1_0_g inmcm3_0 ipsl cm4 miroc3 2 hires miroc3 2 medres miub echo g mpi_echam5 mri cgcm2 3 2a ncar_ccsm3_0 ncar pcmi ukmo hadcm3 ukmo hadgemi do for run in 1 do ncks R 0 D 3 p ftp climate 1llnl gov scn atm mo var mdl run run done end loop over run done end loop over mdl done end loop over scn done end loop over var cd sresalb atm mo tas ukmo_hadcm3 run1 ncks H m v lat lon lat bnds lon bnds M tas Ai nc m bds x 096 y 073 m 33 o DATA data dst_3 75x2 5 nc ukmo_hadcm3 ncview DATA data dst_3 75x2 5 nc msk rgn is California mask on ukmo hadcm3 grid area is correct area weight on ukmo hadcm3 grid ncks A v area msk rgn DATA data dst_3 75x2 5 nc DATA sresalb atm mo tas ukmo_hadcm3 run1 area_msk_ukmo_hadcm3 nc Template for standardized data scn _ md1 _ run _ var _ yyyymm _ yyyymm nc e g raw data DATA sresaib atm mo tas ukmo_hadcm3 runi tas_Al1 nc becomes standardized data Level 0 raw from IPCC site no changes except for name Make symbolic link name match raw data Template scn _ md1 _ run _ var _ yyyymm _ yyyymm nc ln s f tas_A1 nc sresaib_ukmo_hadcm3_runi_tas_200101_209911 nc area_msk_ukmo_hadcm3 nc Level I Add all variables not standardized in time to file containing msk_rgn and area Template scn _ md1 _ run _ yyyymm _ yyyymm nc bin cp area_msk_ukmo_hadcm3 nc sresai
257. e suitably filtered input file list to the end of the ncecat command options The o foo nc switch ensures that the input files supplied by xargs are not confused with the output file name xargs does unfortunately have its own limit usually about 20 000 characters on the size of command lines it can pass Give xargs the x switch to ensure it 7 The exact length which exceeds the operating system internal limit for command line lengths varies from OS to OS and from shell to shell GNU bash may not have any arbitrary fixed limits to the size of command line arguments Many OSs cannot handle command line arguments including results of file globbing exceeding 4096 characters Chapter 2 Operator Strategies 23 dies if it reaches this internal limit When this occurs use either the stdin method above or the symbolic link presented next Even when its internal limits have not been reached the xargs technique may not be sophisticated enough to handle all situations A full scripting language like Perl can handle any level of complexity of filtering input filenames and any number of filenames The technique of last resort is to write a script that creates symbolic links between the irregular input filenames and a set of regular arithmetic filenames that the n loop switch understands For example the following Perl script creates a monotonically enumerated symbolic link to up to one million nc files in a directory If there are 99
258. e swarms RI D ER tages se 147 U ubyte sicnesbeiie cbr Gaede goes ipe eee net 148 UDU nits astaat hie Rp 8 79 124 unt chek a ir peii iia 148 Wlimit 4 Re Maes eth eee eas 23 unary operations s s 25 underlying file format 0 08 40 UNICOS 2 2nbpesib REL ads Ede 23 Unid t tia De CEU E es 8 12 79 UMON onah NE MCENUNA RENTA 43 45 unionof files 423 2 eer Prey ee a demere 19 UNILOA O aga O 148 Uitg ageless has 75 79 80 187 202 UND 8 10 22 29 30 211 unlimited dimension ssseeeesss 197 O ensti ois 152 unpack cintas ocre um bea td 147 unpack x i nbi ee eeu bb b ERSTE eras 105 unpacking 36 105 198 217 228 unstructured grid panen ETEN 68 URL saath sale beled thas tea bg 33 User Guides eee 7 ushort 1 eet ad el 148 V value listo seriei steur yepe pelar nie Ei 140 variable names eee eee eee eee 230 variables appending eee co 256 NCO 4 5 3 alpha03 User Guide VANO conss vec ie aes Verc Ex encor 107 VOTSION a A ERI T me Lathe anne apo 127 Vista Microsoft operating system 10 W Weather and Research Forecast WRF Model AA DUxpeetu p ERREE Testen enis 150 weighted average 000005 235 263 Wenshan Wang edere PIA Rebus 243 WOSIhossoilscelQe pM ROM PST UP RON HPV 72 MESS tee aW Wed deu ee ee 33 wet thr i ete V UR a a a A eat 74 where sn heii vec eee ener ee 149 Wile ocd
259. e the 1 in the denominator compensates for the degree of freedom already used in computing y in the numerator Consult a statistics book for more details When weights are specified it is unclear how to compensate for this extra degree of freedom Weighting the numerator and denominator of the above by w and subtracting one from the denominator is only appropriate when all the weights are 1 0 When the weights are arbitrary e g Gaussian weights subtracting one from the sum in the denominator does not necessarily remove one degree of freedom Therefore when y rmssdn is requested and weights are specified ncwa actually implements the rms procedure nces and ncra which do not allow weights to be specified always implement the rmssdn procedure when asked Chapter 3 Shared Features 111 20130827 Fedora Core 19 FC19 broke here with nco texi 6394 Missing dollarsign inserted Ubuntu always built nco texi fine Adding a dollarsign character right here breaks Ubuntu builds too Hence I must carefully spell out the word dollarsign instead 20130829 Making many smaller TeX environments does not solve problem 20130910 Using latest texinfo tex from GNU does not solve problem 20130910 Karl Berry solved problem by fixing bug in texinfo tex Bug was triggered in Fedora by apostrophe in User s Guide manual title Bug not present in texinfo tex version 2008 04 18 10 used by Ubuntu 13 04 Bug present in texinfo tex version 2013 02 01 11 used by
260. e this capability HDF version 5 became available in 1999 but did not support netCDF or for that matter Fortran as of December 1999 By early 2001 HDF5 did support Fortran90 Thanks to an NSF funded harmonization partnership HDF began to fully support the netCDF3 read interface which is employed by NCO 2 x and later In 2004 Unidata and THG began a project to implement the HDF5 features necessary to support the netCDF API NCO version 3 0 3 added support for reading writing netCDF4 formatted HDF5 files in October 2005 See Section 3 9 File Formats and Conversion page 38 for more details HDF support for netCDF was completed with HDF5 version version 1 8 in 2007 The netCDF front end that uses this HDF5 back end was completed and released soon after as netCDF version 4 Download it from the netCDF4 http my unidata ucar edu content software netcdf netcdf 4 website NCO version 3 9 0 released in May 2007 added support for all netCDF4 atomic data types except NC_STRING Support for NC_STRING including ragged arrays of strings was finally added in version 3 9 9 released in June 2009 Support for additional netCDF4 features has been incremental We add one netCDF4 feature at a time You must build NCO with netCDF4 to obtain this support NCO supports many netCDF4 features including atomic data types Lempel Ziv com pression deflation chunking and groups The new atomic data types are NC_UBYTE NC_USHORT NC_UINT NC
261. e whether d is intended as a truncation of dbg 1lv1 of dimension or of some other long option NCO provides many long options for common switches For example the debugging level may be set in all operators with any of the switches D debug level or dbg lvl This flexibility allows users to choose their favorite mnemonic For some it will be debug an unambiguous truncation of debug level and other will prefer dbg Interactive users usually prefer the minimal amount of typing i e D We recommend that scripts which are re usable employ some form of the long options for future maintainability This manual generally uses the short option syntax in examples This is for historical reasons and to conserve space in printed output Users are expected to pick the unambiguous truncation of each option name that most suits their taste 3 5 Specifying Input Files Availability n nces ncecat ncra ncrcat Availability p All operators OA en AR Short options n p Long options nintap pth path It is important that users be able to specify multiple input files without typing every filename in full often a tedious task even by graduate student standards There are four different ways of specifying input files to NCO explicitly typing each using UNIX shell wild cards and using the NCO n and p switches or their long option equi
262. ea Mask a file to contain FillValue everywhere except where thr min lt msk_var lt thr_max Set masking variable and its scalar thresholds export msk_var three_dmn_var_dbl Masking variable export thr_max 20 Maximum allowed value export thr_min 10 Minimum allowed value ncecat 0 in nc out nc Wrap out nc in degenerate record dimension ncwa 0 a record B msk_var lt thr_max out nc out nc ncecat 0 out nc out nc Wrap out nc in degenerate record dimension ncwa 0 a record B msk var gt thr min out nc out nc After the first use of ncwa out nc contains FillValue where msk var gt thr_ max The process is then repeated on the remaining data to filter out points where msk_ var lt ithr min The resulting out nc contains valid data only where thr min lt msk_var lt thr max 5 ORO stands for Orography in some climate models and in those models ORO lt 0 5 selects ocean gridpoints E Unfortunately the B and mask_condition options are unsupported on Windows with the MVS compiler which lacks a free standard parser and lexer Chapter 5 Contributing 241 5 Contributing We welcome contributions from anyone The project homepage at https sf net projects nco contains more information on how to contribute Financial contributions to NCO development may be made through PayPal https www paypal com xclick business zender 40uci ed
263. eates the binary flag ORO_flg in out nc from the continuous array named ORO in in nc ncap2 s ORO_flg ORO gt 1 0 in nc out nc Suppose your task is to change all values of ORO which equal 2 0 to the new value 3 0 ncap2 s ORO_msk ORO 2 0 ORO 0R0_msk 3 0 0R0_msk 0R0 in nc out nc This creates and uses ORO msk to mask the subsequent arithmetic operation Values of ORO are only changed where ORO msk is true i e where ORO equals 2 0 Using the where statement the above code simplifies to ncap2 s where 0R0 2 0 ORO 3 0 in nc foo nc This example uses ncap2 to compute the covariance of two variables Let the variables u and v be the horizontal wind components The covariance of u and v is defined as the time mean product of the deviations of u and v from their respective time means Symbolically the covariance u v uv u v where x denotes the time average of x gt ed x t dt and x denotes the deviation from the time mean The covariance tells us how much of the correlation of two signals arises from the signal fluctuations versus the mean signals Sometimes this is called the eddy covariance We will store the covariance in the variable uprmvprm ncwa 0 a time v u v in nc foo nc Compute time mean of u v ncrename 0 v u uavg v v vavg foo nc Rename to avoid conflict ncks A v uavg vavg foo nc in nc Place time means with originals ncap2 0 s uprmvprm u v uavg vavg in nc in nc Cov
264. ecause the UDUnits feature is additive not exclusive it works in conjunction with all other hyperslabbing see Section 3 15 Hyperslabs page 58 options and in all operators which support hyperslabbing The following example shows how one might average data in a time period spread across multiple input files ncra d time 1939 09 09 12 00 0 0 1945 05 08 00 00 0 0 ini nc in2 nc in3 nc out nc Note that there is no excess whitespace before or after the individual elements of the d argument This is important since as far as the shell knows d takes only one command line argument Parsing this argument into its component dim min max stride elements see Section 3 15 Hyperslabs page 58 is the job of NCO When unquoted whitespace is present between these elements the shell passes NCO arugment fragments which will not parse as intended NCO implemented support for the UDUnits2 library with version 3 9 2 August 2007 The UDUnits2 http www unidata ucar edu software udunits udunits 2 udunits2 html package supports non ASCII characters and logarithmic units We are interested in user feedback on these features Chapter 3 Shared Features 81 One aspect that deserves mention is that UDUnits and thus NCO supports run time definition of the location of the relevant UDUnits databases With UDUnits version 1 users may specify the directory which contains the UDUnits database udunits dat via
265. ecision weight gw double precision arithmetic NB Values below are printed with C format 7 5 6f using i ncks H C s 5 6f v TREFHT TREFHT dbl out nc Single precision weight area single precision arithmetic ncwa flt 0 a lat lon w area in nc out nc TREFHT 289 246735 TREFHT_dbl 289 239964 Double precision weight gw single precision arithmetic ncwa flt 0 a lat lon w gw in nc out nc TREFHT 289 226135 TREFHT_dbl 289 239964 Single precision weight area double precision arithmetic ncwa dbl 0 a lat lon w area in nc out nc TREFHT 289 239960 TREFHT_dbl 289 239964 Double precision weight gw double precision arithmetic ncwa dbl 0 a lat lon w gw in nc out nc TREFHT 289 239960 TREFHT_dbl 289 239964 First note that the TREFHT_dbl average never changes because TREFHT_db1 lat lon is double precision in the input file As described above NCO automatically converts all operands involving to the highest precision involved in the operation So specifying db1 is redundant for double precision inputs Second the single precision arithmetic averages of the single precision input TREFHT differ by 289 246735 289 226135 0 0206 from eachother and more importantly by as much as 289 239964 289 226135 0 013829 from the correct double precision answer These averages differ in the fifth digit i e they agree only to four significant figures Given
266. ed because there is no type of higher precision Conversion of type NC_FLOAT is discussed in detail below When it occurs it follows the same procedure promotion then arithmetic then demotion as conversion of integer types When these criteria are all met the operator promotes the variable in question to type NC_DOUBLE performs all the arithmetic operations casts the NC_DOUBLE type back to the original type and finally writes the result to disk The result written to disk may not be what you expect because of incommensurate ranges represented by different types and because of lack of rounding First continuing the above example the average e g y avg of 17000s and 17000s is written to disk as 17000s The type conversion feature of NCO makes this possible since the arithmetic and intermediate values are stored as NC DOUBLES i e 34000 0d and only the final result must be represented as an NC SHORT Without the type conversion feature of NCO the average would have been garbage albeit predictable garbage near 15768s Similarly the total e g y tt1 of 17000s and 17000s written to disk is garbage actually 31536s since the final result the true total of 34000 is outside the range of type NC SHORT After arithmetic is computed in double precision for promoted variables the intermediate double precision values must be demoted to the variables original storage type e g from NC DOUBLE to NC SHORT NCO has han
267. ed to execute multiple ncks commands to gather the desired range of data The MSA user order switch msa_usr_rdr or msa user order both of which shorten to msa requests that the multislabs be output in the user specified order from the command line rather than in the input file on disk storage order This allows the user to perform complex data re ordering in one operation that would otherwise require cumbersome steps of hyperslabbing concatenating and permuting Consider the recent example of a user who needed to convert datasets stored with the longitude coordinate Lon ranging from 180 180 to datasets that follow the 0 360 convention ncks H v Lon in nc Lon 0 180 Lon 1 90 Lon 2 0 Lon 3 90 Although simple in theory this task requires both mathematics to change the numerical value of the longitude coordinate data hyperslabbing to split the input on disk arrays at Greenwich and data re ordering within to stitch the western hemisphere onto the eastern hemisphere at the date line The msa user order switch overrides the default that data are output in the same order in which they are stored on disk in the input file and instead stores them in the same order as the multi slabs are given to the command line This default is intuitive and is not important in most uses However the MSA user order switch allows users to meet their output order needs by specifying multi slabs in a certain order
268. edit global attributes or to repeat the editing operation for every group or variable in a file A value of var_nm of global indicates that att nm refers to a global i e root level attribute rather than to a particular variable s attribute This is the method ncatted supports for editing global attributes A value of var nm of group indicates that att nm refers to all groups rather than to a particular variable s or group s attribute The operation will proceed to edit group metadata for every group Finally if var nm is left blank then ncatted attempts to perform the editing operation on every variable in the file This option may be convenient to use if you decide to change the conventions you use for describing the data The value of mode is a single character abbreviation a c d m or o standing for one of five editing modes a Append Append value att_val to current var nm attribute att nm value att val if any If var nm does not have an attribute att nm there is no ef fect c Create Create variable var nm attribute att nm with att val if att nm does not yet exist If var nm already has an attribute att nm there is no effect so the existing attribute is preserved without change d Delete Delete current var nm attribute att_nm If var nm does not have an attribute att_nm there is no effect If att nm is omitted left blank then all at tributes associated with the specified variable are automatically deleted When Dele
269. ee 236 ii iv NCO 4 5 3 alpha03 User Guide 4 12 2 Normalization and Integration 00 0005 237 5 Gon bling cots poems ety ts bets 241 DL Contributors ic ceecebeseene envers Eb eee er eo 241 DL OA OA LY E xtd bet a RN ES oh eee eats 243 5 3 Proposals for Institutional Funding sseeessssse 243 6 Chick Starts sh eco Fes homes e emere rend 245 6 1 Daily data in one file 10 eee e 245 6 2 Monthly data in one file 245 6 3 One time point one file 2 eee eee 246 6 4 Multiple files with multiple time points 246 7 SOMIP5 Example 247 l Combine Files 2 22 er ob eee te RR A oe kee ies 247 7 2 Global Distribution of Long term Average 5 253 7 3 Annual Average over Regions 000 cece eee eens 256 TA Monthly Cycle eee kem ere 263 7 5 Regrid MODIS Data lssessessesseee een 266 7 6 Add Coordinates to MODIS Data 0 ce eee eee eee 269 7 7 Permute MODIS Coordinates 0 c cece eee eens 270 B Parallel ota e ea ve Bons toe ata needs alee 273 9 CCSM Example cec de tse Ch cts 275 10 References 0 eee 283 General de od CONSER 285 Foreword 1 Foreword NCO is the result of software needs that arose while I worked on projects funded by NCAR NASA and ARM Thinking they might prove useful as tools or templates to others it is my pleasure to provide them freely to the scientific community Many
270. ee methods produce identical answers See Section 3 5 Specifying Input Files page 30 for an explanation of the distinctions between these methods Assume the files 85 nc 86 nc 89 nc each contain a record coordinate time of length 12 defined such that the third record in 86 nc contains data from March 1986 etc NCO knows how to hyperslab the record dimension across files Thus to average data from December 1985 through February 1986 ncra d time 11 13 85 nc 86 nc 87 nc 8512 8602 nc ncra F d time 12 14 85 nc 86 nc 87 nc 8512 8602 nc The file 87 nc is superfluous but does not cause an error The F turns on the Fortran 1 based indexing convention The following uses the stride option to average all the March temperature data from multiple input files into a single output file ncra F d time 3 12 v temperature 85 nc 86 nc 87 nc 858687 03 nc See Section 3 16 Stride page 60 for a description of the stride argument Assume the time coordinate is incrementally numbered such that January 1985 1 and December 1989 60 Assuming only expands to the five desired files the following averages June 1985 June 1989 ncra d time 6 54 nc 8506 8906 nc The w wgt option weights input data per file or per timestep ncra w 31 31 28 dec nc jan nc feb nc out nc ncra w delta t in nc in2 nc in3 nc out nc Chapter 4 Reference Manual 227 The first example weights the input differently per file to pro
271. eed for the shell loop used in Step 1 above The new SSC option aggregates more than one input record at a time before performing arithmetic operations and with an additional switch allows us to archive those results in multiple record output MRO files This reduces the task of producing the climatological summertime mean to one step Step 1 Compute climatological summertime mean ncra 0 F d time 6 12 3 n 150 4 1 1850 nc clm JJA nc The SSC option instructs ncra or ncrcat to process files in groups of three records To better understand the meaning of each argument to the d hyperslab option read it this way for the time dimension start with the sixth record continue without end repeat the process every twelfth record and define a sub cycle as three consecutive records A separate option mro instructs ncra to output its results from each sub group and to produce a Multi Record Output MRO file rather than a Single Record Output SRO file Unless mro is specified ncra collects together all the sub groups operates on their ensemble and produces a single output record The addition of mro to the above example causes ncra to archive all 150 annual summertime means to one file Step 1 Archive all 150 summertime means in one file ncra mro O F d time 6 12 3 n 150 4 1 1850 nc 1850 2009 JJA nc or all 150 annual means ncra mro 0 d time 12 12 n 150 4 1 1850 nc 1850 2009
272. eers 183 attributes appending 0008 184 attributes creating esses 184 attributes deleting ooooooomoooo 184 attributes editing 184 256 266 attributes global 22 122 123 127 184 187 203 204 213 231 233 attributes modifying 184 256 266 attributes overwriting 184 256 266 attributesncap2 22 Ve PUR ETPERFuTS 139 amp utoGconf essi hoo ipae a A t cpt ROV np teet 16 autoconversion eee 41 automaglCcic iaa cit NE 9 21 automatic type conversion 113 179 auxiliary coordinates 67 75 126 average 107 236 245 246 253 256 263 averaging data 4 84 194 225 235 OV Gs Belt stb bees See ee Eee ele s 107 AVE edt sssteedst uot Oh bte tas teat cutie 146 AVESQE uo a be thea ee 107 AOS eek dod ds ales 75 289 Bas Couwenberg 0 00 0 eese 242 base tilme i coe td paa 127 Das int ia US nen 4T 211 Bash Shell or i e Ik Rexel x EET 193 Bash shell eei ceed ta 190 211 batch inode 22s ceed ei RE aad sled shy eee ee 121 GOR A ue sra E mH ER Y Ee m RUE 120 benchmarks ita os a iR Ree nbi 29 Bessel function 0 00 cee eee ee eee eee 162 bilinear interpolation 20006 266 Bill Kocik todas bur hR RR EU ETE ah 242 binary forniat 4 ssec pa ER PER EDU 205 binary operations eeeseeee 25 189 binary Operators
273. eference Manual 201 0 5 0 5 Attempting to specify both i and w methods in the same command is an error ncflint does not interpolate variables of type NC CHAR and NC STRING This behavior is hardcoded By default ncflint interpolates or multiplies record coordinate variables e g time is often stored as a record coordinate not other coordinate variables e g latitude and longi tude This is because ncflint is often used to time interpolate between existing files but is rarely used to spatially interpolate Sometimes however users wish to multiply entire files by a constant that does not multiply any coordinate variables The fix_rec_crd switch was implemented for this purpose in NCO version 4 2 6 March 2013 It prevents ncflint from multiplying or interpolating any coordinate variables including record coor dinate variables Depending on your intuition ncflint may treat missing values unexpectedly Consider a point where the value in one input file say vall equals the missing value mss val 1 and at the same point the corresponding value in the other input file val2 is not misssing i e does not equal mss val 2 There are three plausible answers and this creates ambiguity Option one is to set val3 mss_val_1 The rationale is that ncflint is at heart an interpolator and interpolation involving a missing value is intrinsically undefined ncflint currently implements this behavior since it is the
274. ematical functions supported is platform specific Functions man dated by ANSI C are guaranteed to be present and are indicated with an asterisk and are indicated with an asterisk Use the f or fnc_tbl or prn_fnc_tb1 switch to print a complete list of functions supported on your platform 4 4 1 26 Operator precedence and associativity This page lists the ncap2 operators in order of precedence highest to lowest Their asso ciativity indicates in what order operators of equal precedence in an expression are applied Operator O Description Associativity Postfix Increment Decrement Right to Left Parentheses function call Method call Prefix Increment Decrement Right to Left Unary Plus Minus Logical Not Power of Operator Right to Left Multiply Divide Modulus Left To Right Addition Subtraction Left To Right Fortran style array clipping Left to Right Less than Less than or equal to Left to Right Greater than Greater than or equal to Equal to Not equal to Left to Right Logical AND Left to Right Logical OR Left to Right Ternary Operator Right to Left 3 ANSI C compilers are guaranteed to support double precision versions of these functions These func tions normally operate on netCDF variables of type NC_DOUBLE without having to perform intrinsic conversions For example ANSI compilers provide sin for the sine of C type double variables The ANSI standard does not require but many compilers
275. ember dismembers the input file fl in specified in the first argument and places the resulting files in the directory drc out specified by the second argument cat gt ncdismember lt lt EOF bin sh Purpose Dismember netCDF4 HDF5 hierarchical files CF check them Place each input file group in separate netCDF3 output file Described in NCO User Guide at http nco sf net nco html dismember Requirements NCO 4 3 x UNIX shell utilities awk grep sed Optional Decker CFchecker https bitbucket org mde_ cfchecker dk dk db H HF Usage ncdismember fl in drc out cf chk cf vrs opt where fl in is input file URL to dismember drc out is output directory CF compliance check is performed when optional third argument is not 0 dk dk dto Gt 15 Note that the 3 switch should appear after the G and g switches This is due to an artifact of the GPE implementation which we wish to remove in the future NCO 4 5 3 alpha03 User Guide Default checker is Decker s cfchecker installed locally Specify cf chk nerc for smallified uploads to NERC checker Optional fourth argument cf vrs is CF version to check Optional fifth argument opt passes straight through to ncks Arguments must not use shell expansion globbing NB ncdismember does not clean up output directory so user must chmod atx sh ncdismember Examples ncdismember nco data mdl 1 nc data zender tmp ncdismember http dust ess uci edu nco mdl 1
276. emory and swap space Since about 2004 many users have begun to produce or analyze files exceeding 2 GB in size These users should familiarize themselves with NCO s Large File Support LFS capabilities see Section 3 10 Large File Support page 42 The next section will increase your familiarity with NCO s memory requirements With this knowledge you may re design your data reduction approach to divide the problem into pieces solvable in memory limited situations If your local machine has problems working with large files try running NCO from a more powerful machine such as a network server If you get a memory related core dump e g Error exit core dumped on a GNU Linux system or the operation ends before the entire output file is written try increasing the process available memory with ulimit ulimit f unlimited 24 NCO 4 5 3 alpha03 User Guide This may solve constraints on clusters where sufficient hardware resources exist yet where system administrators felt it wise to prevent any individual user from consuming too much of resource Certain machine architectures e g Cray UNICOS have special commands which allow one to increase the amount of interactive memory On Cray systems try to increase the available memory with the ilimit command The speed of the NCO operators also depends on file size When processing large files the operators may appear to hang or do nothing for large periods of time In order to see wha
277. ends of course on the required precision Model results are often computed as NC_DOUBLE then archived as NC_FLOAT to save space This table summarizes the performance of lossless and lossy compression on two typical or at least random netCDF data files The files were taken from representative model simulated and satellite retrieved datasets Only floating point data were compressed No attempt was made to compress integer type variables as they occupy an insignificant fraction of every dataset The columns are Chapter 3 Shared Features 99 Type File type N3 for netCDF CLASSIC N4 for NETCDF4 N7 for NETCDF4 CLASSIC which comprises netCDF3 data types and structures with netCDF4 storage features like compression H4 for HDF4 and H5 for HDF5 N4 7 means results apply to both N4 and N7 filetypes LLC Type of lossless compression employed if any Bare numbers refer to the strength of the DEFLATE algorithm employed internally by netCDF4 HDF5 while numbers prefixed with B refer to the block size employed by the Burrows Wheeler algorithm in bzip2 PPC Number of significant digits retained by the precision preserving compression NSD algorithm Pck Y if the default ncpdq packing algorithm convert floating point types to NC_ SHORT was employed Size Resulting filesize in MB Compression ratio i e resulting filesize relative to original size in percent In some cases the original files is already losslessly compressed The compressi
278. ension time The following hyperslab operations produce identical results a June July August average of the data ncra d time 5 7 85 nc 85 JJA nc ncra F d time 6 8 85 nc 85 JJA nc Printing variable three dmn var in file in nc first with the C indexing convention then with Fortran indexing convention results in the following output formats ncks v three dmn var in nc lat 0 90 lev 0 1000 lon 0 180 three dmn var 0 0 ucks F v three_dmn_var in nc lon 1 0 lev 1 100 lat 1 90 three_dmn_var 1 0 3 15 Hyperslabs Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options d dim min max stride Long options dimension dim min max stride dmn dim min max stride A hyperslab is a subset of a variable s data The coordinates of a hyperslab are specified with the d dim min max stride short option or with the same arguments to the dimension or dmn long options At least one hyperslab argument min max or stride must be present The bounds of the hyperslab to be extracted are specified by the associated min and max values A half open range is specified by omitting either the min or max parameter The separating comma must be present to indicate the omission of one of these arguments The unspecified limit is interpreted as the maximum or minimum value in the unspecified direction A cross sect
279. ent variable GSL_RNG_TYPE If this variable isn t set then the default rng algorithm is gsl_rng_19937 The seed is set with the environ ment variable GSL_RNG_SEED The following wrapper functions in ncap2 provide information about the chosen algorithm gsl_rng_min the minimum value returned by the rng algorithm gsl_rng_max the maximum value returned by the rng algorithm Uniformly Distributed Random Numbers gsl_rng_get var_in This function returns var_in with integers from the chosen rng algorithm The min and max values depend uoon the chosen rng algorthm gsl_rng_uniform_int var_in This function returns var_in with random integers from 0 to n 1 The value n must be less than or equal to the maximum value of the chosen rng algorithm gsl_rng_uniform var_in This function returns var_in with double precision numbers in the range 0 0 1 The range includes 0 0 and excludes 1 0 gsl_rng_uniform_pos var_in This function returns var_in with double precision numbers in the range 0 0 1 excluding both 0 0 and 1 0 Below are examples of gsl_rng_get and gs1_rng_uniform_int in action export GSL_RNG_TYPE ranlux export GSL_RNG_SEED 10 ncap2 v 0 s al1 time 0 a2 gsl_rng_get a1 in nc foo nc 10 random numbers from the range 0 16777215 a2 9056646 12776696 1011656 13354708 5139066 1388751 11163902 7730127 15531 ncap2 v 0 s ai time 21 a2 gs1l rng uniform int ai sort in nc foo nc 10
280. ente 02 cece cece eee eee eee 241 performance 11 18 26 27 85 183 231 Dra tras Belle Stee as 17 23 186 permute dimensions 2 eee ee 217 permute A ay tee debe ae ten 139 Peter Campbell esc uei Rb meis 242 PEOC aeae ne steel tanec d toU mE E dander pte e da 8 PECO sac wales pida aoa 8 PGI on A a Nie Slr eee ce NU I 8 philosophyz ere rk b v EIER 17 PIPES ian E A VASE ENG NS 22 pileta sollen get 4 gu ls e PA EVEIZ A AS 86 pile gp2d ineen lu M IRMeUEE ELM 86 PLE s veedeaverev pce de Dx Deve pua 86 Qlicsrid eese mde x ds 86 plesxpl issliot jh tee ex MEDIAS 86 PLCLESt iclllsiitliefig eeMpi veupBRiBeSER ES 86 portability sd ue eni re R Ne do 8 positional arguments 0 00000 32 POSIX t SiN eas we Melua otis oc au teas eliotn es 29 46 DOWN stds seeder denne 178 POWER o S e IEEE PO 178 power function muaa ees 178 RPC eco tha eee ds nan vei ken tens pne E Sakae 92 prd ees nas wey lees heels peng ibd ep pec 88 precision 4evdbe ARIES PII Ree EIE 180 preprocessor tokens 2 0 e seen eee 10 presentations lin ved ele ier d 7 A pest deri 143 printf eke Go e e aa ILE ST 8 printf Qaei tene eec ea 186 209 214 printing files contents 2 00 203 printing variables 0 00 203 Processor voit snm eb pex cnn E ER 226 228 Processot COM wiicedeees tia ride bp 30 promotion iii PGs RIT UIS 113 115 179 Proposals cies cer rkudjen
281. enting fixed length strings ncks s Ac v fl nm arr in nc Using the s format code on strings which are not NUL terminated and thus not technically strings is likely to result in a core dump Create netCDF out nc containing all variables and any associated coordinates except variable time from netCDF in nc ncks x v time in nc out nc As a special case of this consider how to remove a variable such as time bounds that is identified in a CF Convention see Section 3 41 CF Conventions page 124 compliant ancillary variables bounds climatology or coordinates attribute NCO subsetting assumes the user wants all ancillary variables axes bounds and coordinates associated with all extracted variables see Section 3 12 Subsetting Coordinate Variables page 47 Hence to exclude a ancillary variables bounds climatology or coordinates variable while retaining the parent variable here time one must use the C switch ncks C x v time bounds in nc out nc The C switch tells the operator NOT to necessarily include all the CF ancillary vari ables axes bounds and coordinates Hence the output file will contain time and not time bounds 216 NCO 4 5 3 alpha03 User Guide Extract variables time and pressure from netCDF in nc If out nc does not exist it will be created Otherwise the you will be prompted whether to append to or to overwrite out nc ncks v time pressure in nc out nc ncks C v time p
282. ents ncrcat Chapter 2 Operator Strategies 25 requires MS lt VR ncecat requires MS lt VT ncra requires MS 2FR VR nces requires MS 2FT VT ncbo requires MS lt 3VT both input variables and the output variable ncflint requires MS lt 3VT both input variables and the output variable ncpdq requires MS lt 2VT one input variable and the output variable ncwa requires MS lt 8VT see below Note that only variables that are processed e g averaged concatenated or differenced contribute to MS Variables which do not appear in the output file see Section 3 11 Subsetting Files page 43 are never read and contribute nothing to the memory requirements Further note that some operators perform internal type promotion on some variables prior to arithmetic see Section 3 36 Type Conversion page 113 For example ncra and nces both promote integer types to double precision floating point prior to arithmetic then perform the arithmetic then demote back to the original integer type after arithmetic This preserves the on disk storage type while obtaining the accuracy advantages of floating point arithmetic Since version 4 3 6 released in September 2013 NCO also by default converts single precision floating point to double precision prior to arithmetic which in curs the same RAM penalty Hence the sustained memory required for integer variables and single precision floats are two or four times their on disk unco
283. ers unsigned 64 bit integers floats or doubles use var out array 1s val ncr dmn nm NC SHORT array var out array 1 val ncr dmn nm NC INT array var out array iull val ncr dmn nm NC UINT64 array var out array 1f val ncr dmn nm NC FLOAT array var out array 1 val ncr dmn nm NC DOUBLE array Once the associated dimensions have been defined the start and increment arguments may be supplied as values mathmatical expressions or variables var out array 1 1 time 1 2 3 4 5 6 7 8 9 10 var out array 1 2 2 one time 1 2 3 4 5 6 7 8 9 10 var out array 1 2 three dmn rec var 1 3 5 155 157 159 Hyperslabs in ncap2 are more limited than hyperslabs with the other NCO operators ncap2 does not understand the shell command line syntax used to specify multi slabs wrapped co ordinates negative stride or coordinate value limits However with a bit of syntactic magic they are all are possible ncap2 accepts in fact it requires N hyperslab arguments for a variable of rank N vari argi arg2 argN where each hyperslab argument is of the form start end stride and the arguments for different dimensions are separated by commas If start is omitted it defaults to zero If end is omitted it defaults to dimension size minus one If stride is omitted it defaults to one If a single value is present then it is assumed that that dimension collapses to a single value i e a cross section The number of hype
284. erslab arguments hence this is redundant but will count only once if an arithmetic operation is being performed This example uses index based not coordinate based multislabs because the stride option Chapter 3 Shared Features 65 only supports index based hyper slabbing See Section 3 16 Stride page 60 for more information on the stride option Multislabs are more efficient than the alternative of sequentially performing hyperslab operations and concatenating the results This is because NCO employs a novel multislab algorithm to minimize the number of I O operations when retrieving irregularly spaced data from disk The NCO multislab algorithm retrieves each element from disk once and only once Thus users may take some shortcuts in specifying multislabs and the algorithm will obtain the intended values Specifying redundant ranges is not encouraged but may be useful on occasion and will not result in unintended consequences Suppose the Q variable contains three dimensional arrays of distinct chemical con stituents in no particular order We are interested in the NOy species in a certain geographic range Say that NO NO2 and N205 are elements 0 1 and 5 of the species dimension of Q The multislab specification might look something like ncks d species 0 1 d species 5 d lon 0 4 d lon 2 9 2 in nc out nc Multislabs are powerful because they may be specified for every dimension at the same time Thus multislabs obsolete the ne
285. es modifying 0 270 core dump uod en eens dig tices 15 23 215 COS ls sailed oh EH tue CES 178 COSA Rie advan sh den cies ohm 178 cosine function essor 0 6c cece cece eee eee 178 COVATIANCE s 2 the ties des e sli A Le 177 Pirri dosed elses ita send isles a as 44 COPAY tvs th easels che edie SM aids sett hoes oad 8 23 CSIiiahsbeiuewsUIPQew de Mat bee 47 Csh shell ii add 211 cubed sphere grid ooooooococcococrnrcrn ccoo 74 CA e 8 CNI ir ia ais 10 daily data ine cantata 245 246 Daniel Baumann eee eee eee 242 Daniel Want i iaca e A E EER 241 DAP uidet rete EDD ta E EEEE 35 data access protocol 0 cece eee ee eee 35 data saleby 4 tt pase anges 17 230 data MISS zccni A ta 74 84 183 dates toa ds 124 date writtenj coo eines ger EP RR 125 e A e a O S 124 dbezIylz 14 ds dd 16 24 28 DORA edie Pe Baga rss 243 Debido ace ad 14 debug level sees 16 24 debugging s ssssrerrrrreresrerrero 15 24 28 Decimal Significant Digits 92 DEG deese a a artes e ooo eR nabs 8 defdimC icu sao 135 defining dimensions in ncap2 135 DEFLATE eo Li emei iaai a ea E eres 95 MMA LICH is e eek eene err ud 12 101 208 degenerate dimension 88 111 178 191 199 200 224 225 235 238 degrees east isa ee Doce eek a nado 75 degrees MOTA Mic ta ern 75 delete groups 0ooococcccccccococcncccco 49 delete miss sese 145
286. es are entirely readable by Chapter 3 Shared Features 39 applications that use only the netCDF3 API though the applications must be linked with the netCDF4 library NCO must be built with netCDF4 to write files in the new NETCDF4 and NETCDFA CLASSIC formats and to read files in these formats Datasets in the default CLASSIC or the newer 64BIT formats have maximum backwards compatibility with older applications NCO has deep support for NETCDF4 formats If performance or disk space as important as backwards compatibility then use NETCDF4 CLASSIC instead of CLASSIC format files As of 2014 NCO support for the NETCDF4 format is nearly complete and the most powerful and disk RAM efficient workflows will utilize this format As mentioned above all operators write use the input file format for output files unless told otherwise Toggling the short option 6 or the long option 6 or 64bit or their key value equivalent f1_fmt 64bit produces the netCDF3 64 bit offset format named 64BIT NCO must be built with netCDF 3 6 or higher to produce a 64BIT file Using the 4 switch or its long option equivalents 4 or netcdf4 or setting its key value equivalent f1_fmt netcdf4 produces a NETCDF4 file i e with all supported HDF5 features Using the 7 switch or its long option equivalent 7 or setting its key value equivalent f1 fmt netcdf4 classic produces a NETCDF4 CLASSIC file
287. es of the months June July August and December January February respectively ncra n 3 2 1 85 06 nc 85 0608 nc The n option is a backward compatible superset of the NINTAP option from the NCAR CCM Processor The CCM Processor was custom written Fortran code maintained for many years by Lawrence Buja at NCAR and phased out in the late 1990s NCO stole some ideas like NINTAP functionality from CCM Processor capabilities 32 NCO 4 5 3 alpha03 User Guide ncra n 3 2 1 12 85 12 nc 85 1202 nc ncra n 3 2 1 12 1 85 12 nc 85 1202 nc The first example shows that three arguments to the n option suffice to specify con secutive months 06 07 08 which do not wrap back to a minimum value The second example shows how to use the optional fourth and fifth elements of the n option to specify a wrap value The fourth argument to n when present specifies the maximum integer value of numeric suffix In the example the maximum value is 12 and will be formatted as 12 in the filename string The fifth argument to n when present specifies the minimum integer value of numeric suffix The default minimum filename suffix is 1 which is format ted as 01 in this case Thus the second and third examples have the same effect that is they automatically generate in order the filenames 85 12 nc 85 01 nc and 85 02 nc as input to NCO As of NCO version 4 5 2 September 2015 NCO supports an optional sixth argu
288. ese in bashrc ncattget att nm var nm fl nm What attributes does variable have function ncattget ncks M m 3 grep E i 12 attribute 0 9 1 cut ncunits att val fl nm Which variables have given units function ncunits ncks M m 2 grep E i attribute 0 9 units 1 cu ncavg var nm fl nm What is mean of variable function ncavg ncwa y avg 0 C v 11 2 foo nc ncks H C v 11 foo nc ncavg var nm fl nm What is mean of variable function ncavg ncap2 0 C v s foo 1 avgQ print foo 12 foo nc cut f ncdmnlist f1 nm What dimensions are in file ncdmnlist ncks cdl m 1 cut d f 1 cut d s f 1 ncdmnsz dmn nm fl nm What is dimension size function ncdmnsz ncks m M 2 grep E i 1 size cut f 7 d u nclist f1 nm What variables are in file function nclist ncks m 11 grep E type cut f 1 d sed s so ncmax var nm fl nm What is maximum of variable Chapter 4 Reference Manual 213 function ncmax ncwa y max 0 C v 11 2 foo nc ncks H C v 11 foo nc ncmax var nm f1 nm What is maximum of variable function ncmax ncap2 0 C v s foo 1 max print foo 12 foo nc cut f ncmdn var nm f1 nm What is median of variable function ncmdn ncap2 0 C v s foo gsl stats median from sorted data 1 sort
289. est thousandth or hundred respectively Setting var to default has the special meaning of applying the associated NSD or DSD algorithm to all floating point variables except coordinate variables Variables not affected by default include integer and non numeric atomic types coordinates and variables men tioned in the bounds climatology or coordinates attribute of any variable NCO applies PPC to coordinate variables only if those variables are explicitly specified i e not with the default prc mechanism NCO applies PPC to integer type variables only if those variables are explicitly specified i e not with the default prc and only if the DSD algorithm is invoked with a negative prc To prevent PPC from applying to certain non coordinate vari ables e g gridcell area or gaussian weight explicitly specify a precision exceeding 7 for NC FLOAT or 15 for NC DOUBLE for those variables Since these are the maximum representable precisions in decimal digits NCO turns off PPC i e does nothing when more precision is requested The time penalty for compressing and uncompressing data varies according to the al gorithm The Number of Significant Digit NSD algorithm quantizes by bitmasking and employs no floating point math The Decimal Significant Digit DSD algorithm quantizes by rounding which does require floating point math Hence NSD is likely faster than DSD though the difference has not been measured NSD creates a bi
290. etion of the operation This is accomplished by performing all operations in a temporary copy of output file The name of the temporary output file is constructed by appending pid lt process ID gt lt operator name gt tmp to the user specified output file name When the operator completes its task with no fatal errors the temporary output file is moved to the user specified output file This imbues the pro cess with fault tolerance since fatal error e g disk space fills up affect only the temporary output file leaving the final output file not created if it did not already exist Note the con struction of a temporary output file uses more disk space than just overwriting existing files in place because there may be two copies of the same file on disk until the NCO operation successfully concludes and the temporary output file overwrites the existing output file 1 The ncrename and ncatted operators are exceptions to this rule See Section 4 11 ncrename netCDF Renamer page 230 18 NCO 4 5 3 alpha03 User Guide Also note this feature increases the execution time of the operator by approximately the time it takes to copy the output file Finally note this fault tolerant feature allows the output file to be the same as the input file without any danger of overlap Over time many power users have requested a way to turn off the fault tolerance safety feature of automatically creating a temporary file Often these users
291. expint_E1_scaled_e gsl_sf_expint E2 scaled e esl_sf_expint_En_scaled_e esl_sf_expint_Ei_e esl_sf_expint_Ei_scaled_e gsl sf Shi e gsl sf Chi e gsl sf expint 3 e gsl sf Si e gsl sf Ci e gsl sf atanint e gsl sf fermi dirac m1 e gsl sf fermi dirac 0 e gsl sf fermi dirac 1 e gsl sf fermi dirac 2 e gsl sf fermi dirac int e gsl sf fermi dirac mhalf e gsl sf fermi dirac half e gsl sf fermi dirac 3half e gsl sf fermi dirac inc 0 e gsl sf Ingamma e gsl sf Ingamma sgn e gsl sf gamma e esl_sf_gammastar_e gsl sf gammainv e gsl sf Ingamma complex e gsl sf taylorcoeff e gsl sf fact e esl_sf_doublefact_e esl_sf_Infact_e gsl_sf_Indoublefact_e gsl_sf_Inchoose_e esl_sf_choose_e esl_sf_Inpoch_e esl_sf_Inpoch_sgn e gsl_sf_poch_e gsl_sf_pochrel_e A ZA LA ARAS HA PE LA eU EO LAS AAA E PA ES A A PA AAA A A 2 ee EGER NCO 4 5 3 alpha03 User Guide gsl_sf_exprel dbl_expr gsl_sf_exprel_2 dbl_expr gsl_sf_exprel_n int_expr dbl_expr gsl_sf_exp_err dbl_expr dbl_expr gsl_sf_exp_err_e10 gsl_sf_exp_mult_err gsl_sf_exp_mult_err_e10 gsl_sf_expint_E1 dbl_expr gsl_sf_expint_E2 dbl_expr gsl_sf_expint_En int_expr dbl_expr gsl_sf_expint_E1_scaled dbl_expr gsl_sf_expint_E2_scaled dbl_expr gsl_sf_expint_En_scaled int_expr dbl_expr gsl_sf_expint_Ei dbl_expr gsl_sf_expint_Ei_scaled dbl_expr gsl_sf_Shi dbl_expr gsl_sf_Chi dbl_expr gsl_sf_expint_3 dbl_expr gsl_sf_Si dbl_expr gsl_sf_Ci dbl_expr esl_sf_atanint
292. fault Definition Pack unpacked variables re pack packed variables Alternate invocation ncpack pck plc key values all_new pck all new att Packing and not Re Packing Variables Definition Pack unpacked variables copy packed variables Alternate invocation none pck plc key values all_xst pck all xst att Re Packing Variables Definition Re pack packed variables copy unpacked variables Alternate invocation none pck plc key values xst_new pck xst new att Unpacking Definition Unpack packed variables copy unpacked variables Alternate invocation ncunpack 218 NCO 4 5 3 alpha03 User Guide pck plc key values upk unpack pck_upk Equivalent key values are fully interchangeable Multiple equivalent options are provided to satisfy disparate needs and tastes of NCO users working with scripts and from the command line Regardless of the packing policy selected ncpdq no longer as of NCO version 4 0 4 in October 2010 packs coordinate variables or the special variables weights and other grid properties described in Section 3 41 CF Conventions page 124 Prior ncpdq versions treated coordinate variables and grid properties no differently from other variables How ever coordinate variables are one dimensional so packing saves little space on large files and the resulting files are difficult for humans to read ncpdq will of course unpack co ordinate variables and weights for example
293. ficacy of create_ram is clear Placing the output file in RAM avoids thousands of disk writes It is not unreasonable to for NCO to process a million files like this in a few minutes However there is no substitute for benchmarking with real files A completely independent way to reduce time spent writing files is to refrain from writing temporary output files This is accomplished with the no tmp 1 switch see Section 2 3 Temporary Output Files page 17 3 34 Packed data Availability ncap2 ncbo nces ncflint ncpdq ncra ncwa Short options None Long options hdf_upk hdf_unpack The phrase packed data refers to data which are stored in the standard netCDF3 packing format which employs a lossy algorithm See Section 4 7 ncks netCDF Kitchen Sink page 203 for a description of deflation a lossless compression technique available with netCDF4 only Packed data may be deflated to save additional space Packing Algorithm Packing The standard netCDF packing algorithm described here http www unidata ucar edu software netcdf docs netcdf Attribute Conventions html produces data with the same dynamic range as the original but which requires no more than half 106 NCO 4 5 3 alpha03 User Guide the space to store Like all packing algorithms it is lossy The packed variable is stored usually as type NC SHORT with the two attributes required to unpack the variable scale factor and
294. file names In the example above the encoding n 5 2 1 along with the input file name 85 nc tells NCO to construct five 5 filenames identical to the template 85 nc except that the final two 2 digits are a numeric suffix to be incremented by one 1 for each successive file Currently filetype may be either be empty nc h5 cdf hdf hd5 or he5 If present these filetype suffixes and the preceding are ignored by NCO as it uses the n arguments to locate evaluate and compute the numeric suffix component of filenames Recently the n option has been extended to allow convenient specification of file names with circular characteristics This means it is now possible for NCO to auto matically generate filenames which increment regularly until a specified maximum value and then wrap back to begin again at a specified minimum value The corresponding lt n usage becomes more complex taking one or two additional arguments for a total of four or five respectively n file number digit number numeric increment numeric max numeric min where numeric max if present is the maximum integer value of nu meric_suffix and numeric min if present is the minimum integer value of numeric suffix Consider for example the problem of specifying non consecutive input files where the filename suffixes end with the month index In climate modeling it is common to create summertime and wintertime averages which contain the averag
295. fine a RAM variable simply prefix the variable name with an asterisk when the variable is declared initialized To delete a RAM variables and recover their memory use the ram delete O method To write a RAM variable to disk like a regular variable use ram writeO temp time 1at 10n 10 0 Cast temp_avg temp avg time Regular assign temp ram_delete Delete RAM variable temp_avg ram_write Write Variable to output Create and increment a RAM variable from one in Input one Create RAM variables from the variables three and four in Input Multiply three by 10 and add it to four fourt three 10 three 30 four 34 4 1 13 Where statement A where combines the definition and application of a mask all in one go and can lead to succinct code The full syntax of a where O statement is as follows Single assign the elsewhere block is optional where mask vari expr1 elsewhere varl expr2 Multiple assigns where mask vari expri var2 expr2 Jelsewheret vari expr3 var2 expr4 var3 expro e The only expression allowed in the predicate of a where is assign i e var expr This assign differs from a regular ncap2 assign The LHS var must already exist in Input or Output The RHS expression must evaluate to a scalar or a variable attribute of the same size as the LHS variable e Consider when both the LHS and RHS are variables For every element where mask c
296. following GSL prototype int gsl sf bessel Jn array int nmin int nmax double x double result array bi lon double x 0 5 status gsl_sf_bessel_Jn_array 1 4 x amp b1 print status b120 24226 0 0306 0 00256 0 00016 This calculates the Bessel function of x 0 5 for n 1 to 4 The first three arguments are scalar values If a non scalar variable is supplied as an argument then only the first value is used The final argument is the variable where the results are stored NB the amp indicates Chapter 4 Reference Manual 163 this is a call by reference This final argument must be of type double and must be of least size nmax nmin 1 If either of these conditions is not met then then the function returns an error message The function wrapper returns a status flag Zero indicates success Consider another array function int gsl sf legendre Pl array int lmax double x double result array ai time double x 0 3 status gsl sf legendre P1 array al size 1 x al print status This call calculates P_1 0 3 for 1 0 9 Note that x 1 otherwise there will be a domain error See the GSL documentation for more details The GSL functions implemented in NCO are listed in the table below This table is correct for GSL version 1 10 To see what functions are available on your build run the command ncap2 f grep gsl To see this table along with the GSL C function prototypes look at the spreadsheet doc nco
297. g IDL Yorick Matlab NCL Perl Python and lower level compiled languages e g C Fortran can always perform any task performed by NCO but often with more overhead NCO on the other hand is limited to a much smaller set of arithmetic and metadata operations than these full blown languages Another goal has been to implement enough command line switches so that frequently used sequences of these operators can be executed from a shell script or batch file Finally NCO was written to consume the absolute minimum amount of system memory required to perform a given job The arithmetic operators are extremely efficient their exact memory usage is detailed in Section 2 9 Memory Requirements page 24 2 2 Climate Model Paradigm NCO was developed at NCAR to aid analysis and manipulation of datasets produced by General Circulation Models GCMs GCM datasets share many features with other gridded scientific datasets and so provide a useful paradigm for the explication of the NCO operator set Examples in this manual use GCM paradigm because latitude longitude time temperature and other fields related to our natural environment are as easy to visualize for the layman as the expert 2 3 Temporary Output Files NCO operators are designed to be reasonably fault tolerant so that a system failure or user abort of the operation e g with C c does not cause loss of data The user specified output file is only created upon successful compl
298. g Compression PPC algorithms and since standard terminology for precision is remarkably imprecise so is our nomenclature The operational definition of significant digit in our precision preserving algorithms is that the exact value before rounding or quantization is within one half the value of the decimal place occupied by the Least Significant Digit LSD of the rounded value For example the value pi 3 14 correctly represents the exact mathematical constant pi to three significant digits because the LSD of the rounded value i e 4 is in the one hundredths digit place and the difference between the exact value and the rounded value is less than one half of one one hundredth i e 3 14159265358979323844 3 14 0 00159 0 005 One PPC algorithm preserves the specified total Number of Signifcant Digits NSD of the value For example there is only one significant digit in the weight of most eight hundred pound gorillas that you will encounter i e so nsd 1 This is the most straightforward measure of precision and thus NSD is the default PPC algorithm Chapter 3 Shared Features 93 The other PPC algorithm preserves the number of Decimal Significant Digits DSD i e the number of significant digits following positive by convention or preceding negative the decimal point For example 0 008 and 800 have respectively three and negative two digits digits following the decimal point corresponding d
299. g_2F0 dbl_expr dbl_expr dbl_expr gsl_sf_laguerre_1 dbl_expr dbl_expr gsl_sf_laguerre_2 dbl_expr dbl_expr gsl_sf_laguerre_3 dbl_expr dbl_expr gsl_sf_laguerre_n int_expr dbl_expr dbl_expr gsl sf lambert WO dbl expr gsl sf lambert Wml dbl expr gsl sf legendre Pl int expr dbl expr status gsl sf legendre Pl array int double amp var out esl_sf legendre Pl deriv array gsl_sf_legendre_P1 dbl_expr gsl_sf_legendre_P2 dbl_expr gsl_sf_legendre_P3 dbl_expr gsl_sf_legendre_QO dbl_expr gsl_sf_legendre_Q1 dbl_expr gsl_sf_legendre_QI int_expr dbl_expr gsl_sf_legendre_Plm int_expr int_expr dbl_expr status gsl sf legendre Plm array int int double amp var_out gsl sf legendre PIm deriv array gsl_sf_legendre_sphPlm int_expr int_expr dbl_expr status gsl sf legendre sphPlm array int int double amp var out gsl sf legendre sphPlIm deriv array gsl sf legendre array size esl_sf_conicalP_half dbl_expr dbl_expr 168 gsl_sf_conicalP_mhalf_e esl_sf_conicalP_0_e esl_sf_conicalP_1_e gsl_sf_conicalP_sph_reg_e gsl_sf_conicalP _cyl_reg_e gsl_sf_legendre_H3d_0_e gsl sf legendre H3d 1 e gsl_sf_legendre_H3d_e gsl_sf_legendre_H3d_array gsl sf legendre array size gsl sf log e gsl sf log abs e gsl sf complex log e gsl sf log 1plusx e gsl sf log 1plusx mx e gsl sf mathieu a array gsl sf mathieu b array gsl sf mathieu a gsl sf mathieu b gsl sf mathieu a coeff gsl sf mathieu b
300. grid generation because so many other parameters spacing centering are necessary to disambiuate it 72 NCO 4 5 3 alpha03 User Guide Hence Cap grids have the unusual property that the poles are labeled as being both the centers and the outer interfaces of all polar gridcells Second Cap grids are uniform in angle except at the poles where the latitudes span half the meridional range of the rest of the gridcells Even though in the host dynamical model the Cap grid polar points are melded into caps uniform in angle with the rest of the grid the disk representation on disk is not uniform Nevertheless some call the Cap grid a uniform angle grid because the information contained at the poles is aggregated in memory to span twice the range of a single polar gridcell which has half the normal width The NCL uses the term Fixed grid for a Cap grid The Fixed terminology seems broken Finally Gaussian grids are the Cartesian representation of global spectral trans form models Gaussian grids do not have points at the poles and must have an even number of latitudes All three latitude grid type supported by NCO Uniform Cap and Gaussian are Regular grids in that they are monotonic The lon typ options for global grids are grn_ctr and 180_ctr for the first gridcell centered at Greenwich or 180 degrees respecitvely And grn_wst and 180_wst for Greenwich or 180 degress lying on the western edge of the fir
301. grpL 1 gagl glb h hdf hdr pad nbr L dfl 1vl 1 path M md5 digest mrd n loop no tmp f1 0 o output file p path ppc R r ram a11 t thr nbr u ulm nm unn v var X x input files output file DESCRIPTION ncecat aggregates an arbitrary number of input files into a single output file using using one of two methods Record AGgregation RAG the traditional method employed on flat netCDF3 files and still the default method stores input files as consecutive records in the output file Group AGgregation GAG stores input files as top level groups in the netCDF4 output file Record Aggregation RAG makes numerous assumptions about the structure of input files whereas Group Aggregation GAG makes none Both methods are described in detail below Since ncecat aggregates all the contents of the input files it can easily produce large output files so it is often helpful to invoke subsetting simultaneously see Section 3 11 Subsetting Files page 43 RAG makes each variable except coordinate variables in each input file into a single record of the same variable in the output file Coordinate variables are not concatenated they are instead simply copied from the first input file to the output file All input files must contain all extracted variables or else there would be gaps in the output file A new record dimension is the glue which
302. gsl ods GSL NAME gsl sf airy Ai e gsl sf airy Bi e gsl sf airy Ai scaled e gsl sf airy Bi scaled e gsl sf airy Ai deriv e gsl sf airy Bi deriv e gsl sf airy Ai deriv scaled e gsl sf airy Bi deriv scaled e gsl sf airy zero Ai e gsl sf airy zero Bi e gsl sf airy zero Ai deriv e gsl sf airy zero Bi deriv e gsl sf bessel JO e gsl sf bessel J1 e gsl sf bessel Jn e gsl sf bessel Jn array gsl sf bessel Y0 e esl_sf_bessel_Y1_e gsl sf bessel Yn e gsl sf bessel Yn array esl_sf_bessel_10_e esl_sf_bessel_11_e gsl sf bessel In e gsl sf bessel In array gsl sf bessel I0 scaled e gsl sf bessel I1 scaled e ae ae AA AS A EG reed AA ES EG ee EA AA NCAP FUNCTION CALL gsl_sf_airy_Ai dbl_expr gsl_sf_airy_Bi dbl_expr gsl_sf_airy_Ai_scaled dbl_expr gsl_sf_airy_Bi_scaled dbl_expr esl_sf_airy_Ai_deriv dbl_expr esl_sf_airy_Bi_deriv dbl_expr gsl_sf_airy_Ai_deriv_scaled dbl_expr gsl_sf_airy_Bi_deriv_scaled dbl_expr gsl_sf_airy_zero_Ai uint_expr gsl_sf_airy_zero_Bi uint_expr esl_sf_airy_zero_Ai_deriv uint_expr gsl_sf_airy_zero_Bi_deriv uint_expr gsl_sf_bessel_JO dbl_expr gsl_sf_bessel_J1 dbl_expr gsl_sf_bessel_Jn int_expr dbl_expr status gsl_sf_bessel_Jn_array int int double amp var_out gsl_sf_bessel_Y0 dbl_expr gsl_sf_bessel_Y1 dbl_expr gsl_sf_bessel_Yn int_expr dbl_expr esl_sf_bessel_Yn_array gsl_sf_bessel_I0 dbl_expr gsl_sf_bessel_I1 dbl_expr gsl_sf_bessel_I
303. gt dev null echo gt amp 2 Local cfchecker command not found will fi cf chk Chapter 3 Shared Features 53 Obtain group list grp_lst ncks cdl m fl_in grep group awk 1 2 3 sub 0 IFS n Change Internal Field Separator from lt Space gt lt Tab gt lt Newline gt to lt Newline gt for grp in in grp_lst do Replace slashes by dots for output group filenames grp out echo grp_in sed s sed s M g if grp_out then grp_out root fi Tell older NCO netCDF if HDF4 with hdf4 switch signified by hdf HDF suffix hdf4 echo f1 in awk fif match tolower 1 hdf hdf4 hdf4 print hd Flatten to netCDF3 anchor no history no temporary file padding HDF4 flag o cmd ncks 0 3 G g grp_in h no tmp fl hdr pad 40 hdf4 opt f1 Use eval in case opt contains multiple arguments separated by whitespace eval cmd if chk_dck y then Decker checker needs Conventions lt 1 6 no_bck_sls echo drc_out grp_out sed s g ncatted h a Conventions global o c CF cf_vrs no_bck_sls nc else chk_dck echo drc_out grp_out nc fi chk_dck done if chk_dck y then echo Decker CFchecker reports CF compliance of each group in flat netCDF3 format cfchecker c cf vrs nc fi if chk nrc y then Smallification and NERC upload from qdcf s
304. ha03 User Guide 4 4 nces netCDF Ensemble Statistics SYNTAX nces 3 4 6 7 A C c cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stridel1 F G gpe_dsc g grp glb h hdf hdr_pad nbr L dfl 1vl1 1 path n loop no tmp f1 nsm_fllgrp nsm sfx sfx 0 o output file p path ppc R r ram_all rth db1llflt t thr nbr unn v var X x y op typl input files output file DESCRIPTION nces performs gridpoint statistics on variables across an arbitrary number an ensemble of input files and or of input groups within each file Each file or group receives an equal weight nces was formerly until NCO version 4 3 9 released December 2013 known as ncea netCDF Ensemble Averager For example nces will average a set of files or groups weighting each file or group evenly This is distinct from ncra which performs statistics only over the record dimension s e g time and weights each record in each record dimension evenly The file or group is the logical unit of organization for the results of many scientific studies Often one wishes to generate a file or group which is the statistical product e g average of many separate files or groups This may be to reduce statistical noise by combining the results of a large number of experiments or it may simply
305. haracter 0 2000 49 array delimiters 0 2 00 131 A etre ee be dea e 178 wildcard character lssesssssssus 46 sGh nkSlZz88 7 v ta wee ete aa 205 JDeflateLevel lesse ee 205 EndiannesSS vua vee EUER e RENE 205 FillValue 74 84 105 183 201 219 233 Fletcher isk ins et wad ae Vide ghee 205 JFEOImab Nole iste eee bee MEINES US 205 ZNOREILLiI Di ox seer etd a eonia NS ED teed oy 205 Ziohuffle s 3 4 00D ueeWy cen amp waned ale anu di Weed 205 390URCE FORMAT ccoo lic 205 Sl AA A vs vides eles seeds 205 wi backslash a cene s oce eR CERA 186 protected double quote 186 V protected end quote 005 186 X protected question mark 186 MV ASCII N backslash n nnns 186 protected backslash sssseuse 186 Va ASCII BEL bell 0 0 0 0200 186 NCO 4 5 3 alpha03 User Guide b ASCII BS backspace oooooocccoccccoc 186 f ASCII FF formfeed 005 186 n ASCII LF linefeed 0 5005 186 n linefeed 0 cece cence eee neces 214 r ASCII CR carriage return 186 t ASCII HT horizontal tab 186 Nt horizontal tab sss 214 v ASCII VT vertical tab 186 wildcard character 0 0 0 008 46 0 O NUD seenen in tae ee aes ela ee
306. he most important variables see Section 2 9 Memory Requirements page 24 14 Intersection mode can also be explicitly invoked with the nsx or intersection switches These switches are supplied for clarity and consistency and do absolutely nothing since intersection mode is the default 46 NCO 4 5 3 alpha03 User Guide ncks in nc out nc Extract all groups and variables ncks v scl Extract variable scl from all groups ncks g gl Extract group g1 and descendents ncks x g g1 Extract all groups except gi and descendents ncks g g2 g3 v scl Extract scl from groups g2 and g3 Overwriting and appending work as expected Replace scl in group g2 in out nc with scl from group g2 from in nc ncks A g g2 v scl in nc out nc Due to its special capabilities ncap2 interprets the v switch differently see Section 4 1 ncap2 netCDF Arithmetic Processor page 130 For ncap2 the v switch takes no arguments and indicates that only user defined variables should be output ncap2 neither accepts nor understands the x and g switches Regular expressions the syntax that NCO use pattern match object names in netCDF file against user requests The user can select all variables beginning with the string DST from an input file by supplying the regular expression DST to the v switch i e v DST The meta characters used to express pattern matching operations are 4 43 If the regul
307. he other operations is familiar The masked weighted average of a variable x can be generally represented as i N bee int BTWWUT i N 2a TUR where Z is the j th element of the output hyperslab x is the tth element of the input hyperslab u is 1 unless x equals the missing value m is 1 unless x is masked and w is the weight This formiddable looking formula represents a simple weighted average whose bells and whistles are all explained below It is not too early to note however that when Hi mj w 1 the generic averaging expression above reduces to a simple arithmetic Chapter 3 Shared Features 109 average Furthermore m w 1 for all operators except ncwa These variables are included in the discussion below for completeness and for possible future use in other operators The size J of the output hyperslab for a given variable is the product of all the dimensions of the input variable which are not averaged over The size N of the input hyperslab contributing to each z is simply the product of the sizes of all dimensions which are averaged over i e dimensions specified with a Thus N is the number of input elements which potentially contribute to each output element An input element x contributes to the output element x except in two conditions 1 x equals the missing value see Section 3 27 Missing Values page 84 for the variable 2 x is located at a point where the mask condition see
308. hen creating new variables The intrinsic function change miss see below is typically used to edit values of existing variables change miss expr Sets or changes any pre existing missing value attribute and missing data values to expr NB This is an expensive function since all values must be examined Use this function when changing missing values for pre existing variables get miss Returns the missing value of a variable If the variable exists in Input and Output then the missing value of the variable in Output is returned If the variable has no missing value then an error is returned delete miss Delete the missing value associated with a variable number miss Count the number of missing values a variable contains th three dmn var dbl th change miss 1e10d Set values less than 0 or greater than 50 to missing value where th lt 0 0 th gt 50 0 th th get miss Another example new time lat lon 1 0 new set_miss 997 0 Extract only elements divisible by 3 where three_dmn_var_db1 3 0 new three_dmn_var_dbl elsewhere new new get_miss Print missing value and variable summary mss_val_nbr three_dmn_var_dbl number_miss print three_dmn_var_db1 _FillValue print Number of missing values in three_dmn_var_dbl print mss_val_nbr d print three dmn var db1 4 1 11 Methods and functions The convention within this document is that methods can be used a
309. hese are the chunking policy chunking map chunksize and minimum chunksize The chunking policy determines which variables to chunk and the chunking map determines how with what exact sizes to chunk those variables These are high level mechanisms that apply to an entire file and all variables and dimensions The chunksize option allows per dimension specification of sizes that will override the selected or default chunking map The var_sz parameter is the minimum size in bytes not elements of variables to chunk This threshold is intended to restrict use of chunking to variables for which it is efficient By default this minimum variable size for chunking is twice the system blocksize when available and is 8192 bytes otherwise Users may set this to any value with the cnk_min switch The chunking implementation is similar to a hybrid of the ncpdq packing policies see Section 4 8 ncpdq netCDF Permute Dimensions Quickly page 217 and hyperslab specifi cations see Section 3 15 Hyperslabs page 58 Each aspect is intended to have a sensible default so that many users only need to set one switch to obtain sensible chunking Power users can tune chunking with the three switches in tandem to obtain optimal performance By default NCO preserves the chunking characteristics of the input file in the output file In other words preserving chunking requires no switches or user intervention Users specify the desired chunking pol
310. hin each group This ordering here is used by default in CDL mode printing and may be selected for traditional mode printing with 5 The switch for invocation may change to something more descriptive in the future Those familiar with netCDF mechanics might wish to know what is happening here ncks does not attempt to redefine the variable in output file to match its definition in input file ncks merely copies the values of the variable and its coordinate dimensions if any from input file to output file Chapter 4 Reference Manual 205 a Do not alphabetize extracted fields By default the specified output variables are extracted printed and written to disk in alphabetical order This tends to make long output lists easier to search for particular variables Specifying a results in the variables being extracted printed and written to disk in the order in which they were saved in the input file Thus a retains the original ordering of the variables Also abc and alphabetize b file Activate native machine binary output writing to binary file file Also f1_bnr and binary file Writing packed variables in binary format is not supported Metadata is never output to the binary file Examine the netCDF output file to see the variables in the binary file Use the C switch if necessary to avoid wanting unwanted coordinates to the binary file 4 ncks 0 v one dmn rec var b bnr da
311. hree lossy compression algorithms are Packing see Section 3 34 Packed data page 105 and two precision preserving algorithms Packing quantizes data of a higher precision type into a lower precision type often NC SHORT that thus stores a fewer though constant number of bytes per value Packed data almost unpacks into a much smaller dynamic range than the floating point data can represent The type conversion and re duced dynamic range of the data allows packing to eliminate bits typically used to store an exponent thus improving its packing efficiency Packed data also can also be deflated for additional space savings A limitation of packing is that unpacking data stored as integers into the linear range defined by scale factor and add offset rapidly loses precision outside of a narrow range of floating point values Variables packed as NC SHORT for example can represent only about 64000 discrete values in the range 32768 x scaleyactor add f f set to 32767 x scale ractor add f f set The precision of packed data equals the value of scale factor and scale factor is usually chosen to span the range of valid data not to represent the intrinsic precision of the variable In other words the precision of packed data cannot be specified in advance because it depends on the range of values to quantize NCO implemented the final two lossy compression algorithms in version 4 4 8 Febru ary 2015 These are both Precision Preservin
312. iable wildcarding Numerous hacks ncap2 language Rorik Peterson Original autotools build support Long command line options Original UDUnits support Debianization Numerous bug fixes Joe Hamman Python bindings NCOpy Daniel Wang Script Workflow Analysis for MultiProcessing SWAMP RPM support Harry Mangalam Benchmarking OPeNDAP configuration T Happy users have sent me a few gifts though This includes a box of imported chocolate Mmm Appreciation and gifts are definitely better than money Naturally I m too lazy to split and send gifts to the other developers However unlike some NCO developers I have a steady real job My intent is to split monetary donations among the active developers and to send them their shares via PayPal 242 NCO 4 5 3 alpha03 User Guide Pedro Vicente Windows Visual Studio support netCDF4 groups Russ Rew Advice on NCO structural algorithms Brian Mays Original packaging for Debian GNU Linux nroff man pages George Shapovalov Packaging for Gentoo GNU Linux Bill Kocik Memory management Len Makin NEC SX architecture support Jim Edwards AIX architecture support Juliana Rew Compatibility with large PIDs Karen Schuchardt Auxiliary coordinate support Gayathri Venkitachalam MPI implementation Scott Capps Large work load testing Peter Campbell Martin Dix Mark Flanner Markus Liebig Keith Lindsay Mike Page Martin Schmidt Michael Schulz Lori Sentman Rich
313. iables rlm LImon Realm xpt historical Experiment could be more for var_id in 0 1 do Loop over two variables Names of all the models 1s get file names cut get model names sort uniq remove duplicates awk print mdl set 1s drc_in var var_id _ rlm _ _ xpt 0 _ nc cut d _ f 3 sort uniq c awk print 2 Number of models echo print contents wc count mdl nbr echo mdl_set wc w echo echo There are mdl nbr models for var var_id for mdl in mdl set do Loop over models i Names of all the ensemble members nsm set ls drc_in var var_id _ rlm _ md1 _ xpt 0 _ nc cut d _ f 5 sort uniq c awk print 2 Number of ensemble members in each model nsm_nbr echo nsm_set wc w echo echo Model mdl includes nsm nbr ensemble member s Chapter 7 CMIP5 Example 249 echo nsm_set for nsm in nsm set do Loop over ensemble members Number of files in this ensemble member fl nbr 1s drc_in var var_id _ rlm _ md1 _ xpt 0 _ nsm _ nc wc w If there is only 1 file continue to next loop if fl_nbr le 1 then echo There is only 1 file in nsm continue fi echo There are fl_nbr files in nsm Starting date of data sed the name of the first file
314. iables possible The HDF5 storge layer requires that record variables i e variables that contain at least one record dimension must be chunked Also variables that are compressed or use checksums must be chunked Such variables cannot be unchunked Alternate invocation ncunchunk cnk plc key values uck cnk_uck plc_uck unchunk Mnemonic UnChunK Equivalent key values are fully interchangeable Multiple equivalent options are provided to satisfy disparate needs and tastes of NCO users working with scripts and from the command line The chunking algorithms must know the chunksizes of each dimension of each variable to be chunked The correspondence between the input variable shape and the chunksizes is called the chunking map The user specifies the desired chunking map with the M switch or its long option equivalents cnk_map and chunk_map and its cnk_map argument Eight chunking maps are currently implemented Chunksize Equals Dimension Size Definition Chunksize defaults to dimension size Explicitly specify chunksizes for particular dimensions with cnk_dmn option cnk_map key values dmn cnk_dmn map_dmn Mnemonic DiMeNsion Chunksize Equals Dimension Size except Record Dimension Definition Chunksize equals dimension size except record dimension has size one Explicitly specify chunksizes for particular dimensions with cnk_dmn option cnk_map key values
315. iate some of the NCO user s pain when handling MKS units It connects users who think in human friendly units e g miles millibars days to extract data which are always stored in God s units MKS e g meters Pascals seconds The feature is not intended to encourage writers to store data in esoteric units e g furlongs pounds per square inch fortnights 3 25 Rebasing Time Coordinate Availability ncra ncrcat Short options None Time rebasing is invoked when numerous files share a common record coordinate and the record coordinate units change among input files The rebasing is performed automatically if and only if UDUnits is installed Usually rebasing occurs when the recoordinate is a time based variable and times are recorded in units of a time since basetime and the basetime changes from file to file Since the output file can have only one unit i e one basetime for the record coordinate NCO in such cases chooses the units of the first input file to be the units of the output file It is necessary to rebase all the input record variables to this output time unit in order for the output file to have the correct values For example suppose the time coordinate is in hours and each day in January is stored in its own daily file Each daily file records the temperature variable tpt time with an unadjusted time coordinate value between 0 23 hours and uses the units attribute to advance the base time file01
316. ic grid variables Ne O lon_gds type NC_DOUBLE 1 dimensions 4 attributes chunked no compressed no packed no ID 38 lon gds RAM size is 8 sizeof NC DOUBLE 8 8 64 bytes lon gds dimension 0 gds crd size 8 NC FLOAT dim ID 17 CRD lon gds attribute 0 long name size 9 NC_CHAR value Longitude lon gds attribute 1 standard name size 9 NC CHAR value longitude lon gds attribute 2 units size 6 NC CHAR value degree lon gds attribute 3 purpose size 63 NC CHAR value 1 D longitude coordinate referred to by geodesic grid variables In this example lat_gds and lon_gds represent the latitude or longitude respectively of cell based variables These coordinates must have the same single dimension gds crd in this case as the cell based variables And the coordinates must be one dimensional multidimensional coordinates will not work This infrastructure allows NCO to identify interpret and process e g hyperslab the variables on cell based grids as easily as it works with regular grids To time average all the values between zero and 180 degrees longitude and between plus and minus 30 degress latitude we use ncra 0 X 0 180 30 30 v gds 3dvar in nc out nc NCO accepts multiple X arguments for cell based grid multi slabs just as it accepts multiple d arguments for multi slabs of regular coordinates ncra 0 X 0 180 30 30 X 270 315 45 90 in nc out nc The argu
317. icy with the P switch or its long option equiv alents cnk_plc and chunk_policy and its cnk plc argument As of August 2014 six chunking policies are implemented 21 This behavior became the default in November 2014 with NCO version 4 4 7 Prior versions would always use netCDF default chunking in the output file when no NCO chunking switches were activated regardless of the chunking in the input file Chapter 3 Shared Features 87 Chunk All Variables Definition Chunk all variables possible For obvious reasons scalar variables cannot be chunked Alternate invocation ncchunk cnk_plc key values all cnk_a11 plc all Mnemonic All Chunk Variables with at least Two Dimensions default Definition Chunk all variables possible with at least two dimensions Alternate invocation none cnk_plc key values g2d cnk_g2d plc_g2d Mnemonic Greater than or equal to 2 Dimensions Chunk Variables with at least Three Dimensions Definition Chunk all variables possible with at least three dimensions Alternate invocation none cnk plc key values g3d cnk_g3d plc_g3d Mnemonic Greater than or equal to 3 Dimensions Chunk One Dimensional Record Variables Definition Chunk all 1 D record variables Alternate invocation none Any specified with cnk_dmn record dimension chunksizes will be applied only to 1 D record variables and to no other variables Other dimensions
318. ifying groups whether in addition to or instead of variables is quite similar and is controlled by the g grp and x options A list of variables or groups to extract is specified following the v and g options e g v time lat lon or g grp1 grp2 Both options may be specified simultaneously and NCO will extract the intersection of the lists i e only variables of the specified names found in groups of the specified names The unn option causes NCO to extract the union rather than the intersection of the specified groups and variables Not using the v or g option 13 Linux and AIX do support LFS 44 NCO 4 5 3 alpha03 User Guide is equivalent to specifying all variables or groupp respectively The x option causes the list of variables specified with v to be excluded rather than extracted Thus x saves typing when you only want to extract fewer than half of the variables in a file 6 Variables or groups explicitly specified for extraction with v var or g erpl must be present in the input file or an error will result Variables explic itly specified for exclusion with x v var need not be present in the input file To accord with the sophistication of the underlying hierarchy group subsetting is controlled by a few powerful yet subtle syntactical distinctions When learning this syntax it is helpful to keep in mind the
319. ility Since no objective statistics were ever performed to quantify the difference in speed between the Fortran and C code the performance penalty incurred by this decision is unknown Sup porting Fortran involves maintaining two sets of routines for every arithmetic operation The USE FORTRAN ARITHMETIC flag is still retained in the Makefile The file containing the Fortran code nco fortran F has been deprecated but a volunteer Dr Frankenstein could resurrect it If you would like to volunteer to maintain nco_fortran F please contact me 3 NCO may still build with an ANSI or ISO C89 or C94 95 compliant compiler if the C pre processor undefines the restrict type qualifier e g by invoking the compiler with Drestrict 10 NCO 4 5 3 alpha03 User Guide 1 3 1 Compiling NCO for Microsoft Windows OS NCO has been successfully ported and tested on most Microsoft Windows operating systems including XP SP2 Vista 7 Support is provided for compiling either native Windows executables using the Microsoft Visual Studio 2010 Compiler or with Cygwin the UNIX emulating compatibility layer with the GNU toolchain The switches necessary to accomplish both are included in the standard distribution of NCO Using Microsoft Visual Studio MVS one must build NCO with the C compiler since MVS does not support C99 Qt a convenient integrated development environment was used to convert the project files to MVS format The Qt files the
320. im since 1on has already been hyperslabbed there should be no reason to hyperslab 1on again Should you need to hyperslab 1on again be sure to give dimensional indices as the hyperslab arguments rather than coordinate values see Section 3 15 Hyperslabs page 58 3 21 Auxiliary Coordinates Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat Short options X lon min lon max lat min lat max Long options auxiliary lon min lon max lat min lat max Utilize auxiliary coordinates specified in values of the coordinate variable s standard name attributes if any when interpreting hyperslab and multi slab options Also auxiliary This switch supports hyperslabbing cell based grids aka unstructured grids over coordinate ranges This works on datasets that associate coordinate variables to grid mappings using the CF convention see Section 3 41 CF Conventions page 124 coordinates and standard name attributes described here http cfconventions org 68 NCO 4 5 3 alpha03 User Guide 1 6 html coordinate system Currently NCO understands auxiliary coordinate vari ables pointed to by the standard_name attributes for latitude and longitude Cells that contain a value within the user specified range lon_min lon_max lat_min lat_max are in cluded in the output hyperslab A cell based or unstructured grid collapses the horizontal spatial information latitude and longitude and stores it along a one dimensional
321. in case some other non NCO software packed them in the first place Concurrently Gaussian and area weights and other grid properties are often used to derive fields in re inflated unpacked files so packing such grid properties causes a consid erable loss of precision in downstream data processing If users express strong wishes to pack grid properties we will implement new packing policies An immediate workaround for those needing to pack grid properties now is to use the ncap2 packing functions or to rename the grid properties prior to calling ncpdq We welcome your feedback To reduce required memorization of these complex policy switches ncpdq may also be invoked via a synonym or with switches that imply a particular policy ncpack is a synonym for ncpdq and behaves the same in all respects Both ncpdq and ncpack assume a default packing policy request of all_new Hence ncpack may be invoked without any P switch unlike ncpdq Similarly ncunpack is a synonym for ncpdq except that ncpack implicitly assumes a request to unpack i e P pck upk Finally the ncpdq U switch or its long option equivalents upk and unpack requires no argument It simply requests unpacking Given the menagerie of synonyms equivalent options and implied options a short list of some equivalent commands is appropriate The following commands are equivalent for packing ncpdq P all new ncpdq pck_plc all_new and nc
322. in data processing is often creating seasonal cycles Imagine a 100 year simulation with its 1200 monthly mean files Our goal is to create a single file containing 12 months of data Each month in the output file is the mean of 100 input files Normally we store the reduced data in a smaller local directory 276 NCO 4 5 3 alpha03 User Guide caseid T42x1_40 drc_in DATA archive caseid atm drc_in DATA caseid drc_out DATA caseid mkdir p drc_out cd drc_out Method 1 Assume all data in directory applies for mth in 1 12 do mm printf 02d mth ncra 0 D 1 o drc_out caseid _clm mm nc drc_in caseid cam2 h0O mm nc done end loop over mth Method 2 Use shell globbing to construct input filenames for mth in 1 12 do mm printf 02d mth ncra 0 D 1 o drc_out caseid _clm mm nc drc_in caseid cam2 h0 007 mm nc drc_in caseid cam2 h0 0100 mm nc done end loop over mth Method 3 Construct input filename list explicitly for mth in 1 12 do mm printf 02d mth fl lst in for yr in 1 100 do yyyy printf 404d yr f1_in caseid cam2 h0 yyyy mm nc fl_lst_in fl_lst_in caseid cam2 h0 yyyy mm nc done end loop over yr ncra 0 D 1 o drc_out caseid _clm mm nc p drc_in A fl_lst_in done end loop over mth Make sure the output file averages correct input files ncks M p
323. ing see Section 3 11 Subsetting Files page 43 yields powerful control over the extracted or excluded variables and groups and their placement in the output file as shown by the following commands All commands below may be assumed to end with in nc out nc 50 Prepending paths without editing NCO 4 5 3 alpha03 User Guide ncks g v gt g v ncks v vi g v1 gt g v1 ncks g g1 g1 v gt g1 v ncks G ol g v gt o1 g v ncks G ol g gl g1 v gt o01 g1 v ncks g gl v vi gi vi gt g1 v1 ncks G ol v vi g v1 gt 01 g v1 ncks G o1 g gl v vi g1 v1 gt 01 g1 v1 ncks G gi g v vi v1 gt gi vi ncks G gi g2 v vi g vi gt g1 g2 g v1 Delete mode Delete from and Prepend to path head Syntax G ppn lvl_nbr of levels to delete ncks G 1 g gi v vi g1 v1 gt vi ncks G 1 g g1 g1 v vi g1 g1 v1 gt g1 v1 ncks G 2 g g1 g1 v vi g1 g1 v1 gt vl ncks G 2 g gl v vi g1 v1 gt vi ncks G g2 1 g gl v vi g1 v1 g2 v1 ncks G g2 2 g gi gi v vi g1 g1 v1 gt g2 v1 ncks G g2 1 g v vi vl gt g2 v1 ncks G g2 1 v vi vl gt g2 v1 ncks G g2 1 g gl gl v vi g1 g1 v1 gt g2 g1 v1 Flatten mode Remove all input path components Syntax G apn colon without numerical argument ncks G v v1 tt g vl gt vl ncks G g g1 v vi tt g1 v1 gt vl ncks G g g
324. ing and or dimension reordering e Broadcasting a variable means creating data in non existing dimensions by copying data in existing dimensions e More specifically If the numbers of dimensions in varl is greater than or equal to the number of dimensions in var2 then an attempt is made to make var2 conform to varl else varl is made to conform to var2 If conformance is not possible then an error message will be emitted and script execution will cease Even though the logical operators return True 1 or False 0 they are treated in the same way as the arithmetic operators with regard to precision and rank Examples dimensions time 10 lat 2 lon 4 Suppose we have the two variables double P time lat lon float PZO 1on lat PZ0 1 2 3 4 5 6 7 8 Consider now the expression PZ P PZO PZO is made to conform to P and the result is PZO 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 1 3 5 7 2 4 6 8 9 9 Once the expression is evaluated then PZ will be of type double Chapter 4 Reference Manual 133 Consider now start four att varOdouble att start 69 and is of type intger four pow four 3 0f four pow 64 and is of type float three nw three dmn var sht 1 0f type is now float StartOni att varOshort att att var Oint att start n1i 5329 and is type int Binary Operators Unlike
325. ing in the netCDF interface It is unclear whether such problems exist in present day 2007 netCDF NCO environ ments where unnecessary byte swapping has been reduced or eliminated Chapter 3 Shared Features 27 3 Shared Features Many features have been implemented in more than one operator and are described here for brevity The description of each feature is preceded by a box listing the operators for which the feature is implemented Command line switches for a given feature are consistent across all operators wherever possible If no key switches are listed for a feature then that particular feature is automatic and cannot be controlled by the user 3 1 Internationalization Availability All operators NCO support for internationalization of textual input and output e g Warning mes sages is nascent We introduced the first foreign language string catalogues French and Spanish in 2004 yet did not activate these in distributions because the catalogues were nearly empty We seek volunteers to populate our templates with translations for their favorite languages 3 2 Metadata Optimization Availability All operators Short options None Long options hdr_pad header_pad NCO supports padding headers to improve the speed of future metadata operations Use the hdr_pad and header_pad switches to request that hdr pad bytes be inserted into the metadata section of the output file Future
326. ing of NC CHAR attributes is a bit like Perl in that it attempts to do what you want by default but this sometimes causes unexpected results if you want unusual data storage If the att type is NC CHAR then the argument is interpreted as a string and it may contain C language escape sequences e g Nn which NCO will interpret before writing anything to disk NCO translates valid escape sequences and stores the appropriate ASCIT code instead Since two byte escape sequences e g Nn represent one byte ASCII codes e g ASCII 10 decimal the stored string attribute is one byte shorter than the input string length for each embedded escape sequence The most frequently used C language escape sequences are An for linefeed and Mt for horizontal tab These sequences in particular allow convenient editing of formatted text attributes The other valid ASCII codes are Na Nb f Mr v and See Section 4 7 ncks netCDF Kitchen Sink page 203 for more examples of string formatting with the ncks s option with special characters Analogous to printf other special characters are also allowed by ncatted if they are protected by a backslash The characters and may be input to the shell as V and NCO simply strips away the leading backslash from these characters before editing the attribute No other characters require protection by a backslash Backslashes which precede any other character e g 3 m amp
327. ion at a specific coordinate is extracted by specifying only the min limit and omitting a trailing comma Dimensions not mentioned are passed with no reduction in range The dimensionality of variables is not reduced in the case of a cross section the size of the constant dimension will be one First and second longitudes necks F d lon 1 2 in nc out nc Second and third longitudes ncks d lon 1 2 in nc out nc As of version 4 2 1 August 2012 NCO allows one to extract the last N elements of a hyperslab Negative integers as min or max elements of a hyperslab specification indicate offsets from the end Python also uses this convention Consistent with this convention the value 1 negative one indicates the last element of a dimension and negative zero is Chapter 3 Shared Features 59 algebraically equivalent to zero and so indicates the first element of a dimension Previously for example d time 2 1 caused a domain error Now it means select the penultimate and last timesteps independent of the size of the time dimension Select only the first and last timesteps respectively with d time 0 and d time 1 Negative integers work for min and max indices though not for stride Second through penultimate longitudes necks d lon 1 2 in nc out nc Second through last longitude ncks d lon i 1 in nc out nc Second to last to last longitude ncks d lon 3 1 in nc out nc Second to last to last long
328. ion is the coordinate variable value closest to the specified value and must lie within the range or coordinate variable values The stride argument if any must be a dimension index not a coordinate value See Section 3 16 Stride page 60 for more information on the stride option All longitude values between 1 and 2 degrees ncks d lon 1 0 2 0 in nc out nc All longitude values between 1 and 2 degrees ncks F d 10on 1 0 2 0 in nc out nc Every other longitude value between O and 90 degrees ncks F d 10n 0 0 90 0 2 in nc out nc 60 NCO 4 5 3 alpha03 User Guide As shown we recommend using a full floating point suffix of 0 instead of simply in order to make obvious the selection of hyperslab elements based on coordinate value rather than index User specified coordinate limits are promoted to double precision values while searching for the indices which bracket the range Thus hyperslabs on coordinates of type NC CHAR are computed numerically rather than lexically so the results are unpredictable The relative magnitude of min and max indicate to the operator whether to expect a wrapped coordinate see Section 3 20 Wrapped Coordinates page 66 such as longitude If min gt max the NCO expects the coordinate to be wrapped and a warning message will be printed When this occurs NCO selects all values outside the domain max lt min i e all the values exclusive of the values which would have been selected if min and
329. ion of single to double precision is also warranted Understanding the narrowness of these situations takes time and we hope the reader appreciates the following detailed discussion Consider the average of the two NC SHORTs 17000s and 17000s A straightforward average without promotion results in garbage since the intermediate value which holds their 114 NCO 4 5 3 alpha03 User Guide sum is also of type NC SHORT and thus overflows on i e cannot represent values greater than 32 767 There are valid reasons for expecting this operation to succeed and the NCO philosophy is to make operators do what you want not what is purest Thus unlike C and Fortran but like many other higher level interpreted languages NCO arithmetic operators will perform automatic type conversion on integers when all the following conditions are met 1 The requested operation is arithmetic This is why type conversion is limited to the operators ncap2 ncbo nces ncflint ncra and ncwa 2 The arithmetic operation could benefit from type conversion Operations that could benefit include averaging summation or any hard arithmetic that could overflow or underflow Larger representable sums help avoid overflow and more precision helps to avoid underflow Type conversion does not benefit searching for minima and maxima y min or y max 3 The variable on disk is of type NC_BYTE NC_CHAR NC_SHORT or NC_INT Type NC_ DOUBLE is not promot
330. ions 1 attribute chunked no compressed no 1i sizeof NC FLOAT 1 4 4 bytes 3 NC CHAR value one zender roulee ncks cdl v one nco data in nc netcdf in variables float one one long name one data one 1 group 5 ncgen converts CDL mode output into a netCDF file ncks cdl v one nco data in nc gt in cdl ncgen k netCDF 4 b o in nc in cdl ncks v one in nc The HDF version of ncgen often netCDF3 CDL into an HDF file usr hdf4 bin ncgen b o usr bin hncgen b o usr bin ncgen hdf b o hdp dumpsds in hdf named hncgen or ncgen hdf converts in hdf in cdl HDF ncgen local builds in hdf in cdl Same as HDF ncgen RPM package in hdf in cdl Same as HDF ncgen Debian pack ncdump h5dump equivalent for H Note that HDF4 does not support netCDF style groups so the above com mands fail when the input file contains groups Only netCDF4 and HDF5 sup port groups In our experience the HDF ncgen command by whatever name installed is not robust and can fail on valid netCDF3 CDL mk_rec_dmn dim Change existing dimension dim to a record dimension in the output file This is the most straightforward way of changing a dimension to a the record dimen sion and works fine in most cases See Section 4 5 ncecat netCDF Ensemble Chapter 4 Reference Manual 207 M Concatenator page 197 and Section 4 8 ncpdq netCDF Permute Dimension
331. is feature may be used to augment a target file rather than construct it from scratch This helps for example when a timeseries is concatenated from input data that becomes available in stages rather than all at once In such cases this switch significantly speeds writing 62 NCO 4 5 3 alpha03 User Guide Consider the use case where one wishes to preserve the contents of 1 1 nc and add to them new records contained in 1_2 nc Previously the output had to be placed in a third file 1_3 nc which could also safely be named 1 2 nc via ncrcat 0 fl 1 nc f1 2 nc f1_3 nc Under the hood this operation copies all information in 1 1 nc and 1 2 nc not once but twice The first copy is performed through the netCDF interface as all data from fl 1 nc and f1_2 nc are extracted and placed in the output file The second copy occurs usually much more quickly as the by default temporary output file is copied sometimes a quick re link suffices to the final output file see Section 2 3 Temporary Output Files page 17 All this copying is expensive for large files The record append switch appends all records in 1 2 nc to the end after the last record of 1 1 nc ncrcat rec apn fl 2 nc f1_1 nc The ordering of the filename arguments may seem non intuitive If the record variable represents time in these files then the values in 1 1 nc precede those in 1 2 nc so why do the files appear in the reverse order on the
332. is of type NC STRING In this case the receiving variable need not have any memory allocated to hold the string because netCDF4 handles that memory allocation Suppose one creates a file containing an ensemble of model results and wishes to label the record coordinate with the name of each model The NC STRING type is well suited to this because it facilitates storing arrays of strings of arbitrary length This is sophisticated though easy with ncap2 4 ncecat 0 u model cesm nc ecmwf nc giss nc out nc ncap2 4 0 s model model cesm s ecmwf s giss s out nc out nc The key here to place an s character after each double quoted string value to indicate an NC STRING type The 4 ensures the output filetype is netCDF4 in case the input filetype is not 142 NCO 4 5 3 alpha03 User Guide netCDF3 4 Types bIB NC_BYTE a signed 1 byte integer none NC_CHAR an ISO ASCII character s S NC_SHORT a signed 2 byte integer IL NC_INT a signed 4 byte integer f F NC FLOAT a single precision 4 byte floating point number d D NC_DOUBLE a double precision 8 byte floating point number netCDF4 Types ub UB NC_UBYTE an unsigned 1 byte integer us US NC USHORT an unsigned 2 byte integer u Ulull UL NC UINT an unsigned 4 byte integer HILL NC_INT64 a signed 8 byte integer ul ULL NC UINT64 an unsigned 8 byte integer S NC STRING a string of arbitrary length 4 1 8 if statement The syntax of the if statemen
333. itch is now obsolete though harmless for backwards compatibility Prior to version 4 3 7 October 2013 NCO lacked the software necessary to workaround netCDF library flaws handling HDF4 files and thus NCO failed to convert HDF4 files to netCDF files In those cases use the ncl_convert2nc command distributed with NCL to convert HDF4 files to netCDF ncpdq does not support packing data using the HDF convention Although it is now straightforward to support this we think it might sow more confusion than it reaps Let us know if you disagree and would like NCO to support packing data with HDF algorithm Chapter 4 Reference Manual 223 Suppose the dimension lat represents latitude and monotonically increases increases from south to north Reversing the lat dimension means re ordering the data so that latitude values decrease monotonically from north to south Accomplish this with ncpdq a lat in nc out nc 4 ncks C v lat in nc lat 0 90 lat 1 90 ncks C v lat out nc lat 0 90 lat 1 90 This operation reversed the latitude dimension of all variables Whitespace immediately preceding the negative sign that specifies dimension reversal may be dangerous Quotes and long options can help protect negative signs that should indicate dimension reversal from being interpreted by the shell as dashes that indicate new command line switches ncpdq a lat in nc out nc Dangerous Whitespace before lat ncpdq a lat in
334. ithout explicit user intervention In contrast manual type conversion refers to explicit user commands to change the type of a variable or attribute Most type conversion happens automatically yet there are situations in which manual type conversion is advantageous 3 36 1 Automatic type conversion There are at least two reasons to avoid type conversions First type conversions are expen sive since they require creating temporary buffers and casting each element of a variable from its storage type to some other type and then often converting it back Second a dataset s creator perhaps had a good reason for storing data as say NC FLOAT rather than NC DOUBLE In a scientific framework there is no reason to store data with more precision than the observations merit Normally this is single precision which guarantees 6 9 digits of precision Reasons to engage in type conversion include avoiding rounding errors and out of range limitations of less precise types This is the case with most integers Thus NCO defaults to automatically promote integer types to floating point when performing lengthy arithmetic yet NCO defaults to not promoting single to double precision floats Before discussing the more subtle floating point issues we first examine integer promo tion We will show how following parsimonious conversion rules dogmatically can cause problems and what NCO does about that That said there are situations in which implicit convers
335. itude ncks d lon 3 in nc out nc The F argument if any applies the Fortran index convention only to indices specified as positive integers First through penultimate longitudes ncks F d lon 1 2 in nc out nc F affects only start index First through last longitude ncks F d lon 1 1 in nc out nc Second to last to penultimate longitude F has no effect ncks F d lon 3 1 in nc out nc Second to last to last longitude F has no effect ncks F d lon 3 in nc out nc Coordinate values should be specified using real notation with a decimal point required in the value whereas dimension indices are specified using integer notation without a decimal point This convention serves only to differentiate coordinate values from dimension indices It is independent of the type of any netCDF coordinate variables For a given dimension the specified limits must both be coordinate values with decimal points or dimension indices no decimal points If values of a coordinate variable are used to specify a range or cross section then the coordinate variable must be monotonic values either increasing or decreasing In this case command line values need not exactly match coordinate values for the specified dimension Ranges are determined by seeking the first coordinate value to occur in the closed range min max and including all subsequent values until one falls outside the range The coor dinate value for a cross sect
336. ize UTF 8 strings to NFC to facilitate matching and queries Chapter 4 Reference Manual 183 4 2 ncatted netCDF Attribute Editor SYNTAX ncatted a att dsc a D dbg h hdr pad nbr 1 path 0 o output file p path R r ram a11 input file output filel DESCRIPTION ncatted edits attributes in a netCDF file If you are editing attributes then you are spending too much time in the world of metadata and ncatted was written to get you back out as quickly and painlessly as possible ncatted can append create delete modify and overwrite attributes all explained below ncatted allows each editing operation to be applied to every variable in a file This saves time when changing attribute conventions throughout a file ncatted is for writing attributes To read attribute values in plain text use ncks m M or define something like ncattget as a shell command see Section 4 7 2 Filters for ncks page 211 Because repeated use of ncatted can considerably increase the size of the history global attribute see Section 3 39 History Attribute page 122 the h switch is provided to override automatically appending the command to the history global attribute in the output file According to the netCDF User Guide altering metadata in netCDF files does not incur the penalty of recopying the entire file when the new metadata occupies less space than the old metadata Thus ncatted may run much faster a
337. k HDF file into netCDF file ncpdq hdf4 hdf upk P xst new modis hdf modis nc HDF4 ncpdq hdf upk P xst new modis h5 modis nc HDF5 Two step procedure For NCO 4 3 6 and earlier 1 Convert HDF file to netCDF file ncl_convert2nc modis hdf 2 Unpack using HDF convention and repack using netCDF convention ncpdq hdf upk P xst new modis nc modis nc NCO now automatically detects HDF4 files In this case it produces an output file modis nc which preserves the HDF packing used in the input file The ncpdq command first unpacks all packed variables using the HDF unpacking algorithm as specified by hdf upk and then repacks those same variables using the netCDF algorithm be cause that is the only algorithm NCO packs with As described above the P xst_new packing policy only repacks variables that are already packed Not packed variables are copied directly without loss of precision Re order file in nc so that the dimension lon always precedes the dimension lat and store the results in out nc ncpdq a lon lat in nc out nc ncpdq v three dmn var a lon lat in nc out nc The first command re orders every variable in the input file The second command extracts and re orders only the variable three dmn var Prior to NCO 4 4 0 and netCDF 4 3 1 January 2014 NCO requires the hdf4 switch to correctly read HDF4 input files For example ncpdq hdf4 hdf upk P xst new modis hdf modis nc That sw
338. l nc foo avg mth2 nc foo avg dff nc ncks cdl foo avg dff nc tpt dbl 5 6843418860808e 14 tpt flt 3 051758e 05 Although the two methods are arithmetically equivalent they produce slightly different answers due to the different order of operations Moreover it appears at first glance that the single precision answers suffer from greater error than the double precision answers In fact both precisions suffer from non zero rounding errors The answers differ negligibly to machine precision which is about seven significant figures for single precision floats tpt_ flt and sixteen significant figures for double precision tpt dbl The input precision determines the answer precision IEEE arithmetic guarantees that two methods will produce bit for bit identical answers only if they compute the same operations in the same order Bit for bit identical answers may also occur by happenstance when rounding errors exactly compensate one another This is demonstrated by repeating the example above with the dbl or rth_db1 for clarity option which forces conversion of single precision numbers to double precision prior to arithmetic Now ncra will treat the first value of tpt flt 273 1000f as 273 1000000000000d Arithmetic on tpt flt then proceeds in double precision until the final answer which is converted back to single precision for final storage Average 1st five Januaries elements 1 11 21 31 41 ncra
339. l operators ncap2 0 4 D 4 cnk_scl 8 S nco data ncap2_tst nco nco data in nc foo nc ncbo 0 4 D 4 cnk scl 8 p nco data in nc in nc foo nc ncecat 0 4 D 4 n 12 2 1 cnk dmn 1at 32 p data zender dstmch90 dstmch90_clm01 nc foo nc ncflint 0 4 D 4 cnk_scl 8 nco data in nc foo nc ncpdq 0 4 D 4 P all new cnk scl 8 L 5 nco data in nc foo nc ncrcat 0 4 D 4 n 12 2 1 cnk dmn 1at 32 p data zender dstmch90 dstmch90_clm01 nc foo nc ncwa 0 4 D 4 a time cnk plc g2d cnk_map rd1 cnk dmn 1at 32 cnk dmn lon 128 dstmch90_clm_0112 nc foo nc Chunking policy rid changes the chunksize of 1 D record variables and no other vari ables to that specified with cnk_dmn chunksize Any specified record dimension chunk sizes will be applied to 1 D record variables only Other dimensions may be chunked with their own cnk_dmn options that will apply to all variables For example ncks cnk plc rid cnk dmn time 1000 in nc out nc This sets time chunks to 1000 only in 1 D record variables Without the rid policy time chunks would change in all variables It is appropriate to conclude by informing users about an aspect of chunking that may not be expected Three types of variables are always chunked Record variables Deflated compressed variables and Checksummed variables Hence all variables that contain a record dimension are also chunked since data
340. l to usr local 56 NCO 4 5 3 alpha03 User Guide 1 HDF4 Build in non default manner Turn off its own netCDF support Per http www unidata ucar edu software netcdf docs build hdf4 html HDF4 support not necessary though it makes ncdismember more comprehensive wget c http www hdfgroup org ftp HDF HDF Current src hdf 4 2 9 tar gz tar xvzf hdf 4 2 9 tar gz cd hdf 4 2 9 configure enable shared disable netcdf disable fortran prefix H4DIR make amp amp make check amp amp make install 2 HDF5 Build normally RPM may work too Please let me know if so HDF5 is a necessary pre requisite for netCDF4 wget c ftp ftp unidata ucar edu pub netcdf netcdf 4 hdf5 1 8 11 tar gz tar xvzf hdf5 1 8 11 tar gz cd hdf5 1 8 11 configure enable shared prefix H5DIR make amp amp make check amp amp make install 3 netCDF version 4 3 1 or later Build in non default manner with HDF4 Per http www unidata ucar edu software netcdf docs build hdf4 html Earlier versions of netCDF may fail checking some HDF4 files wget c ftp ftp unidata ucar edu pub netcdf netcdf 4 3 2 tar gz tar xvzf netcdf 4 3 2 tar gz cd netcdf 4 3 2 CPPFLAGS I H5DIR include I H4DIR include LDFLAGS L H5DIR lib L H4DIR lib configure enable hdf4 enable hdf4 file tests make amp amp make check amp amp make install 4 NCO version 4 4 0 or later Some RPMs available Or install by hand Late
341. le a 128x256 Uniform grid is the offset or staggered version of a 129x256 Cap grid aka FV grid Referring the saucer like cap points at the poles NCO uses the term Cap grid to describe the latitude portion of the FV scalar grid as used by the CAM FV Lin Rood dynamics formulation NCO accepts the shorthand FV and the more descriptive Yarmulke as synonyms for Cap A Cap latitude grid differs from a Uniform latitude grid in many ways Most importantly Cap grids are 2D representations of numerical grids with cap midpoints instead of zonal teeth convergence at the poles The rectangular 2D representation of each cap contains gridcells shaped like sharp teeth that converge at the poles similar to the Uniform grid but the Cap gridcells are meant to be aggregated into a single cell centered at the pole in a dynamical transport algorithm In other words the polar teeth are a convenient way to encode a non rectangular grid in memory into a rectangular array on disk 18 A Uniform grid in latitude could be called equi angular in latitude but NCO reserves the term Equi angular or eqa for grids that have the same uniform spacing in both latitude and longitude e g 1 x1 or 2 x2 NCO reserves the term Regular to refer to grids that are monotonic and rectangular grids Confusingly the angular spacing in a Regular grid need not be uniform it could be irregular such as in a Gaussian grid The term Regular is not too useful in
342. le concatenation 2e eee 197 Equi Angular grid 00 cece eee eee 69 equiangular Plis ts za ae 74 Abu o bes DAA 178 eh eu is MAU I pee du pue E CU re 178 Enc Blake inii eree Eua aA EREN TPE 242 error function veru AD Lm Penes 178 error tolerance 0 00 eee eee eee eee eens 17 ESME ana DLERRRSIILGIG IA PRRABEDRLS A 69 73 Etienne Tourigny 0 000005 243 exclusion cece eee eee ee eee 43 213 215 execution time 11 18 26 27 85 183 231 EXD A D Mea wee eee 178 EXPONent 12400222 Betas kat ME eee So RUE TA 120 exponentiation lucia ete tea we vet Y Sea ts 178 exponentiation function 000 178 expressions vel pect peek Tag bea eee pees 131 extended file format 0 cee eee eee 40 extended regular expressions 22 46 177 184 187 extensive variable eee e eee eee ee 76 exi factlon ivo sid ane dad se enia 43 213 215 291 F POOR Soe E EA 10 features requesting 0 eee eee ee ee eee 15 File buffers 1 2 pauses lx ese ade REDI 103 file combination 0 e eee eee eee 247 file deletion i222 eee ee esas eri Ii 37 file multiplication 0 0 e cece ee eee 202 flesemovalicsilst ite whRe 4 RR deen tds 37 file retention 222124 4 eating rhe 37 hil s multiple eem reme 31 files numerous input ssseeeeeeeees 21 Filipe Fernandes cece eee eee 242
343. le is placed in its own top level group in output file which is automatically made as a netCDF4 format file GAG has two methods to specify group names for the output file The G option or its long option equivalent gpe takes as argument a group path editing description gpe_dsc of where to place the results Each input file needs a distinct output group name to avoid namespace conflicts in the output file Hence ncecat automatically creates unique output group names based on either the input filenames or the gpe dsc arguments When the user provides gpe dsc i e with G then the output groups are formed by enumerating sequential two digit numeric suffixes starting with zero and appending them to the specified group path see Section 3 13 Group Path Editing page 48 When gpe_dsc is not provided i e user requests GAG with gag instead of G then ncecat forms the output groups by stripping the input file name of any type suffix e g nc and all but the final component of the full filename ncecat gag 85 nc 86 nc 87 nc 8587 nc Output groups 85 86 87 ncecat G 85 a nc b nc c nc 8589 nc Output groups 85 00 85 01 85 02 ncecat G 85 a nc b nc c nc 8589 nc Output groups 85 00 85 01 85 02 With both RAG and GAG the output file size is the sum of the sizes of the extracted variables in the input files See Section 2 6 Statistics vs Concatenation page 20 for a description of the distinctio
344. limatology convention described in Section 3 41 CF Conventions page 124 As of NCO version 3 9 6 January 2009 both c and C honor the CF coordinates convention described in Section 3 41 CF Conventions page 124 3 13 Group Path Editing Options G gpe_dsc Availability ncbo ncecat nces ncflint ncks ncpdq ncra ncrcat ncwa Short options G Long options gpe Group Path Editing or GPE allows the user to restructure i e add remove and rename groups in the output file relative to the input file based on the instructions they provide As of NCO 4 2 3 November 2012 all operators that accept netCDF4 files with groups accept the G switch or its long option equivalent gpe To master GPE one must understand the meaning of the required gpe_dsc structure argument that specifies the transformation of input to output group paths Each gpe_dsc contains up to three elements two are optional in the following order gpe_dsc grp_pth lvl_nbr or grp pthGlvl nbr grp pth Group Path This optional component specifies the output group path that should be appended after any editing i e deletion or truncation of the input path is performed Ivl_nbr The number of levels to delete from the head or truncate from the tail of the input path Chapter 3 Shared Features 49 If both components of the argument are present then a single character either the colon or at sign or
345. login zender password bushlied Eschew using valuable passwords for FTP transactions since netrc passwords are po tentially exposed to eavesdropping software 3 NCO does not implement command line options to specify FTP logins and passwords because copying those data into the history global attribute in the output file done by default poses an unacceptable security risk 34 NCO 4 5 3 alpha03 User Guide SFTP i e secure FTP uses SSH based security protocols that solve the security issues associated with plain FTP NCO supports SFTP protocol access to files specified with a homebrew syntax of the form Sftp machine domain tld path to filename Note the second colon following the top level domain tld This syntax is a hybrid between an FTP URL and standard remote file syntax To access a file using rcp or scp specify the Internet address of the remote file Of course in this case you must have rcp or scp privileges which allow transparent no password entry required access to the remote machine This means that rhosts or ssh authorized keys must be set accordingly on both local and remote machines To access a file on a High Performance Storage System HPSS such as that at NCAR ECMWF LANL DKRZ LLNL specify the full HPSS pathname of the remote file NCO will attempt to detect whether the local machine has direct synchronous HPSS access In this case NCO attempts to use the Hierarchical Storage Interface HSI com
346. lon nw 2 th nw 1at 10n nw th 0 0 1 th 9 0 1 Hyperslabs on the Left Hand Side of an assign When hyperslabing on the LHS the expression on the RHS must evaluate to a scalar or a variable attribute with the same number of elements as the LHS hyperslab Set all elements of the last record to zero th 9 0 0 Set first element of each lon element to 1 0 th 0 1 0 Chapter 4 Reference Manual 139 One may hyperslab on both sides of an assign For example this sets the last record to the first record th 9 th 0 Say th0 represents pressure at height 0 and th1 represents pressure at height 1 Then it is possible to insert these hyperslabs into the records prs time height 1at 10n 0 0 prs 0 th0 prs 1 th1 Reverse method Use the reverse method to reverse a dimension s elements in a variable with at least one dimension This is equivalent to a negative stride e g th_rv th 1 reverse lon 12 11 10 9 16 15 14 13 od rv od reverse time 38 36 34 32 30 28 26 24 22 20 Permute methodp Use the permute method to swap the dimensions of a variable The number and names of dimension arguments must match the dimensions in the variable If the first dimension in the variable is of record type then this must remain the first dimension If you want to change the record dimension then consider using ncpdq Consider the variable float three dmn var lat lev lon
347. long name time interval endpoints group The raw model output lacks the CF coordinates and bounds attributes that the re gridder adds The metadata turns lat and lon into auxiliary coordinate variables see Section 3 21 Auxiliary Coordinates page 67 which can then be hyperslabbed with X using latitude longitude coordinates bounding the region of interest ncks u H X 314 6 315 3 35 6 35 1 v FSNT dogfood nc time 0 31 ncol 0 FSNT 0 2344 575 W m2 ncol 0 lat 0 35 2643896828 degrees_north ncol 0 nv 0 lat_vertices 0 35 5977213708 ncol 0 nv 1 lat vertices 1 35 5977213708 ncol 0 nv 2 lat_vertices 2 35 0972113817 ncol 0 nv 3 lat vertices 3 2 35 0972113817 ncol 0 nv 4 lat_vertices 4 35 0972113817 ncol 0 lon 0 315 degrees_east ncol 0 nv 0 lon_vertices 0 315 ncol 0 nv 1 lon_vertices 1 315 ncol 0 nv 2 lon_vertices 2 315 352825437 ncol 0 nv 3 lon_vertices 3 314 647174563 Chapter 3 Shared Features 79 ncol 0 nv 4 lon_vertices 4 314 647174563 time 0 31 days since 1979 01 01 00 00 00 time 0 31 nbnd 0 time_bnds 0 0 time 0 31 nbnd 1 time_bnds 1 31 Thus auxiliary coordinate variables help to structure unstructured grids The expanded metadata annotations from an identity regridding may obviate the need to place unstruc tured data on a rectangular grid For example statistics for regions that can be expressed as unions of rectangular regions can now be performed
348. lute value of x y mabs is T max uimqwi oi i9 maws 2 uy mw ww z 110 NCO 4 5 3 alpha03 User Guide Thus the minimum and maximum absolute values are determined after any weights are applied The mean absolute value of x y mebs is i 2 3 Ge eue Tj i N Mul Bmauws The square of the mean value of x y sqravg is i N 2 m i BUT Tj i N et p MW The mean of the sum of squares of x y avgsqr is i N 2 gt Dial Mz iS i N Pici BTWUN If x represents a deviation from the mean of another variable x y y possibly created by ncbo in a previous step then applying avgsqr to x computes the approximate variance of y Computing the true variance of y requires subtracting 1 from the denominator discussed below For a large sample size however the two results will be nearly indistinguishable The root mean square of x y rms is i N 2 Seat LUTZ Tj a et p TT Thus rms simply computes the squareroot of the quantity computed by avgsqr The root mean square of z with standard deviation like normalization y rmssdn is implemented as follows When weights are not specified this function is the same as the root mean square of z except one is subtracted from the sum in the denominator Da imix TS i N 1 M um If x represents the deviation from the mean of another variable z y y then applying rmssdn to x computes the standard deviation of y In this cas
349. ly it is now easy to convert HDF4 files to netCDF files directly e g ncks fl hdf fl nc Convert HDF4 gt netCDF4 NCO 4 4 0 netCDF 4 3 1 ncks hdf4 fl hdf fl nc Convert HDF4 gt netCDF4 NCO 4 3 7 4 3 9 The most efficient and accurate way to convert HDF4 data to netCDF format is to convert to netCDF4 using NCO as above Many HDF4 producers NASA love to use netCDF4 types e g unsigned bytes so this procedure is the most typical Conversion of 8 The procedure for doing this is documented at http www unidata ucar edu software netcdf docs build hdf4 html 14 NCO 4 5 3 alpha03 User Guide HDF4 to netCDF4 as above suffices when the data will only be processed by NCO and other netCDF4 aware tools However many tools are not fully netCDF4 aware and so conversion to netCDF3 may be desirable Obtaining any netCDF file from an HDF4 is easy ncks 3 fl hdf fl nc HDF4 gt netCDF3 NCO 4 4 0 netCDF 4 3 1 ncks 4 fl hdf fl nc HDF4 gt netCDF4 NCO 4 4 0 netCDF 4 3 14 ncks 6 fl hdf fl nc HDF4 gt netCDF3 64 bit NCO 4 4 0 ncks 7 L 1 fl hdf fl nc HDF4 gt netCDF4 classic NCO 4 4 0 ncks hdf4 3 fl hdf fl nc HDF4 gt netCDF3 netCDF 4 3 0 ncks hdf4 4 fl hdf fl nc HDF4 gt netCDF4 netCDF 4 3 0 ncks hdf4 6 fl hdf fl nc HDF4 gt netCDF3 64 bit netCDF 4 3 0 ncks hdf4 7 fl hdf fl nc HDF4 gt netCDF4 classic netCDF 4 3 0 As of NCO version 4 4 0 January 2014 these comm
350. m per screen line with all dimension subscripts and coordinate values if any preceding the datum Option s or long options sng fmt and string permits the user to format data using C style format strings while option cdl outputs CDL and option xml outputs NcML ncks exposes many flexible controls over printed output including CDL and NcML Options 5 feat cdl CEP Hs hdn lt M m P Q q 8 tu V and xml and their long option counterparts control the formatted appearance of the data ncks extracts and optionally creates a new netCDF file comprised of only selected variables from the input file similar to the old ncextr specification Only variables and coordinates may be specifically included or excluded all global attributes and any attribute associated with an extracted variable are copied to the screen and or output netCDF file Options c C v and x and their long option synonyms control which variables are extracted ncks extracts hyperslabs from the specified variables ncks implements the original nccut specification Option d controls the hyperslab specification Input dimensions that are not associated with any output variable do not appear in the output netCDF This feature removes superfluous dimensions from netCDF files 204 NCO 4 5 3 alpha03 User Guide ncks will append variables and attri
351. mand hsi get The following examples show how one might analyze files stored on remote systems ncks 1 ftp dust ess uci edu pub zender nco in nc ncks 1 sftp dust ess uci edu home ftp pub zender nco in nc ncks 1 dust ess uci edu home zender nco data in nc ncks 1 ZENDER nco in nc ncks 1 home zender nco in nc ncks 1 http thredds test ucar edu thredds dodsC testdods in nc The first example works verbatim if your system is connected to the Internet and is not behind a firewall The second example works if you have sftp access to the machine dust ess uci edu The third example works if you have rcp or scp access to the machine dust ess uci edu The fourth and fifth examples work on NCAR computers with local access to the HPSS hsi get command The sixth command works if your local version of NCO is OPeNDAP enabled this is fully described in Section 3 7 1 OPeNDAP page 35 or if the remote file is accessible via wget The above commands can be rewritten using the p input path option as follows ncks p ftp dust ess uci edu pub zender nco 1 in nc ncks p sftp dust ess uci edu home ftp pub zender nco 1 in nc ncks p dust ess uci edu home zender nco 1 in nc ncks p ZENDER nco 1 in nc ncks p home zender nco 1 in nc HPSS ncks p http thredds test ucar edu thredds dodsC testdods The hsi command must be in the user s path in one of the following directories usr local bin opt h
352. map to var in var out unmap var in srt map Reverse what srt map did to var in dsr map invert map srt map Produce de sort map that inverts srt map The first two functions sort and asort sort in ascending order all the elements of var in which can be a variable or attribute without regard to any dimensions The third function dsort does the same but sorts in descending order Remember that ascending and descending sorts are specified by asort and dsort C respectively These three functions are overloaded to take a second optional argument called the sort map srt map which should be supplied as a call by reference variable i e preceded with an ampersand If the sort map does not yet exist then it will be created and returned as an integer type the same shape as the input variable Chapter 4 Reference Manual 153 The output var out of each sort function is a sorted version of the input var_in The output var_out of the two mapping functions the result of applying with remap or un applying with unmap the sort map srt map to the input var_in To apply the sort map with remap the size of the variable must be exactly divisible by the size of the sort map The final function invert_map returns the so called de sorting map dsr map which is inverse map of the input map srt_map This gives the user access to both the forward and inverse sorting maps which can be useful in special situations a1 time
353. mbert_W0_e gsl sf lambert_Wml _e gsl_sf_legendre_Pl_e gsl_sf_legendre_Pl_array gsl sf legendre Pl deriv array esl_sf_legendre_P1_e gsl sf legendre P2 e gsl_sf_legendre_P3_e gsl_sf_legendre_QO_e esl_sf legendre_Q1_e gsl_sf_legendre_Ql_e gsl_sf_legendre_Plm_e gsl_sf_legendre_Plm_array gsl_sf_legendre_Plm_deriv_array gsl sf legendre sphPlm e gsl sf legendre sphPlm array gsl sf legendre sphPlm deriv arra gsl sf legendre array size gsl sf conicalP half e KZZKKZKKKKKKKKZKMKKMKKMKKKKKKKKKKKZKZKKKKKZKK KKZKZKKKK 167 gsl sf gamma inc Q dbl expr dbl expr gsl sf gamma inc P dbl expr dbl expr gsl sf gamma inc dbl expr dbl expr gsl_sf_Inbeta dbl_expr dbl_expr gsl sf Inbeta sgn gsl sf beta dbl expr dbl expr gsl sf beta inc gsl_sf_gegenpoly_1 dbl_expr dbl_expr gsl_sf_gegenpoly_2 dbl_expr dbl_expr gsl_sf_gegenpoly_3 dbl_expr dbl_expr gsl sf gegenpoly n gsl sf gegenpoly array esl_sf_hyperg 0F1 dbl_expr dbl_expr esl_sf_hyperg 1F1_int int_expr int_expr dbl_expr esl_sf_hyperg 1F1 dbl_expr dbl_expr dbl_expr esl_sf_hyperg U_int int_expr int_expr dbl_expr esl_sf_hyperg U_int_el0 gsl_sf_hyperg_U dbl_expr dbl_expr dbl_expr gsl_sf_hyperg_U_e10 gsl_sf_hyperg_2F 1 dbl_expr dbl_expr dbl_expr dbl_expr gsl_sf_hyperg_2F 1_conj dbl_expr dbl_expr dbl_expr dbl_expr esl_sf_hyperg 2F1_renorm dbl_expr dbl_expr dbl_expr dbl_expr esl_sf_hyperg 2F1_conj_renorm dbl_expr dbl_expr dbl_expr dbl_expr gsl_sf_hyper
354. me a D dbg d old name new name d g old name nev name g glb h hdf hdr_pad nbr 1 path 0 o output file p path R r 7v old name new name v input file output filel DESCRIPTION ncrename renames netCDF dimensions variables attributes and groups Each object that has a name in the list of old names is renamed using the corresponding name in the list of new names All the new names must be unique Every old name must exist in the input file unless the old name is preceded by the period or dot character The validity of old name is not checked prior to the renaming Thus if old name is specified without the prefix that indicates the presence of old_name is optional and old name is not present in input file then ncrename will abort The new_name should never be prefixed by a or else the period will be included as part of the new name As of NCO version 4 4 6 released October 2014 the old_name and new_name arguments may include or be for groups partial or full group paths The OPTIONS and EXAMPLES show how to select specific variables whose attributes are to be renamed S Caveat lector Unforunately from 2007 present October 2014 the netCDF library versions 4 0 0 4 3 2 contained bugs or limitations that sometimes prevent NCO from cor rectly renaming coordinate variables dimensions and groups in netCDF4 files To our kno
355. ment to lt n the month indicator The month indicator affirms to NCO that the right most digits being manipulated in the generated filenames correspond to month numbers with January formatted as 01 and December as 12 Moreover it assumes digits to the left of the month are the year The full six argument n usage takes the form n file number digit number month increment max month min month yyyymm The yyyymm string is a clunky way can you think of a clearer way to tell NCO to enumerate files in year month mode When present yyyymm string causes NCO to automatically generate series of filenames whose right most two digits increment from min month by month increment up to max month and then the leftmost digits i e the year increment by one and the whole process is reapeated until the file number filenames are generated ncrcat n 3 6 1 12 1 198512 nc 198512 198502 nc ncrcat n 3 6 1 12 1 yyyymm 198512 nc 198512 198602 nc ncrcat n 3 6 1 12 12 yyyymm 198512 nc 198512 198712 nc The first command concatenates three files 198512 nc 198501 nc 198502 nc into the output file The second command concatenates three files 198512 nc 198601 nc 198602 nc The yyyymm indicator causes the left most digits to increment each time the right most two digits reach their maximum and then wrap The first command does not have the indicator so it is always 1985 The third command concatenates three files 198512 n
356. ments to X are always interpreted as floating point numbers i e as coordinate values rather than dimension indices so that these two commands produce identical results ncra X 0 180 30 30 in nc out nc ncra X 0 180 30 30 in nc out nc In contrast arguments to d require decimal places to be recognized as coordinates not indices see Section 3 15 Hyperslabs page 58 We recommend always using decimal points with X arguments to avoid confusion 3 22 Grid Generation Availability ncks Short options None Long options rgr key val multiple invocations allowed 70 NCO 4 5 3 alpha03 User Guide As of NCO version 4 5 2 August 2015 ncks generates accurate and complete SCRIP format gridfiles for select grid types including uniform capped and Gaussian rectangular latitude longitude grids The grids are stored in an external grid file All options pertinent to the grid geometry and metadata are passed to NCO via key value pairs prefixed by the rgr option Perhaps the two most important key value pair arguments are The text strings that describe the grid and name the file are important aids to convey the grid geometry to other users These arguments and their corresponding keys are the grid title grd_ttl and grid filename grid respectively The numbers of latitudes lat_nbr and longitudes lon_nbr are independent and together determine the grid storage size These four options should
357. monic LeFter Product Chunksize Equals Existing Chunksize Definition Chunksizes are copied from the input to the output file for every variable that is chunked in the input file Variables not chunked in the input file will be chunked with default mappings cnk map key values xst cnk_xst map xst Mnemonic EXiST Chunksize Balances 1D and N 1 D Access to N D Variable default for netCDF4 input Definition Chunksizes are chosen so that 1 D and N 1 D hyperslabs of 3 D variables e g point timeseries orn latitude longitude surfaces of 3 D fields both require approximately the number of chunks Hence their access time should be balanced Russ Rew explains the motivation and derivation for this strategy here http www unidata ucar edu blogs developer en entry chunking data choosing shapes cnk map key values rew cnk_rew map rew Mnemonic Russ REW Chunksizes use netCDF 4 defaults Definition Chunksizes are determined by the underlying netCDF library All variables selected by the current chunking policy have their chunksizes deter mined by netCDF library defaults The default algorithm netCDF uses to determine chunksizes has changed through the years and thus depends on the 90 NCO 4 5 3 alpha03 User Guide netCDF library version This map can be used to reset portions of previously chunked files to default chunking values cnk map key values nc4 cnk_nc4 map nc4 Mnemonic Net CDFZ
358. moo o 24 104 memory leaks ia EP iol 25 memory requirements 24 45 104 merging files os 4 bb dr ep 19 203 metadata rs ras a 208 metadata Optimizati0O 0oooooocccocommcm 27 metadata global ooooomooooo o o 198 207 MIDS caida 107 A ye tA E RUE 146 Michael Decker seeeeeeeseeeeeees 54 Michael Prather 0 0 e cece eee eee 119 Michael Schulz aminas 242 Microsoft eere rubent D eae Ss 8 10 Microsoft Visual Studio 200 0 10 Mike Folk vy ure eter Ras 11 Mike Pages eb RD ethan Aiden earte wate dhe 242 MIA nance Meee N mace ne C Mt nnd 107 mun scatueucatireks tEtem i rem eta 146 TOWER eine Rina ELERAPODRETAEARITRE ER 107 missing values 7A 84 109 183 201 208 missing values ncap2 cece ee eee 144 missing_value 74 84 105 233 MES Units ch one bee teak odes nk ew 79 82 MODIS nord 266 269 Modulus ssori ieue ie ein eb9g Dee b 178 monotonic CoOrdinateS ooocoooooococcccco oo 26 monthly average 200 00 245 263 monthly datan setos doren jv e 245 246 MOVE BTOUPS lt a eee ee rendi 50 MRO we poean Er ee gi e Mragee da eas 62 MSN eae ena Ree Ree eee 64 MS ey eed se Pe woke Be ad Ba Wedd Eie 124 MSY CP AA y eee ood ten i MI 34 37 MST OA voee eeDN HE S 34 MSS Ld td MEM RN au ced ri uut 33 multi file operators 24 31 33 195 198 225 228 multi hyperslab 0 020 e eee
359. mpressed unpacked sizes if they meet the rules for automatic internal promotion Put another way disabling auto promotion of single precision variables with f1t considerably reduces the RAM footprint of arithmetic operators The open_ram switch and switches that invoke it like ram all and diskless all incurs a RAM penalty These switches cause each input file to be copied to RAM upon opening Hence any operator invoking these switches utilizes an additional FT of RAM i e MS FT See Section 3 33 RAM disks page 104 for further details ncwa consumes between two and seven times the memory of a variable in order to process it Peak consumption occurs when storing simultaneously in memory one input variable one tally array one input weight one conformed working weight one weight tally one input mask one conformed working mask and one output variable When invoked the weighting and masking features contribute up to three sevenths and two sevenths of these requirements apiece If weights and masks are not specified i e no w or a options then ncwa requirements drop to MS lt 3VT one input variable one tally array and the output variable The above memory requirements must be multiplied by the number of threads thr nbr see Section 3 3 OpenMP Threading page 27 If this causes problems then reduce with t thr nbr the number of threads 2 9 2 Memory for ncap2 ncap2 has unique
360. mselves are distributed in the nco qt directory Using the freely available Cygwin formerly gnu win32 development environment the compilation process is very similar to installing NCO on a UNIX system Set the PVM_ARCH preprocessor token to WIN32 Note that defining WIN32 has the side effect of disabling Internet features of NCO see below NCO should now build like it does on UNIX The least portable section of the code is the use of standard UNIX and Internet protocols e g ftp rcp scp sftp getuid gethostname and header files lt arpa nameser h gt and resolv h Fortunately these UNIX y calls are only invoked by the single NCO subroutine which is responsible for retrieving files stored on remote systems see Section 3 7 Remote storage page 33 In order to support NCO on the Microsoft Windows platforms this single feature was disabled on Windows OS only This was required by Cygwin 18 x newer versions of Cygwin may support these protocols let me know if this is the case The NCO operators should behave identically on Windows and UNIX platforms in all other respects 1 4 Symbolic Links NCO relies on a common set of underlying algorithms To minimize duplication of source code multiple operators sometimes share the same underlying source This is accomplished by symbolic links from a single underlying executable program to one or more invoked executable names For example nces and ncrcat are symbolically linked to
361. must be chunked in all dimensions not just one Unless otherwise specified by the user the other fixed non record dimensions of record variables are assigned default chunk sizes The HDF5 layer does all this automati cally to optimize the on disk variable file storage geometry of record variables Do not be surprised to learn that files created without any explicit instructions to activate chunking nevertheless contain chunked variables 3 29 Compression 92 NCO 4 5 3 alpha03 User Guide 7 Availability ncbo ncecat nces ncflint ncks ncpdq ncra ncrcat ncwa Short options None Long options ppc var1 var2 prc precision preserving compression var1 var2 prc quantize var1 var2 prc NCO implements or accesses four different compression algorithms the standard lossless DEFLATE algorithm and three lossy compression algorithms All four algorithms reduce the on disk size of a dataset while sacrificing no lossless or a tolerable amount lossy of precision First NCO can access the lossless DEFLATE algorithm a combination of Lempel Ziv encoding and Huffman coding algorithm on any netCDF4 dataset see Section 3 30 Deflation page 101 Because it is lossless this algorithm re inflates deflated data to their full original precision This algorithm is accessed via the HDF5 library layer which itself calls the zlib library also used by gzip and is unavailable with netCDF3 The t
362. mutations which change the record dimension identity T he record dimension is re ordered like any other dimension This unique ncpdq capability makes it possible to concatenate files along any dimension See Section 2 6 1 Concatenation page 20 for a detailed example The record dimension is always the most slowly varying dimension in a record variable see Section 3 14 C and Fortran Index Conventions page 57 The specified re ordering fails if it requires creating more than one record dimension amongst all the output variables Two special cases of dimension re ordering and reversal deserve special mention First it may be desirable to completely reverse the storage order of a variable To do this include all the variable s dimensions in the dimension re order list in their original order and prefix each dimension name with the negative sign Second it may useful to transpose a variable s storage order e g from C to Fortran data storage order see Section 3 14 C and Fortran Index Conventions page 57 To do this include all the variable s dimensions in the dimension re order list in reversed order Explicit examples of these two techniques appear below NB fxm ncpdq documentation will evolve through Fall 2004 I will upload updates to documentation linked to by the NCO homepage ncpdq is a powerful operator and I am unfamiliar with the terminology needed to describe what ncpdq does Sequences sets sheesh I just know that it doe
363. n found in CF files ntrm ntrn ntrk ndbase nsbase nbdate nbsec mdt mhisf These variables contain scalar metadata such as the resolution of the host geophysical model and it makes no sense to change their values Furthermore the size and rank preserving arithmetic operators try not to operate on certain grid properties These operators are ncap2 ncbo nces ncflint and ncpdq when used for packing not for permutation These operators do not operate by default on i e add subtract pack etc the following variables ORO area datesec date gw hyai hyam hybi hybm lat bnds lon bnds msk These variables represent Gaussian weights land sea masks time fields hybrid pressure coefficients and latititude longitude boundaries We call these fields non coordinate grid properties Coordinate grid properties are easy to identify because they are coordinate variables such as latitude and longitude Users usually want all grid properties to remain unaltered in the output file To be treated as a grid property the variable name must exactly match a name in the above list or be a coordinate variable The handling of msk is exceptional in that any variable name beginning with the string msk_ is considered to be a mask and is thus preserved not operated on arithmetically As of NCO version 4 5 0 June 2015 NCO began to support behavior required for the DOE ACME program and we refer to these rules collectively as the ACME
364. n int_expr dbl_expr status gsl sf bessel In array int int double amp var out gsl sf bessel I0 scaled dbl expr gsl sf bessel I1 scaled dbl expr 164 gsl sf bessel In scaled e gsl sf bessel In scaled array esl_sf_bessel_K0_e esl_sf_bessel_K1_e esl_sf_bessel_Kn_e esl_sf_bessel_Kn_array esl_sf_bessel_K0_scaled_e esl_sf_bessel_K1_scaled_e esl_sf_bessel_Kn_scaled_e esl_sf_bessel_Kn_scaled_array esl_sf_bessel_j0_e esl_sf_bessel_j1_e esl_sf_bessel_j2_e gsl_sf_bessel_jl_e gsl_sf_bessel_jl_array gsl_sf_bessel_jl_steed_array esl_sf_bessel_y0_e esl_sf_bessel_y1_e esl_sf_bessel_y2_e esl_sf_bessel_yl_e gsl_sf_bessel_yl_array gsl_sf_bessel_i0_scaled_e gsl_sf_bessel_il_scaled_e gsl_sf_bessel_i2_scaled_e gsl_sf_bessel_il_scaled_e esl_sf_bessel_il_scaled_array esl_sf_bessel_k0_scaled_e esl_sf_bessel_k1_scaled_e esl_sf_bessel_k2_scaled_e esl_sf bessel kl scaled e gsl sf bessel kl scaled array esl_sf_bessel_Jnu_e gsl sf bessel Ynu e gsl sf bessel sequence Jnu e gsl sf bessel Inu scaled e gsl sf bessel Inu e gsl sf bessel Knu scaled e esl_sf_bessel_Knu_e esl_sf_bessel_InKnu_e esl_sf_bessel_zero_J0_e esl_sf_bessel_zero_J1_e esl_sf_bessel_zero_Jnu_e esl_sf_clausen_e esl_sf_hydrogenicR_1_e gsl sf hydrogenicR e gsl sf coulomb wave FG e gsl sf coulomb wave F array ZAZAZAZKAZKRKKKK KK AK KKK KKK KK KKK KKK KKK KKK KKK KKK KKK KKK NCO 4 5 3 alpha03 User Guide gsl_sf_bessel_In_scaled int_expr dbl_ex
365. n modern Linux systems the block size defaults to 8192 B The GLADE filesystem at NCAR has a block size of 512 kB 104 NCO 4 5 3 alpha03 User Guide ncks INFO nc__open opened file with buffer size 2097152 bytes 3 33 RAM disks Availability All operators Short options Long options ram all create ram open_ram diskless all As of NCO version 4 2 1 August 2012 NCO supports the use of diskless files aka RAM disks for file access and creation Two independent switches open_ram and create ram control this feature Before describing the specifics of these switches we describe why many NCO operations will not benefit from them Essentially reading writing from to RAM rather than disk only hastens the task when reads writes to disk are avoided Most NCO operations are simple enough that they require a single read from write to disk for every block of input output Diskless access does not change this but it does add an extra read from write to RAM However this extra RAM write read does avoid contention for limited system resources like disk head access Operators which may benefit from RAM disks include ncwa which may need to read weighting variables multiple times the multi file operators ncra ncrcat and ncecat which may try to write output at least once per input file and ncap2 scripts which may be arbitrarily long and convoluted The open_ram switch causes input files to copied
366. n nc foo nc grep MD5 ncrcat INFO MD5 wnd_spd 74699bb0a72b7f16456badb2c995f1a1 ncrcat INFO MD5 digests of RAM and disk contents for wnd_spd agree Regardless of the debugging level an error is returned when the digests of the variable read from the source file and from the output file disagree These rules are evolving and as NCO pays more attention to data integrity We welcome feedback and suggestions from users 3 32 Buffer sizes Availability All operators Short options Long options bfr_sz_hnt buffer size hint As of NCO version 4 2 0 May 2012 NCO allows the user to request specific buffer sizes to allocate for reading and writing files This buffer size determines how many system calls the netCDF layer must invoke to read and write files By default netCDF uses the preferred I O block size returned as the st_blksize member of the stat structure returned by the stat system call Otherwise netCDF uses twice the system pagesize Larger sizes can increase access speed by reducing the number of system calls netCDF makes to read write data from to disk Because netCDF cannot guarantee the buffer size request will be met the actual buffer size granted by the system is printed as an INFO statement Request 2 MB file buffer instead of default 8 kB buffer gt ncks 0 D 3 bfr sz 2097152 nco data in nc foo nc ncks INFO nc open will request file buffer size 2097152 bytes 25 O
367. n nc in nc out nc ncatted a units prs sfc o c millibar out nc Chapter 4 Reference Manual 203 4 7 ncks netCDF Kitchen Sink SYNTAX ncks 3 4 5 6 7 A a b binary file C c ca1 cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min lL max stridell F fix rec dmn dim G gpe_dsc g grpl 1 glb grp xtr var xcl H h hdn hdr pad nbr L df1 1vl 1 path M m map map file md5 mk rec dmn dim no blank no tmp f1 0 o output file P p path ppc Q q R r rad ram a11 rgr rnr wgt s format u unn V v var X x xm1 input file output filel DESCRIPTION The nickname kitchen sink is a catch all because ncks combines most features of ncdump and nccopy with extra features to extract hyperslab multi slab sub set and translate into one versatile utility ncks extracts a subset of the data from input file regrids it according to map file if specified then writes in netCDF format to output file and optionally writes it in flat binary format to binary file and optionally prints it to screen ncks prints netCDF input data in ASCII CDL or NcML text formats stdout like an extended version of ncdump By default ncks prints data in a tabular format intended to be easy to search for the data you want one datu
368. n sz cnk plc plc cnk scl sz D dbg d dim min max stride F fix rec crd G gpe dscl g grpl 1 glb h hdr pad nbr i var val3 L dfl_1v1 1 path no tmp f1 0 o file 3 p path ppe R r ram_al1 t thr nbr unn v var 1 w wgtil wgt2 X x file 1 file 2 file 3 DESCRIPTION ncflint creates an output file that is a linear combination of the input files This linear combination is a weighted average a normalized weighted average or an interpolation of the input files Coordinate variables are not acted upon in any case they are simply copied from file 1 There are two conceptually distinct methods of using ncflint The first method is to specify the weight each input file contributes to the output file In this method the value val3 of a variable in the output file file 3 is determined from its values vall and val2 in the two input files according to val3 wgtl x vall wgt2 x val2 Here at least wgtl and optionally wgt2 are specified on the command line with the w or weight or wgt var switch If only wgtl is specified then wgt2 is automatically computed as wgt2 1 wgtl Note that weights larger than 1 are allowed Thus it is possible to specify wgtl 2 and wgt2 3 One can use this functionality to multiply all the values in a given file by a constant The second method of using ncflint is to
369. nc x 10 nc contain time evolving datasets from spatially adjacent regions The time and spatial coordinates are time and x respectively Initially the record dimension is time Our goal is to create a single file that contains joins all the spatially adjacent regions into one single time evolving dataset for idx in 01 02 03 04 05 06 07 08 09 10 do Bourne Shell ncpdq a x time x_ idx nc foo_ idx nc Make x record dimension done ncrcat foo 7 nc out nc Concatenate along x ncpdq a time x out nc out nc Revert to time as record dimension Note that ncrcat will not concatenate fixed length variables whereas ncecat concate nates both fixed length and record variables along a new record variable To conserve system memory use ncrcat where possible 2 6 2 Averagers nces ncra and ncwa The differences between the averagers ncra and nces are analogous to the differences be tween the concatenators ncra is designed for averaging record variables from at least one file while nces is designed for averaging fixed length variables from multiple files ncra per forms a simple arithmetic average over the record dimension of all the input files with each record having an equal weight in the average nces performs a simple arithmetic average of all the input files with each file having an equal weight in the average Note that ncra cannot average fixed length variables but nces can average both fixed length and record variables To conse
370. nc ncks foo nc EXAMPLES Given file 85 0112 nc netcdf 85 0112 4 dimensions 238 NCO 4 5 3 alpha03 User Guide lat 64 lev 18 lon 128 time UNLIMITED 12 currently variables float lat lat float lev lev float lon lon float time time float scalar var float three dmn var lat lev lon float two dmn var lat lev float mask lat lon float gw lat Average all variables in in nc over all dimensions and store results in out nc ncwa in nc out nc All variables in in nc are reduced to scalars in out nc since ncwa averages over all dimen sions unless otherwise specified with a Store the zonal longitudinal mean of in nc in out nc ncwa a lon in nc outi nc ncwa a lon b in nc out2 nc The first command turns lon into a scalar and the second retains lon as a degenerate dimension in all variables 4 ncks C H v lon outi nc lon 135 ncks C H v lon out2 nc lon 0 135 In either case the tally is simply the size of lon i e 180 for the 85_0112 nc file described by the sample header above Compute the meridional latitudinal mean with values weighted by the corresponding element of gw ncwa w gw a lat in nc out nc Here the tally is simply the size of lat or 64 The sum of the Gaussian weights is 2 0 Compute the area mean over the tropical Pacific ncwa w gw a lat lon d 1at 20 20 d lon 120 270 in nc out nc Here the tally is 64 x 128
371. nc out nc OK Quotes protect in lat ncpdq a lon lat in nc out nc OK No whitespace before ncpdq rdr lat in nc out nc Preferred Uses not whitespace To create the mathematical transpose of a variable place all its dimensions in the dimen sion re order list in reversed order This example creates the transpose of three dmn var ncpdq a lon lev lat v three dmn var in nc out nc ncks C v three dmn var in nc lat 0 2 90 lev 0 100 lon 0 0 three dmn var 0 0 lat 0 2 90 lev 0 100 lon 1 90 three dmn var 1 1 lat 0 2 90 lev 0 100 lon 2 180 three dmn var 2 2 lat 1 290 lev 2 1000 lon 1 90 three dmn var 21 21 lat 1 290 lev 2 1000 lon 2 180 three dmn var 22 22 lat 1 290 lev 2 1000 lon 3 270 three dmn var 23 23 4 ncks C v three dmn var out nc lon 0 20 lev 0 100 lat 0 90 three dmn var 0 0 lon 0 0 lev 0 100 lat 1 90 three dmn var 1 12 lon 0 20 lev 1 500 lat 0 90 three dmn var 2 4 lon 3 270 lev 1 500 lat 1 90 three dmn var 21 19 lon 3 270 lev 2 1000 lat 0 90 three dmn var 22 11 lon 3 270 lev 2 1000 lat 1 90 three dmn var 23 23 To completely reverse the storage order of a variable include all its dimensions in the re order list each prefixed by a negative sign This example reverses the storage order of three dmn var ncpdq a lat lev lon v three dmn var in nc out nc 4 ncks C v three dmn var in nc lat 0 2 90 lev 0 100 lon 0 0 three dmn var 0 0 224 NCO
372. ncatenating record variables while ncecat is designed for concatenating fixed length variables Consider five files 85 nc 86 nc 89 nc each con taining a year s worth of data Say you wish to create from them a single file 8589 nc containing all the data i e spanning all five years If the annual files make use of the same record variable then ncrcat will do the job nicely with e g ncrcat 8 nc 8589 nc The number of records in the input files is arbitrary and can vary from file to file See Section 4 10 ncrcat netCDF Record Concatenator page 228 for a complete description of ncrcat However suppose the annual files have no record variable and thus their data are all fixed length For example the files may not be conceptually sequential but rather members of the same group or ensemble Members of an ensemble may have no reason to contain a record dimension ncecat will create a new record dimension named record by default with which to glue together the individual files into the single ensemble file If ncecat is used on files which contain an existing record dimension that record dimension is converted to a fixed length dimension of the same name and a new record dimension named record is created Consider five realizations 85a nc 85b nc 85e nc of 1985 predictions from the same climate model Then ncecat 85 nc 85 ens nc glues together the individual realizations into the single file 85 ens nc If an input variable w
373. ndicates that a UDUnits units conversion is requested Some date formats like YYYYMMDD that are valid in UDUnits are ambiguous to NCO because it cannot distinguish a purely numerical date i e no dashes or text characters in it from a coordinate or index value d time 1918 11 11 Interpreted as the date November 11 1918 d time 19181111 Interpreted as time dimension index 19181111 d time 19181111 Interpreted as time coordinate value 19181111 0 Hence use the Y Y Y Y MM DD format rather than Y YY YMMDD for dates As of version 4 0 0 January 2010 NCO supports some calendar attributes specified by the CF conventions Supported types 365 day noleap 360 day gregorian standard Unsupported types 366 day all leap proleptic gregorian julian none Unsupported types default to mixed Gregorian Julian as defined by UDUnits 82 NCO 4 5 3 alpha03 User Guide An Example Consider the following netCDF variable variables double lon cal lon cal lon cal long name lon cal lon cal units days since 1964 2 28 0 0 0 lon cal calendar 365 day data lon cal 1 2 3 4 5 6 7 8 9 10 ncks v lon cal d lon cal 1964 3 1 0 00 0 0 1964 3 4 00 00 0 0 results in lon cal 1 2 3 4 netCDF variables should always be stored with MKS i e God s units so that appli cation programs may assume MKS dimensions apply to all input variables The UDUnits feature is intended to allev
374. nds for such structured grids parallel the oft used temporal bounds dimen sion nbnd 2 and variable time bnds Bounds are attached to the hor izontal spatial dimensions via their bounds attributes Change the spatial bounds dimension with the option rgr bnd bnd_nm Rename the spatial bounds variables with the options rgr lat bnd 1at bnd nm and rgr lon bnd 1lon bnd nm Unstructured Grid Bounds Variables bnd lat bnd lon bnd Unstructured grids with 1D coordinates use the dimension bnd_nm which de faults to nv number of vertices for the spatial bounds variables lat_bnd_nm and 76 NCO 4 5 3 alpha03 User Guide lon bnd nm which default to lat vertices and lon vertices respectively It may be impossible to re use the temporal bounds dimension often nbnd for unstructure grids because the gridcells are not rectangles and thus require specification of all vertices for each gridpoint rather than only two parallel interfaces per dimension These bounds are attached to the horizontal spatial dimensions via their bounds attributes Change the spatial bounds dimension with the option rgr bnd bnd nm Rename the spatial bounds variables with the options rgr lat bnd lat bnd nm and rgr lon bnd lon bnd nm The temporal bounds dimension in unstructured grid output remains as in the input file usually nbnd Gridcell Area area The variable area nm which defaults to area is always re created in the output file
375. ne switch which controls the normalization of the averages appearing in the output file Short option N or long options nmr or numerator prevents ncwa from dividing the weighted sum of the variable the numerator in the averaging expression by the weighted sum of the weights the denominator in the averaging expression Thus N tells ncwa to return just the numerator of the arithmetic expression defining the operation see Section 3 35 Operation Types page 107 With this normalization option ncwa can integrate variables Averages are first com puted as sums and then normalized to obtain the average The original sum i e the numerator of the expression in Section 3 35 Operation Types page 107 is output if de fault normalization is turned off with N This sum is the integral not the average over the specified with a or all if none are specified dimensions The weighting variable if specified with w plays the role of the differential increment and thus permits more so phisticated integrals i e weighted sums to be output For example consider the variable lev where lev 100 500 1000 weighted by the weight lev_wgt where lev_wgt 10 2 1 The vertical integral of lev weighted by lev_wgt is the dot product of lev and lev wgt That this is is 3000 0 can be seen by inspection and verified with the integration command ncwa N a lev v lev w lev wgt in nc foo
376. ns between the various statistics tools and concatenators As a multi file operator ncecat will read the list of input files from stdin if they are not specified as positional arguments on the command line see Section 2 7 Large Numbers of Files page 21 Suppress global metadata copying By default NCO s multi file operators copy the global metadata from the first input file into output file This helps to preserve the provenance of the output data However the use of metadata is burgeoning and sometimes one en counters files with excessive amounts of extraneous metadata Extracting small bits of data from such files leads to output files which are much larger than necessary due to the automatically copied metadata ncecat supports turning off the default copying of global metadata via the M switch or its long option equivalents no_glb_mtd and suppress global metadata Consider five realizations 85a nc 85b nc 85e nc of 1985 predictions from the same climate model Then ncecat 85 nc 85 ens nc glues together the individual realizations into the single file 85 ens nc If an input variable was dimensioned lat lon it will by default have dimensions record lat lon in the output file A restriction of ncecat is that the hyperslabs of the processed variables must be the same from file to file Normally this means all the input files are the same size and contain data on different realizations of the same variables
377. nstead store output in a new child group of the parent created by attaching the suffix to the parent group s name e g nsm_sfx _avg would store results in the output group cesm cesm avg nces nsm grp mdli nc mdl2 nc mdl3 nc out nc nces nsm grp nsm sfx avg mdli nc mdl2 nc mdl3 nc out nc See Section 2 6 Statistics vs Concatenation page 20 for a description of the distinc tions between the statistics tools and concatenators As a multi file operator nces will read the list of input files from stdin if they are not specified as positional arguments on the command line see Section 2 7 Large Numbers of Files page 21 Like ncra and ncwa nces treats coordinate variables as a special case Coordinate variables are assumed to be the same in all ensemble members so nces simply copies the coordinate variables that appear in ensemble members directly to the output file This has the same effect as averaging the coordinate variable across the ensemble yet does not incur the time or precision penalties of actually averaging them ncra and ncwa allow coordinate variables to be processed only by the linear average operation regardless of the arithmetic operation type performed on the non coordinate variables see Section 3 35 Operation Types page 107 Thus it can be said that the three operators ncra ncwa and nces all average coordinate variables even though nces simply copies them All other requested arithmetic o
378. o sourceforge net nco html Hyperslabs of ncbo the script will be shortened bin bash Monthly cycle of each ensemble member in Greenland Input file from cmb_fl_grpsh sn_LImon_all mdl_all xpt_all nsm_199001 200512 nc Online http nco sourceforge net nco htmlitMonthly Cycle 266 NCO 4 5 3 alpha03 User Guide Execute this script in command line bash mcc_grp sh Input and output directory drc data grp Constants pfx sn_LImon_all mdl_all xpt_all nsm_200001 200512 Greenland ncwa w gw d lat 60 0 75 0 d lon 300 0 340 0 a lat lon drc pfx nc drc 4 pfx grl nc Anomaly from annual average of each year for yyyy in 2000 2005 do Annual average ncwa 0 d time yyyyl 01 O1 yyyyl 12 31 drc pfx _grl nc drc pfx _gr1_ yyyy nc Anomaly ncbo 0 op_typ d time yyyy 01 01 yyyy 12 31 drc pfx _grl nc drc pfx _gr1l_ yyyy ne drc pfx _gr1_ yyyy _anm nc done Monthly cycle for moy in 1 12 do mm printf 02d moy Change to 2 digit format ncra 0 d time 2000 mm 01 12 drc pfx _grl_ _anm nc drc pfx _gr1_ mm _anm nc done Concatenate 12 months together ncrcat 0 drc pfx _gr1_ _anm nc drc pfx _grl_mth_anm nc 7 5 Regrid MODIS Data In order to compare the results between MODIS and CMIP5 models one usually regrids one or both datasets so that the spatial resolution
379. o be printed they are not printed by default Also retain_all_dimensions orphan_dimensions and rph_dmn s format String format for text output Accepts C language escape sequences and printf formats Also string and sng_fmt secret Print summary of ncks hidden features These hidden or secret features are used mainly by developers They are not supported for general use and may change at any time This demonstrates conclusively that I cannot keep a secret Also ssh and scr u Toggle the printing of a variable s units attribute if any with its values Also units eV Print variable values only Do not print variable and dimension names indices and coordinate values when printing arrays 210 NCO 4 5 3 alpha03 User Guide zender roulee ncks V v three dmn rec var C H nco data in nc 1 6 See also the Q option which prints variable names and indices but not di mension names indices or coordinate values when printing arrays Using V is the same as specifying Q no nm prn xml ncml As of NCO version 4 3 3 July 2013 ncks can print extracted metadata to screen i e stdout as XML in NcML the netCDF Markup Language ncks supports for XML more completely than of ncdump x With ncks one can translate entire netCDF3 and netCDF4 files into NcML including metadata and data using all NCO s subsetting and hype
380. o differences between in nc and out nc First the history global attribute see Section 3 39 History Attribute page 122 will contain the command used to create out nc Second the variables in out nc will be defined in alphabetical order Of course the internal storage of variable in a netCDF file should be transparent to the user but there are cases when alphabetizing a file is useful see description of a switch Copy all global attributes and no variables from in nc to out nc ncks A x nco data in nc out nc The x switch tells NCO to use the complement of the extraction list see Section 3 11 Subsetting Files page 43 Since no extraction list is explicitly specified with v the default is to extract all variables The complement of all variables is no variables Without any variables to extract the append A command see Section 2 4 Appending Variables page 19 has only to extract and copy i e append global attributes to the output file Copy append metadata not data from variables in one file to variables in a second file When copying subsetting appending files as opposed to printing them the copying of data variable metadata and global group metadata are now turned OFF by H m and 214 NCO 4 5 3 alpha03 User Guide M respectively This is the opposite sense in which these switches work when printing a file One can use these switches to easily replace data or meta
381. oesn t conform to varl then an attempt is made to make it conform to varl If expl is an attribute it must have unity size or else have the same number of elements as varl If exprl has a different type to varl the it is converted to the varl type zi four one 10 zi 14 four 14 one 10 time 2 time 1 0 1 2 3 4 5 6 7 8 Increment Decrement Operators These work in a similar fashion to their regular C counterparts If say the variable four is input only then the statement four effectively means read four from input increment each element by one then write the new values to Output Example n2 four n2 5 four 5 n3 one 20 n3 21 one 0 n4 time n4 time 0 1 2 3 4 5 6 7 8 9 Conditional Operator expl exp2 exp3 The conditional operator or ternary Operator is a succinct way of writing an if then else If expl evaluates to true then exp2 is returned else exp3 is returned Example weight avg weight avg weight avgOunits weight avg 1 kilo kilos PS nw PS PS min gt 100000 100000 0 Clipping Operators Less than Clipping For arrays the less than selection operator selects all values in the left operand that are less than the corresponding value in the right operand If the value of the left side is greater than or equal to the corresponding value of the right side then the right side value is placed in the result Greater than Clipping For arrays the greate
382. of many which will accomplish the same result First use ncwa to create the five year mean ncwa a time 8501 8912 nc 8589 nc Next use ncbo to create a file containing the difference of each month s data from the five year mean ncbo 8501 8912 nc 8589 nc t anm 8501 8912 nc Now use ncks to group together the five January anomalies in one file and use ncra to create the average anomaly for all five Januarys These commands are embedded in a shell loop so they are repeated for all twelve months for idx in 1 12 do Bash Shell version 3 0 idx printf 02d idx Zero pad to preserve order ncks F d time idx 12 t anm 8501 8912 nc foo idx ncra foo idx t anm 8589 1 idx nc done for idx in 01 02 03 04 05 06 07 08 09 10 11 12 do Bourne Shell ncks F d time idx 12 t anm 8501 8912 nc foo idx ncra foo idx t anm 8589 1 idx nc done foreach idx 01 02 03 04 05 06 07 08 09 10 11 12 C Shell ncks F d time idx 12 t anm 8501 8912 nc foo idx ncra foo idx t anm 8589 1 idx nc end Note that ncra understands the stride argument so the two commands inside the loop may be combined into the single command ncra F d time idx 12 t anm 8501 8912 nc foo idx Finally use ncrcat to concatenate the 12 average monthly anomaly files into one twelve record file which contains the entire seasonal cycle of the monthly anomalies ncrcat t anm 8589 nc t anm 8589 0112 nc 194 NCO 4 5 3 alp
383. of ncwa is to arithmetically average every numerical variable over all dimensions and to produce a scalar result for each Averaged dimensions are by default eliminated as dimensions Their corresponding coordinates if any are output as scalar variables The b switch and its long option equivalents rdd and retain degenerate dimensions causes ncwa to retain aver aged dimensions as degenerate size 1 dimensions This maintains the association between a dimension or coordinate and variables after averaging and simplifies for instance later concatenation along the degenerate dimension To average variables over only a subset of their dimensions specify these dimensions in a comma separated list following a e g a time lat lon As with all arithmetic operators the operation may be restricted to an arbitrary hyperslab by employing the d option see Section 3 15 Hyperslabs page 58 ncwa also handles values matching the variable s _FillValue attribute correctly Moreover ncwa understands how to manipulate user specified weights masks and normalization options With these options ncwa can compute sophisticated averages and integrals from the command line mask_var and weight if specified are broadcast to conform to the variables being aver aged The rank of variables is reduced by the number of dimensions which they are averaged over Thus arrays which are one dimensional in the input file an
384. of what we term autoconversion NCO employs three algorithms to downconvert netCDF4 to netCDF3 1 Autoconversion of atomic types Autoconversion automatically promotes NC UBYTE to NC SHORT and NC USHORT to NC INT It automatically demotes the three types NC_ UINT NC UINT64 and NC INT64 to NC INT And it converts NC STRING to NC CHAR All numeric conversions work for attributes and variables of any rank Two numeric types NC UBYTE and NC USHORT are promoted to types with greater range and greater storage This extra range is often not used so promotion perhaps conveys the wrong impression However promotion never truncates values or loses data this perhaps justifies the extra storage Three numeric types NC UINT NC UINT64 and NC INT64 12 The switches 5 5 and pnetcdf are reserved for PnetCDF files i e NC FORMAT PNETCDF Such files are similar to netCDF3 classic files yet also support 64 bit offsets and the additional netCDF4 atomic types 42 NCO 4 5 3 alpha03 User Guide are demoted Since the input range is larger than the output range demotion can result in numeric truncation and thus loss of data In such cases it would possible to convert the data to floating point values instead If this feature interests you please be the squeaky wheel and let us know String conversions to NC CHAR work for all attributes but not for variables This is because attributes are at most one dimensi
385. oint format No exceptions are raised for inexact conversions Example nearbyint 0 1 0 0 Power Value of x is raised to the power of y Exceptions are raised for do main errors Due to type limitations in the C language pow function integer arguments are promoted see Section 3 36 Type Conversion page 113 to type NC FLOAT before evaluation Example pow 2 3 8 Round exactly Nearest integer to x is returned in floating point format Ex ceptions are raised for inexact conversions Example rint 0 1 0 Round Nearest integer to x is returned in floating point format Round halfway cases away from zero regardless of current IEEE rounding direction Example round 0 5 1 0 Sine Sine of x where x is specified in radians Example sin 1 57079632679489661922 1 0 180 sinh x sqrt x tan x tanh x trunc x NCO 4 5 3 alpha03 User Guide Hyperbolic sine Hyperbolic sine of x where x is specified in radians Example sinh 1 0 1 1752 Square Root Square Root of x yx Example sqrt 4 0 2 0 Tangent langent of x where x is specified in radians Example tan 0 78539816339744830961 1 0 Hyperbolic tangent Hyperbolic tangent of x where x is specified in radians Example tanh 1 0 0 761594155956 Truncate Nearest integer to x is returned in floating point format Round half way cases toward zero regardless of current IEEE rounding direction Example trunc 0 5 0 0 The complete list of math
386. om sorted data a1 2 5 n3 gsl stats median from sorted data a1 2 Following statements are NOT equvalent n4 gsl stats kurtosis a1 3 2 n4 gsl stats kurtosis a1 3 default n 4 The following example illustrates some of the weighted functions The data are randomly generated In this case the value of the weight for each datum is either 0 0 or 1 0 defdim ri 2000 data ri 21 0 Fill with random numbers 0 0 10 0 data 10 0 gsl rng uniform data Create a weighting variable weight data gt 4 0 wmean gsl_stats_wmean weight 1 data 1 r1 size print wmean wsd gsl_stats_wsd weight 1 data 1 r1 size print wsd number of values in data that are greater than 4 weight size weight total print weight size print min max of data dmin data gsl stats min dmax data gsl stats max 174 NCO 4 5 3 alpha03 User Guide print dmin print dmax 4 1 23 GSL random number generation The GSL library has a large number of random number generators In addition there are a large set of functions for turning uniform random numbers into discrete or continuous probabilty distributions The random number generator algorithms vary in terms of quality numbers output speed of execution and maximum number output For more information see the GSL documentation The algorithm and seed are set via environment variables these are picked up by the ncap2 code Setup The number algorithm is set by the environm
387. ommended method is now to use ncks fix rec dmn yet it is still instructive to present the original procedure as it shows how multiple operators can achieve the same ends by different means ncecat 0 in nc out nc Add degenerate record dimension named record ncpdq 0 a time record out nc out nc Switch record and time ncwa a record out nc out nc Remove degenerate record The first step creates a degenerate size equals one record dimension named by default record The second step swaps the ordering of the dimensions named time and record Since time now occupies the position of the first least rapidly varying dimension it be comes the record dimension The dimension named record is no longer a record dimension The third step averages over this degenerate record dimension Averaging over a degener ate dimension does not alter the data The ordering of other dimensions in the file lat lon etc is immaterial to this procedure See Section 4 5 ncecat netCDF Ensemble Concate nator page 197 and Section 4 7 ncks netCDF Kitchen Sink page 203 for other methods of changing variable dimensionality including the record dimension Chapter 4 Reference Manual 225 4 9 ncra netCDF Record Averager SYNTAX ncra 3 4 6 7 A C cl cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stride subcycle1111 F G gpe dsc g grp
388. omplete NCO source distribution is currently distributed as a compressed tarfile from http sf net projects nco and from http dust ess uci edu nco nco tar gz The compressed tarfile must be uncompressed and untarred before building NCO Uncompress the file with gunzip nco tar gz Extract the source files from the resulting tarfile with tar xvf nco tar GNU tar lets you perform both operations in one step with tar xvzf nco tar gz The documentation for NCO is called the NCO User Guide The User Guide is available in PDF Postscript HTML DVI TpXinfo and Info formats These formats are included in the source distribution in the files nco pdf nco ps nco html nco dvi nco texi and nco info respectively All the documentation descends from a single source file nco texi Hence the documentation in every format is very similar However some of the complex mathematical expressions needed to describe ncwa can only be displayed in DVI Postscript and PDF formats A complete list of papers and publications on about NCO is available on the NCO home page Most of these are freely available The primary refereed publications are ZeM06 and Zen08 These contain copyright restrictions which limit their redistribution but they are freely available in preprint form from the NCO If you want to quickly see what the latest improvements in NCO are without downloading the entire source distribution visit the NCO homepage at http
389. on ratios reported are relative to the size of the original file as distributed not as optimally losslessly compressed A dash indicates the associated compression feature was not employed dstmch90_clm nc Type LLC PPC Pck Size Flags and Notes N3 5 34 7 100 0 Original is not compressed N3 Bl 28 9 83 2 bzip2 1 N3 B9 29 3 84 4 bzip2 9 N7 35 0 101 0 N7 1 28 2 81 3 L 1 N7 9 28 0 80 8 L9 N7 Y 17 6 50 9 ncpdq L O N7 de say N 7 9 22 8 ncpdq L 1 N7 1 T 28 2 81 3 ppc default 7 N7 1 or 27 9 80 6 ppc default 6 N7 1 5 25 9 74 6 ppc default 5 N7 1 4 22 3 64 3 ppc default 4 N7 1 3 18 9 54 6 ppc default 3 N7 1 2 14 5 43 2 ppc default 2 N7 1 1 10 0 29 0 ppc default 1 b1850c5cn_doe_polar_merged_0_cesm1_2_0_HD MAM4 tun2b hp e003 cam h0 0001 01 nc Type LLC PPC Pck Size Flags and Notes N3 119 8 100 0 Original is not compressed N3 B1 84 2 70 3 bzip2 1 N3 B9 84 8 70 9 bzip2 9 N7 120 5 100 7 N7 1 82 6 69 0 L1 N7 9 82 1 68 6 L9 100 N7 N7 N7 N7 N7 N7 N7 N7 N7 RRRR BR SR e E e NUU PB ODN 60 26 82 81 TT 69 59 49 38 YNOWON OD ON 50 21 69 68 64 57 49 41 31 oOoOWNA A FO ON ncpdq ncpdq ppe ppe ppe ppe ppe ppe ppe NCO 4 5 3 alpha03 User Guide L 0 L 1 default 7 default 6 default 5 default 4 default 3 default 2 default 1
390. on 3 12 Subsetting Coordinate Variables page 47 to ncwa to prevent any scalar time variable from being written to the output file Knowing when to use ncwa a time rather than the default ncra for time averaging takes well time 4 1 25 Intrinsic mathematical methods ncap2 supports the standard mathematical functions supplied with most operating systems Standard calculator notation is used for addition subtraction multiplication division exponentiation and modulus X The available elementary mathematical functions are abs x Absolute value Absolute value of x x Example abs 1 1 acos x Arc cosine Arc cosine of x where x is specified in radians Example acos 1 0 0 0 acosh x Hyperbolic arc cosine Hyperbolic arc cosine of x where x is specified in radians Example acosh 1 0 0 0 asin x Arc sine Arc sine of x where x is specified in radians Example asin 1 0 1 57079632679489661922 asinh x Hyperbolic arc sine Hyperbolic arc sine of x where x is specified in radians Example asinh 1 0 0 88137358702 atan x Arc tangent Arc tangent of x where x is specified in radians between 7 2 and 7 2 Example atan 1 0 0 78539816339744830961 atan2 y x Arc tangent2 Arc tangent of y x Chapter 4 Reference Manual 179 atanh x ceil x cos x cosh x erf x erfc x exp x floor x gamma x Hyperbolic arc tangent Hyperbolic arc tangent of x where x is specified in radi
391. onal and may be of any size whereas variables require gridded dimensions that usually do not fit the ragged sizes of text strings Hence scalar NC STRING attributes are correctly converted to and stored as NC CHAR attributes in the netCDF3 output file but NC STRING variables are not correctly converted If this limitation annoys or enrages you please let us know by being the squeaky wheel 2 Convert multiple record dimensions to fixed size dimensions Many netCDF4 and HDF5 datasets have multiple unlimited dimensions Since a netCDF3 file may have at most one unlimited dimension all but possibly one unlimited dimension from the input file must be converted to fixed length dimensions prior to storing netCDF4 input as netCDF3 output By invoking fix rec dmn all the user ensures the output file will adhere to netCDF3 conventions and the user need not know the names of the specific record dimensions to fix See Section 4 7 ncks netCDF Kitchen Sink page 203 for a description of the fix rec dmn option 3 Flattening removal of groups Many netCDF4 and HDF5 datasets have group hierar chies Since a netCDF3 file may not have any groups groups in the input file must be removed This is also called flattening the hierarchical file See Section 3 13 Group Path Editing page 48 for a description of the GPE option G to flatten files Putting the three algorithms together one sees that the recipe to convert netCDF4 to netCDF4 becomes
392. ondition is True the corresponding LHS variable element is re assigned to its partner 150 NCO 4 5 3 alpha03 User Guide element on the RHS In the elsewhere part the mask is logically inverted and the assign process proceeds as before e If the mask dimensions are a subset of the LHS variable s dimensions then it is made to conform if it cannot be made to conform then script execution halts e Missing values in the mask evaluate to False in the where code block statement and to True in the elsewhere block statement LHS variable elements set to missing value are not re assigned For these reasons do not explicitly reference missing values in the masking condition e g where foo foo get missingO foo 1 will not work as expected Example Consider the variables float lon 2D rct lat lon and float var msk lat lon Suppose we wish to multiply by two the elements for which var msk equals 1 where var msk 1 lon 2D rct 2 1lon 2D rct Suppose that we have the variable int RDM time and that we want to set its values less than 8 or greater than 80 to 0 where RDM 8 RDM gt 80 RDM 0 Consider irregu larly gridded data described using rank 2 coordinates double lat south_north east_ west double lon south north east west double temperature south_north east_ west This type of structure is often found in regional weather climate model such as WRF output and in satellite swath data For this reason we call it Swath
393. ons operation op_typ The y op_typ switch allows specification of many different types of operations Set op_typ to the abbreviated key for the corresponding operation avg Mean value sqravg Square of the mean avgsqr Mean of sum of squares max Maximum value min Minimum value mabs Maximum absolute value mebs Mean absolute value mabs Minimum absolute value rms Root mean square normalized by N rmssdn Root mean square normalized by N 1 sqrt Square root of the mean ttl Sum of values NCO assumes coordinate variables represent grid axes e g longitude The only rank reduction which makes sense for coordinate variables is averaging Hence NCO implements the operation type requested with y on all non coordinate variables not on coordinate variables When an operation requires a coordinate variable to be reduced in rank i e from one dimension to a scalar or from one dimension to a degenerate single value array then NCO always averages the coordinate variable regardless of the arithmetic operation type performed on the non coordinate variables The mathematical definition of each arithmetic operation is given below See Section 4 12 ncwa netCDF Weighted Averager page 235 for additional information on masks and normalization If an operation type is not specified with y then the operator performs an arithmetic average by default Averaging is described first so the terminology for t
394. ontents after the cover sheet before you staple the manual 8 NCO 4 5 3 alpha03 User Guide These can be found directly by prepending the operator name with the xmp tag e g http nco sf net nco html xmp_ncks Also users can type the operator name on the shell command line to see all the available options or type e g man ncks to see a help man page NCO is a command line language You may either use an operator after the prompt e g here like operator options input output or write all commands lines into a shell script as in the CMIP5 Example see Chapter 7 CMIP5 Example page 247 If you are new to NCO the Quick Start see Chapter 6 Quick Start page 245 shows simple examples about how to use NCO on different kinds of data files More detailed real world examples are in the Chapter 7 CMIP5 Example page 247 The General Index page 285 is presents multiple keyword entries for the same subject If these resources do not help enough please see Section 1 7 Help Requests and Bug Reports page 15 1 3 Operating systems compatible with NCO In its time on Earth NCO has been successfully ported and tested on so many 32 and 64 bit platforms that if we did not write them down here we would forget their names IBM AIX 4 x 5 x FreeBSD 4 x GNU Linux 2 x LinuxPPC LinuxAlpha LinuxARM LinuxSparc64 LinuxAMD64 SGI IRIX 5 x and 6 x MacOS X 10 x DEC OSF NEC Super UX 10 x Sun SunOS 4 1 x Sol
395. or example the Saharan desert stretches from roughly 340 to 50 degrees East Extracting the hyperslab of data representing the Sahara from a global dataset presents special problems when the global dataset is stored consecutively in longitude from 0 to 360 degrees This is because the data for the Sahara will not be contiguous in the input file but is expected by the user to be contiguous in the output file In this case ncks must invoke special software routines to assemble the desired output hyperslab from multiple reads of the input file Assume the domain of the monotonically increasing longitude coordinate lon is 0 lon 360 ncks will extract a hyperslab which crosses the Greenwich meridian simply by specifying the westernmost longitude as min and the easternmost longitude as max The following commands extract a hyperslab containing the Saharan desert ncks d lon 340 50 in nc out nc ncks d lon 340 50 d 1at 10 35 in nc out nc The first example selects data in the same longitude range as the Sahara The second example further constrains the data to having the same latitude as the Sahara The coor dinate lon in the output file out nc will no longer be monotonic The values of lon will be e g 340 350 0 10 20 30 40 50 This can have serious implications should you run out nc through another operation which expects the 1on coordinate to be monotoni cally increasing Fortunately the chances of this happening are sl
396. oring The command mv src dst moves recursively the source directory src to the destination directory dst If src begins with the slash character then the specified path is relative to the root directory otherwise the path is relative to the current working directory In other words an initial slash character anchors the subsequent path to the root directory In NCO an initial slash anchors the path at the root group Paths that begin and end with slash characters e g g1 and g1 g2 are both anchored and non recursive Consider the following commands all of which may be assumed to end with in nc out nc ncks g gl Extract recursively all groups with a gi component ncks g gl Extract non recursively all groups terminating in gl ncks g gl Extract recursively root group gl ncks g gi Extract non recursively root group gl ncks g Extract non recursively the root group The first command is probably the most useful and common It would extract these groups if present and all their direct ancestors and children g1 g2 g1 and g3 g1 g2 In other words the simplest form of g grp grabs all groups that and their direct ancestors and children recursively that have grp as a complete component of their path A simple Chapter 3 Shared Features 45 string match is insufficient grp must be a complete component i e group name in the path The option g g1 would not extract these groups because
397. orms binary operations on variables in file 1 and the corresponding variables those with the same name in file 2 and stores the results in file 3 The binary operation operates on the entire files modulo any excluded variables See Section 3 27 Missing Values page 84 for treatment of missing values One of the four standard arithmetic binary operations currently supported must be selected with the y op typ switch or long options op_typ or operation The valid binary operations for ncbo their definitions corresponding values of the op_typ key and alternate invocations are Addition Definition file 3 file_1 file 2 Alternate invocation ncadd op_typ key values add addition Examples ncbo op typ add 1 nc 2 nc 3 nc ncadd 1 nc 2 nc 3 nc Subtraction Definition file 3 file 1 file 2 Alternate invocations ncdiff ncsub ncsubtract op_typ key values sbt dff diff sub subtract subtraction Examples ncbo op_typ 1 nc 2 nc 3 nc ncdiff 1 nc 2 nc 3 nc Multiplication Definition file 3 file 1 file_2 Alternate invocations ncmult ncmultiply op_typ key values mlt mult multiply multiplication Examples ncbo op_typ mlt 1 nc 2 nc 3 nc ncmult 1 nc 2 nc 3 nc Division Definition file 3 file_1 file 2 Alternate invocation ncdivide op_typ key values dvd
398. output file already exists the user is prompted whether to overwrite append replace or exit from the command Selecting overwrite tells the operator to erase the existing output file and replace it with the results of the operation Selecting exit causes the operator to exit the output file will not be touched in this case Selecting append replace causes the operator to attempt to place the results of the operation in the existing output file See Section 4 7 ncks netCDF Kitchen Sink page 203 The simplest way to create the union of two files is ncks A fl 1 nc fl 2 nc This puts the contents of f1 1 nc into fl 2 nc The A is optional On output fl 2 nc is the union of the input files regardless of whether they share dimensions and variables or are completely disjoint The append fails if the input files have differently named record dimensions since netCDF supports only one or have dimensions of the same name but different sizes 2 5 Simple Arithmetic and Interpolation Users comfortable with NCO semantics may find it easier to perform some simple mathe matical operations in NCO rather than higher level languages ncbo see Section 4 3 ncbo netCDF Binary Operator page 189 does file addition subtraction multiplication divi sion and broadcasting It even does group broadcasting ncflint see Section 4 6 ncflint netCDF File Interpolator page 200 does file addition subtraction multiplication and in terpolation Se
399. pabilities have dramatically improved since 1994 We have accordingly replaced all Fortran subroutines with C functions This greatly simplifies the task of building NCO on nominally unsupported platforms As of August 1999 NCO built entirely in C by default This al lowed NCO to compile on any machine with an ANSI C compiler In August 2004 the first C99 feature the restrict type qualifier entered NCO in version 2 9 8 C compilers can obtain better performance with C99 restricted pointers since they inform the compiler when it may make Fortran like assumptions regarding pointer contents alteration Subsequently NCO requires a C99 compiler to build correctly In January 2009 NCO version 3 9 6 was the first to link to the GNU Scientific Library GSL GSL must be version 1 4 or later NCO in particular ncap2 uses the GSL spe cial function library to evaluate geoscience relevant mathematics such as Bessel functions Legendre polynomials and incomplete gamma functions see Section 4 1 19 GSL special functions page 161 In June 2005 NCO version 3 0 1 began to take advantage of C99 mathematical spe cial functions These include the standarized gamma function called tgamma for true gamma NCO automagically takes advantage of some GNU Compiler Collection GCC extensions to ANSI C As of July 2000 and NCO version 1 2 NCO no longer performs arithmetic operations in Fortran We decided to sacrifice executable speed for code maintainab
400. pack The following com mands are equivalent for unpacking ncpdq P upk ncpdq U ncpdq pck plc unpack and ncunpack Equivalent commands for other packing policies e g all_xst follow by analogy Note that ncpdq synonyms are subject to the same constraints and recommen dations discussed in the secion on ncbo synonyms see Section 4 3 ncbo netCDF Binary Operator page 189 That is symbolic links must exist from the synonym to ncpdq or else the user must define an alias The ncpdq packing algorithms must know to which type particular types of input vari ables are to be packed The correspondence between the input variable type and the output packed type is called the packing map The user specifies the desired packing map with the M switch or its long option equivalents pck map and map and its pck map argument Five packing maps are currently implemented Chapter 4 Reference Manual 219 Pack Floating Precisions to NC SHORT default Definition Pack floating precision types to NC SHORT Map Pack NC DOUBLE NC FLOAT to NC SHORT Types copied instead of packed NC_INT64 NC_UINT64 NC_INT NC_UINT NC_ SHORT NC_USHORT NC_CHAR NC_BYTE NC_UBYTE pck_map key values flt sht pck map flt sht Pack Floating Precisions to NC BYTE Definition Pack floating precision types to NC BYTE Map Pack NC DOUBLE NC FLOAT to NC BYTE Types copied instead of packed NC_INT64 NC_UINT64 NC_INT NC_UINT NC_ SHORT
401. page 136 The user must manually define the proper metadata for variables defined using left hand casting 1 ncap2 is the successor to ncap which was put into maintenance mode in November 2006 This documen tation refers to ncap2 which has a superset of the ncap functionality Eventually ncap will be deprecated in favor ncap2 ncap2 may be renamed ncap in 2013 Chapter 4 Reference Manual 131 4 1 1 Syntax of ncap2 statements Mastering ncap2 is relatively simple Each valid statement statement consists of standard forward algebraic expression The fl nco if present is simply a list of such statements whitespace and comments The syntax of statements is most like the computer language C The following characteristics of C are preserved Array syntax Arrays elements are placed within characters Array indexing Arrays are 0 based Array storage Last dimension is most rapidly varying Assignment statements A semi colon indicates the end of an assignment statement Comments Multi line comments are enclosed within characters Single line comments are preceded by characters Nesting Files may be nested in scripts using include script Note that the include command is not followed by a semi colon because it is a pre processor directive not an assignment statement The filename script is interpreted relative to the run directory Attribute syntax The at sign is used to delineate an attribute name from
402. par ent group node1 var In the final output file sn LImon all mdl all xpt nsm avg tm avg nc sub groups with a suffix of avg are the long term averages of each model One thing to notice is that for now ensembles with only one ensemble member will be left untouched 256 7 3 Snow Cover Anomaly Snow Cover Anomaly 96 NCO 4 5 3 alpha03 User Guide Annual Average over Regions Snow Cover Annual Trends a Greenland 1960 1970 1980 1990 2000 Year 1980 1970 1980 1990 2000 Year Figure 7 2 Annual Average over Regions This section illustrates how to calculate the annual average over specific regions see Figure 7 2 Key steps include 1 Spatial average using ncap2 see Section 4 1 ncap2 netCDF Arithmetic Processor page 130 and ncwa see Section 4 12 ncwa netCDF Weighted Averager page 235 Change dimension order using ncpdq see Section 4 8 ncpdq netCDF Permute Dimen sions Quickly page 217 3 Annual average using ncra see Section 4 9 ncra netCDF Record Averager page 225 4 Anomaly from long term average using ncbo see Section 4 3 ncbo netCDF Binary Operator page 189 Standard deviation using ncbo see Section 4 3 ncbo netCDF Binary Operator page 189 and nces see Section 4 4 nces netCDF Ensemble Statistics page 194 Rename variables using ncrename see Section 4 11 ncrename netCDF Renamer page 230 Edit attributions using ncatted see Section 4
403. perations e g maximization square root RMS are applied only to non coordinate variables In these cases the linear average of the coordinate variable will be returned EXAMPLES Consider a model experiment which generated five realizations of one year of data say 1985 Imagine that the experimenter slightly perturbs the initial conditions of the problem before generating each new solution Assume each file contains all twelve months a seasonal cycle of data and we want to produce a single file containing the ensemble average mean seasonal cycle Here the numeric filename suffix denotes the realization number not the month nces 85 O1 nc 85 02 nc 85 03 nc 85 04 nc 85_05 nc 85 nc nces 85 O 1 5 nc 85 nc nces n 5 2 1 85 O1 nc 85 nc 196 NCO 4 5 3 alpha03 User Guide These three commands produce identical answers See Section 3 5 Specifying Input Files page 30 for an explanation of the distinctions between these methods The output file 85 nc is the same size as the inputs files It contains 12 months of data which might or might not be stored in the record dimension depending on the input files but each value in the output file is the average of the five values in the input files In the previous example the user could have obtained the ensemble average values in a particular spatio temporal region by adding a hyperslab argument to the command e g nces d time 0 2 d 1at 23 5 23 5 85_ nc 85 nc In this case the out
404. perators can add user specified global attributes to output files As of NCO ver sion 4 5 2 July 2015 NCO supports multiple uses of the g1b or equivalent gaa or glb att add switch The switch takes mandatory arguments g1b att nm att val where att_nm is the desired name of the global attriute to add and att_val is its value Currently only text attributes are supported recorded as type NC CHAR and regular expres sions are not allowed unlike see Section 4 2 ncatted netCDF Attribute Editor page 183 ncra glb machine HOSTNAME glb created by USER in nc out nc Multiple invocations simplify the annotation of output file at creation or modification time This feature helps to avoid the performance penalty incurred by having to use ncatted separately to annotate large files Should users emit a loud hue and cry we will consider ading the the functionality of ncatted to the front end of all operators i e accepting valid ncatted arguments to modify attributes of any type and to apply regular expressions 3 39 History Attribute Availability All operators Short options h Long options hst history All operators automatically append a history global attribute to any file they create or modify The history attribute consists of a timestamp and the full string of the invocation command to the operator e g Mon May 26 20 10 24 1997 ncks in nc out nc The full contents
405. plifies generating lists of related variables ncks ppc Q 5 ppc FS FL 4 ppc RH 3 in nc out nc Although PPC rounding instantly reduces data precision on disk storage reduction only occurs once the data are compressed How can one be sure the lossy data are sufficiently precise PPC preserves all significant digits of every value The DSD algorithm uses floating point math to round each value optimally so that it has the maximum number of zeroed bits that preserve the specified precision The NSD algorithm uses a theoretical approach 3 2 bits per base 10 digit tuned and tested to ensure the worst case quantization error is less than half the value of the minimum increment in the least significant digit We define several metrics to quantify the quantization error The mean error and mean absolute error incurred in quantizing a variable from its true values x to quantized values q are respectively i N i N 2 cA iM W X qi PS un Lim w L qi IN and E IN dm 22g 4TTV DU LM wi where u is 1 unless x equals the missing value m is 1 unless x is masked and w is the weight The maximum and minimum errors max and min are both signed E Emax Max x qi and Emin min z qi while the maximum and minimum absolute errors et are positive definite mabs Emabs max z qi max max min shows how to compress IEEE floating point data by zeroing insignifican
406. pr staus gsl_sf_bessel_In_scaled_array int int double amp var_out gsl sf bessel KO dbl expr gsl sf bessel K1 dbl expr gsl_sf_bessel_Kn int_expr dbl_expr status gsl_sf_bessel_Kn_array int int double amp var_out gsl sf bessel K0 scaled dbl expr gsl_sf_bessel_K1_scaled dbl_expr esl_sf_bessel_Kn_scaled int_expr dbl_expr status gsl sf bessel Kn scaled array int int double amp var out gsl sf bessel JO dbl expr gsl_sf_bessel_J1 dbl_expr gsl_sf_bessel_j2 dbl_expr gsl_sf_bessel_jl int_expr dbl_expr status gsl sf bessel jl array int double amp var out gsl sf bessel jl steed array gsl sf bessel YO dbl expr gsl_sf_bessel_Y1 dbl_expr gsl_sf_bessel_y2 dbl_expr gsl_sf_bessel_yl int_expr dbl_expr status gsl_sf_bessel_yl_array int double amp var_out gsl_sf_bessel_I0_scaled dbl_expr gsl_sf_bessel_I1_scaled dbl_expr gsl_sf_bessel_i2_scaled dbl_expr esl_sf_bessel_il_scaled int_expr dbl_expr status gsl_sf_bessel_il_scaled_array int double amp var_out esl_sf_bessel_K0_scaled dbl_expr gsl_sf_bessel_K1_scaled dbl_expr gsl_sf_bessel_k2_scaled dbl_expr gsl_sf_bessel_kl_scaled int_expr dbl_expr status gsl_sf_bessel_kl_scaled array int double amp var_out gsl_sf_bessel_Jnu dbl_expr dbl_expr gsl_sf_bessel_ Ynu dbl_expr dbl_expr gsl_sf_bessel_sequence_Jnu gsl_sf_bessel_Inu_scaled dbl_expr dbl_expr gsl_sf_bessel_Inu dbl_expr dbl_expr gsl_sf_bessel_Knu_scaled dbl_expr dbl_expr esl_sf_bessel_Knu dbl_exp
407. pss bin or ncar opt hpss hsi Tell us if the HPSS installation at your site places the hsi command in a different location and we will add that location to the list of acceptable paths to search for hsi NCO supported the old NCAR Mass Storage System MSS until version 4 0 7 in April 2011 NCO supported MSS retrievals via a variety of mechanisms including the msread msrcp and nrnet commands invoked either automatically or with sentinels like ncks p mss ZENDER nco 1 in nc Once the MSS was decommissioned in March 2011 support for these retrieval mechanisms was replaced by support for HPSS in NCO al Chapter 3 Shared Features 35 l in nc Using p is recommended because it clearly separates the input path from the filename itself sometimes called the stub When input path is not explicitly specified using p NCO internally generates an input path from the first input filename The automatically generated input path is constructed by stripping the input filename of everything following the final character i e removing the stub The 1 output path option tells NCO where to store the remotely retrieved file It has no effect on locally retrieved files or on the output file Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file If 1 is not specified then NCO internally generates an output path by simply setting output path
408. pt to change the attribute name in both Global or Group and variable attributes The examples with the character attempt to change only global and group Convention attributes and leave unchanged any Convention attributes attached directly to variables Attributes prefixed with a period Convention need not be present Attributes not prefixed with a period Convention must be present Variables prefixed with a period or global need not be present Variables not prefixed with a period global must be present Chapter 4 Reference Manual 235 4 12 ncwa netCDF Weighted Averager SYNTAX ncwa 3 4 6 7 A a dim B mask cond b C c cnk_dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stridel1 F G gpe dscl g grp glb h hdr pad nbr 1 L dfl 1vl1 1 path M mask val m mask var N no tmp f1 0 o output file p path ppc R r ram all rth db1llflt T mask comp t thr nbr unn v var w weight X x Ey Opt yp input file output file DESCRIPTION ncwa averages variables in a single file over arbitrary dimensions with options to specify weights masks and normalization See Section 2 6 Statistics vs Concatenation page 20 for a description of the distinctions between the various statistics tools and concatenators The default behavior
409. put file would contain only three slices of data in the time dimension These three slices are the average of the first three slices from the input files Additionally only data inside the tropics is included As of NCO version 4 3 9 released December 2013 nces also works with groups rather than files as the fundamental unit of the ensemble Consider two ensembles ecmwf and cesm stored across three input files md11 nc md12 nc and md13 nc Ensemble members would be leaf groups with names like ecmwf 01 ecmwf 02 etc and cesm 01 cesm 02 etc These commands average both ensembles nces nsm grp mdli nc mdl2 nc mdl3 nc out nc nces nsm grp nsm sfx min op typ min n 3 1 1 mdli nc out nc nces nsm grp g cesm v tas d time 0 3 n 3 1 1 mdli nc out nc The first command stores averages in the output groups cesm and ecmwf while the second stores minima in the output groups cesm cesm min and ecmwf ecmwf min The third command demonstrates that sub setting and hyperslabbing work as expected Note that each input file may contain different numbers of members of each ensemble as long as all distinct ensembles contain at least one member in the first file Chapter 4 Reference Manual 197 4 5 ncecat netCDF Ensemble Concatenator SYNTAX ncecat 3 4 6 7 4 C c cnk dmn nm sz cnk map map cnk min sz cnk plc plc cnk scl sz D dbg d dim min max stride F G gpe_dsc g
410. quences of these commands can accomplish simple yet powerful operations from the command line 3 The terminology merging is reserved for an unwritten operator which replaces hyperslabs of a variable in one file with hyperslabs of the same variable from another file i Yes the terminology is confusing By all means mail me if you think of a better nomenclature Should NCO use paste instead of append 20 NCO 4 5 3 alpha03 User Guide 2 6 Statistics vs Concatenation The most frequently used operators of NCO are probably the statisticians i e tools that do statistics and concatenators Because there are so many types of statistics like averaging e g across files within a file over the record dimension over other dimensions with or without weights and masks and of concatenating across files along the record dimension along other dimensions there are currently no fewer than five operators which tackle these two purposes ncra nces ncwa ncrcat and ncecat These operators do share many capabilities though each has its unique specialty Two of these operators ncrcat and ncecat concatenate hyperslabs across files The other two operators ncra and nces compute statistics across and or within files First let s describe the concatenators then the statistics tools 2 6 1 Concatenators ncrcat and ncecat Joining together independent files along a common record dimension is called concatena tion ncrcat is designed for co
411. question to the IEEE NaN value ncatted a FillValue o f NaN in nc Then change the missing value from the IEEE NaN value to a normal IEEE number like 1 0e36 or to whatever the original missing value was ncatted a _FillValue m f 1 0e36 in nc Delete all existing units attributes ncatted a units d in nc The value of var nm was left blank in order to select all variables in the file The values of att type and att val were left blank because they are superfluous in Delete mode Delete all attributes associated with the tpt variable and delete all global attributes ncatted a tpt d a global d in nc The value of att nm was left blank in order to select all attributes associated with the variable To delete all global attributes simply replace tpt with global in the above Modify all existing units attributes to meter second 1 ncatted a units m c meter second 1 in nc Add a units attribute of kilogram kilogram 1 to all variables whose first three char acters are H20 ncatted a units H20 c c kilogram kilogram 1 in nc Overwrite the quanta attribute of variable energy to an array of four integers ncatted a quanta energy o s 010 101 111 121 in nc As of NCO 3 9 6 January 2009 ncatted accepts extended regular expressions as ar guments for variable names and since NCO 4 5 1 July 2015 for attribute names ncatted a isotope H20 c s 18 in nc ncatted a _iso
412. r 0 1 map map nc in nc out nc Renormalized regridding The first example uses the default conservative algorithm The second example specifies that valid values must cover at least 1096 of the destination gridcell to meet the threshold for a non missing destination value For a valid destination area of say 2596 the renormalized algorithm produces a destination value four times greater than the conservative algorithm Chapter 3 Shared Features 75 NCO automatically annotates the output with relevant metadata such as coordinate bounds axes and vertices la CF These annotations include Horizontal Dimension Names lat lon The name of the horizontal spatial dimensions assumed to represent latitude and longitude in 2D rectangular input files are Jat_nm and lon_nm which default to lat and lon respectively Variables that contain a lat_nm dimension and a lon nm dimension on a 2D rectangular input grid will be regridded and vari ables regridded to a 2D rectangular output grid will all contain the lat nm and lon nm dimensions and variables regridded to a 1D unstructured output grid will have lat nm and Jon_nm as auxiliary coordinate variables To treat differ ent dimensions and variables as latitude and longitude use the options rgr lat lat nm and rgr lon lon_nm Note that for now at least lat nm and lon nm indicate both the variable names associated and where applicable i e on 2D grids the dimensions of the ho
413. r dbl_expr gsl_sf_bessel_InKnu dbl_expr dbl_expr gsl_sf_bessel_zero_JO uint_expr gsl_sf_bessel_zero_J1 uint_expr gsl_sf_bessel_zero_Jnu gsl_sf_clausen dbl_expr gsl_sf_hydrogenicR_1 gsl_sf_hydrogenicR gsl_sf_coulomb_wave_FG gsl_sf_coulomb_wave_F_array Chapter 4 Reference Manual gsl sf coulomb wave FG array gsl sf coulomb wave F Gp array gsl_sf_coulomb_wave_sphF_array gsl_sf_coulomb_CL_e gsl_sf_coulomb_CL_array gsl sf coupling 3j e gsl sf coupling 6j e gsl sf coupling RacahW e gsl sf coupling 9j e gsl sf coupling 6j INCORRECT gsl sf dawson e esl_sf_debye_1_e esl_sf_debye_2_e esl_sf_debye_3_e esl_sf_debye_4_e esl_sf_debye_5_e esl_sf_debye_6_e esl_sf_dilog_e gsl_sf_complex_dilog_xy_e esl_sf_complex_dilog_e gsl_sf_complex_spence_xy_e gsl_sf_multiply_e gsl_sf_multiply_err_e esl_sf_ellint_Kcomp e esl_sf_ellint_Ecomp e esl_sf_ellint_Pcomp e esl_sf_ellint_Dcomp e esl_sf_ellint_F_e esl_sf_ellint_E_e gsl sf ellint_P_e gsl_sf_ellint_D_e gsl_sf_ellint_RC_e esl_sf_ellint_RD_e esl_sf_ellint_RF_e esl_sf_ellint_RJ_e gsl_sf_elljac_e gsl_sf_erfc_e esl_sf_log_erfc_e gsl sf erf e gsl sf erf Z e gsl sf erf Q e gsl sf hazard e gsl sf exp e gsl_sf_exp_el0_e gsl_sf_exp_mult_e gsl sf exp mult el0 e gsl sf expml e N DL p E RR E E E d E NE EE E i Er Z ER M E B E ig AZ Z Z ESA Z Z 165 gsl sf coulomb wave FG array gsl sf coulomb wave FGp array gsl sf coulomb wave
414. r interest ex tends beyond NASA datasets Taken together NCO s features autoconversion to netCDF3 atomic types fixing multiple record dimensions autosensing HDF4 input scoping rules for CF conventions make ncdismember reliable and friendly for both dismembering hierarchi cal files and for CF compliance checks Most HDF4 and HDF5 datasets can be checked for CF compliance with a one line command Example compliance checks of common NASA datasets are at http dust ess uci edu diwg Our long term goal is to enrich the hierarchical data model with the expressivity and syntactic power of CF conventions NASA asked the DIWG to prepare a one page summary of the procedure necessary to check HDF files for CF compliance cat gt ncdismember txt lt lt EUOF Preparing an RPM based OS to Test HDF netCDF Files for CF Compliance By Charlie Zender UCI amp NASA Dataset Interoperability Working Group DIWG Installation Summary HDF4 with internal netCDF support disabled HDF5 netCDF with external HDF4 support _enabled_ NCO numpy netcdf4 python python 1xml CFunits python CFChecker 10 ncdismember oono APUNE All 10 packages can use default installs except HDF4 and netCDF Following instructions for Fedora Core 20 FC20 an RPM based Linux OS Feedback and changes for other Linux based 0S s welcome to zender at uci edu H4DIR H5DIR NETCDFDIR NCODIR may all be different For simplicity CZ sets them al
415. r than selection operator selects all values in the left operand that are greater than the corresponding value in the right operand If the value of the left side is less than or equal to the corresponding value of the right side then the right side value is placed in the result Example RDM2 RDM gt gt 100 0 100 100 100 100 126 126 100 100 100 100 double RDM2 RDM lt lt 90s 1 9 36 84 90 90 84 36 9 1 int Chapter 4 Reference Manual 135 4 1 3 Dimensions Dimensions are defined in Output using the defdim function defdim cnt 10 Dimension size is fixed by default defdim cnt 10 NC UNLIMITED Dimension is unlimited record dimension defdim cnt 10 0 4 Dimension is unlimited record dimension defdim cnt 10 1 Dimension size is fixed defdim cnt 10 737 All non zero values indicate dimension size is fixed This dimension name must then be prefixed with a dollar sign when referred to in method arguments or left hand casting e g new var cnt time temperature time lat lon 35 5 temp avg temperature avg time The size method allows the dimension size to be used in an arithmetic expression time avg time total time size Increase the size of a new variable by one and set new member to zero defdim cnt new cnt size 1 new_var cnt_new 0 0 new_var 0 cnt_new size 2 old_var Dimension Abbreviations It is possible to use dimension abbreviations as method arg
416. r versions of NCO have much better support for ncdismember wget http nco sourceforge net src nco 4 4 4 tar gz tar xvzf nco 4 4 4 tar gz cd nco 4 4 4 configure prefix NCODIR make amp amp make install 5 numpy sudo yum install numpy y 6 netcdf4 python sudo yum install netcdf4 python y 7 python 1xml sudo yum install python lxml y 8 CFunits python No RPM available Must install by hand http code google com p cfunits python Chapter 3 Shared Features 57 wget http cfunits python googlecode com files cfunits 0 9 6 tar gz tar xvzf cfunits 0 9 6 tar gz cd cfunits 0 9 6 sudo python setup py install 9 CFChecker No RPM available Must install by hand https bitbucket org mde cfchecker wget https bitbucket org mde cfchecker downloads CFchecker 1 5 15 tar bz2 tar xvjf CFchecker 1 5 15 tar bz2 cd CFchecker sudo python setup py install 10 ncdismember Copy script from http nco sf net nco html ncdismember Store dismembered files somewhere e g DATA nco tmp hdf mkdir p DATA nco tmp hdf Many datasets work with a simpler command ncdismember nco data in nc DATA nco tmp hdf cf 1 5 ncdismember nco data mdl 1 nc DATA nco tmp hdf cf 1 5 ncdismember DATAT hdf AMSR E L2 Rain V10 200905312326 A hdf DATA nco tmp hdf cf 1 5 ncdismember DATA hdf BUV Nimbus04_L3zm_v01 00 2012m0203t144121 h5 DATA nco tmp hdf cf 1 5 ncdismember DATA hdf HIRDLS Au
417. ra_L3ZAD_v06 00 00 c02_2005d022 2008d077 he5 DATA Some datasets typically h5 require the fix rec dmn all argument ncdismember_ DATA hdf GATMO_npp_d20100906_t1935191_e1935505_b00012_c2011070715593206 ncdismember DATA hdf mabel_12_20130927t201800_008_1 h5 DATA nco tmp hdf cf 1 5 fix rec dmn all EOF A PDF version of these instructions is available here http dust ess uci edu diwg ncdismember pdf 3 14 C and Fortran Index conventions Availability ncbo nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options F Long options fortran The F switch changes NCO to read and write with the Fortran index convention By default NCO uses C style 0 based indices for all I O In C indices count from 0 rather than 1 and dimensions are ordered from slowest inner most to fastest outer most vary ing In Fortran indices count from 1 rather than 0 and dimensions are ordered from fastest inner most to slowest outer most varying Hence C and Fortran data storage conventions represent mathematical transposes of eachother Note that record variables contain the record dimension as the most slowly varying dimension See Section 4 8 ncpdq 58 NCO 4 5 3 alpha03 User Guide netCDF Permute Dimensions Quickly page 217 for techniques to re order including trans pose dimensions and to reverse data storage order Consider a file 85 nc containing 12 months of data in the record dim
418. random numbers from the range 0 20 Chapter 4 Reference Manual 175 a2 1 1 6 9 11 13 13 15 16 19 The following example produces an ncap2 runtime error This is because the chose rng algo rithm has a maximum value greater than NC MAX INT 2147483647 the wrapper functions to gsl rng get and gsl rng uniform int return variable of type NC INT Please be aware of this when using random number distribution functions functions from the GSL library which return unsigned int Examples of these are gsl ran geometric and gsl ran pascal export GSL RNG TYPE mt 19937 ncap2 v 0 s ai time 0 a2 gsl rng get ai1 in nc foo nc To find the maximum value of the chosen rng algorithm use the following code snippet ncap2 v 0 s rng max gsl rng max print rng max in nc foo nc Random Number Distributions The GSL library has a rich set of random number disribution functions The library also provides cumulative distribution functions and inverse cumulative distribution functions sometimes referred to a quantile functions To see whats available on your build use the shell command ncap2 f grep e ran e _cdf The following examples all return variables of type NC INT defdim out 15 al 0ut 0 5 a2 gsl ran binomial a1 30 sort a2 10 11 12 12 13 14 14 15 15 16 16 16 16 17 22 a3 gsl ran geometric a2 sort a25 1 1 1 1 1 1 1 1 2 2 2 2 3 4 5j a4 gsl ran pascal a2 50
419. rchy The script selects two variables snc and snd snow cover and snow depth 250 NCO 4 5 3 alpha03 User Guide bin bash dj2 222222 2 22 2 2 2 2 2 2 2 2 2 2 2 2222222222222 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 22 22 2 2 Aggregate models to one group file Method Create files with groups by ncecat gag Append groups level by level using ncks Input files like snc_LImon_CCSM4_historical_riiip1_199001 200512 nc snd_LImon_CESM1 BGC_esmHistorical_r1iip1_199001 200512 nc Output files like sn_LImon_199001 200512 nc Online http nco sourceforge net nco html Combine Files Execute this script bash cmb_fl_grp sh dp 2222 2 2 22 2 2 2 2 2 2 2 2 2 2 2 2 2222222222222 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 22 2 2 Directories drc_in data drc_out data grp Constants rlm LImon Realm LandIce Time frequency monthly tms 200001 200512 Timeseries flt nc File Type Geographical weights Can be skipped when ncap2 works on group data Loop over all snc files for fn in ls drc_in snc_ rlm _ _ tms f1t do ncap2 0 s A gu float cos 1at 3 1416 180 gwOlong name geographical weight fn fn done var snc snd xpt esmHistorical historical mdl CCSM4 CESM1 BGC CESM1 CAMB for i in 0 1 do Loop over variables for j in 0 1 do Loop over experiments for k in 0 2 do
420. rd1 cnk_rd1 map rd1 Mnemonic Record Dimension size 1 Chunksize Equals Scalar Size Specified Definition Chunksize for all dimensions is set with the cnk_scl option cnk map key values xp1 cnk_xp1 map xpl Chapter 3 Shared Features 89 Mnemonic E XPLicitly specified dimensions Chunksize Product Matches Scalar Size Specified Definition The product of the chunksizes for each variable matches approxi mately equals the size specified with the cnk_scl option A dimension of size one is said to be degenerate For a variable of rank R i e with R non degenerate dimensions the chunksize in each non degenerate dimension is the Rth root of cnk_scl cnk map key values pra cnk_prd map prd Mnemonic PRoDuct Chunksize Lefter Product Matches Scalar Size Specified default for netCDF3 input Definition The product of the chunksizes for each variable approximately equals the size specified with the cnk_scl option This is accomplished by using dimension sizes as chunksizes for the rightmost most rapidly varying dimensions and then flexing the chunksize of the left most least rapidly varying dimensions such that the product of all chunksizes matches the specified size All dimensions to the left of and including the first record dimension define the left hand side This map was first proposed by Chris Barker cnk map key values 1fp cnk_lfp map lfp Mne
421. rdinate dimensions if any must agree conform in both files Then the variable values and any coordinate dimension values from input file will overwrite the corresponding variable values and coordinate dimension values if any in output file Since there can only be one record dimension in a file the record dimension must have the same name though not necessarily the same size in both files if a record dimension variable is to be appended If the record dimensions are of differing sizes the record dimension of output file will become the greater of the two record dimension sizes the record variable from input file will overwrite any counterpart in output file and fill values will be written to any gaps left in the rest of the record variables I think In all cases variable attributes in output file are superseded by attributes of the same name from input file and left alone if there is no name conflict Some users may wish to avoid interactive ncks queries about whether to overwrite ex isting data For example batch scripts will fail if ncks does not receive responses to its queries Options 0 and A are available to force overwriting existing files and variables respectively Options specific to ncks The following summarizes features unique to ncks Features common to many operators are described in Chapter 3 Shared features page 27 5 Print data to screen alphabetically by group and alphabetically by variable wit
422. reates or appends to an existing cell_methods attribute a space separated list of couplets of the form dmn op where dmn is a comma separated list of dimensions previously contained in the variable that have been reduced by the arithmetic operation op For example the cell_methods value time mean says that the variable in question was averaged over the time dimension In such cases time will either be a scalar variable or a degenerate dimension or coordinate This simply means that it has been averaged over The value time lon mean lat max says that the variable in question is the maximum zonal mean of the time averaged original variable Which is to say that the variable was first averaged over time and longitude and then the residual latitudinal array was reduced by choosing the maximum value Since the cell methods convention may alter metadata in an undesirable or possibly incorrect fash ion we provide switches to ensure it is always or never used Use long options cl1l_mth or cell_methods to invoke the algorithm true by default and options no_cll_mth or no cell methods to turn it off These options are only available in the operators ncwa and ncra Chapter 3 Shared Features 127 3 42 ARM Conventions Availability ncrcat Short options None ncrcat has been programmed to correctly handle data files which utilize the Atmospheric Radiation Measurement ARM Program convention http www arm gov d
423. ressure in nc out nc The first version of the command creates an out nc which contains time pressure and any coordinate variables associated with pressure The out nc from the second version is guaranteed to contain only two variables time and pressure Create netCDF out nc containing all variables from file in nc Restrict the dimensions of these variables to a hyperslab Print with H the hyperslabs to the screen for good measure The specified hyperslab is the fifth value in dimension time the half open range lat gt 0 in coordinate lat the half open range lon 330 in coordinate lon the closed interval 0 3 band 0 5 in coordinate band and cross section closest to 1000 in coordinate lev Note that limits applied to coordinate values are specified with a decimal point and limits applied to dimension indices do not have a decimal point See Section 3 15 Hyperslabs page 58 ncks H d time 5 d 1at 0 0 d lon 330 0 d band 0 3 0 5 d lev 1000 0 in nc out nc Assume the domain of the monotonically increasing longitude coordinate lon is 0 lon 360 Here 1on is an example of a wrapped coordinate ncks will extract a hyperslab which crosses the Greenwich meridian simply by specifying the westernmost longitude as min and the easternmost longitude as max as follows ncks d lon 260 0 45 0 in nc out nc For more details See Section 3 20 Wrapped Coordinates page 66 Chapter 4 Reference Manual 217 4 8 ncpdq netCDF
424. ributes may appear something like this in output global attributes history Thu Jun 4 14 19 04 2015 ncks A home zender foo3 nc home zender tmp nc Thu Jun 4 14 19 04 2015 ncks A home zender foo2 nc home zender tmp nc n Thu Jun 4 14 19 04 2015 ncatted D a atti global o c global metadata only in foo original history from the ur file serving as the basis for subsequent appends history of appended files Thu Jun 4 14 19 04 2015 Appended file home zender foo3 nc had following history attribute Wn Thu Jun 4 14 19 04 2015 ncatted O a att2 global o c global metadata only in foo history from foo3 from which data was appended to fool after data from foo2 was app Thu Jun 4 14 19 04 2015 Appended file home zender foo2 nc had following histor Thu Jun 4 14 19 04 2015 ncatted O a att2 global o c global metadata only in foo history of some totally different file foo2 from which data was appended to fool be atti global metadata only in fool Note that the history of appended files attribute is only created and will only exist in a file that is or descends from a file that was appended to The optional switch h or hst or history also overrides automatically appending the history of appended files attribute 3 40 File List Attributes Availability nces ncecat ncra ncrcat Short options H Long options fl_lst_in file_list Many method
425. rints global metadata ncks M drc_out caseid _clm01 nc The input files ncra used to create the climatological monthly mean will appear in the global attribute named history Use ncrcat to aggregate the climatological monthly means ncrcat 0 D 1 Chapter 9 CCSM Example 277 drc_out caseid _clm nc drc_out caseid _clm_0112 nc Finally create climatological means for reference The climatological time mean ncra 0 D 1 drc_out caseid _clm_0112 nc drc_out caseid _clm nc The climatological zonal mean ncwa 0 D 1 a lon drc_out caseid _clm nc drc_out caseid _clm_x nc The climatological time and spatial mean ncwa 0 D 1 a lon lat time w gw drc_out caseid _clm nc drc_out caseid _clm_xyt nc This file contains only scalars e g global mean temperature used for summarizing global results of a climate experiment Climatological monthly anomalies Annual Cycle Subtract climatological mean from climatological monthly means Result is annual cycle i e climate mean has been removed ncbo 0 D 1 o drc_out caseid _clm_0112_anm nc drc_out caseid _clm_0112 nc drc_out caseid _clm_xyt nc k k ak ak 3k 3k 3K 3K 3K K K K K aK K aK AAAI AI ARR 3K 3K 3K 3K 3K 3K K K K K K K K K 2k 2k 2k 2k 2k 2k 2k Task 2 Correcting monthly averages k ak ak ak aK 3K 3K 3K 3K K K K K aK aK K AAAI AIAG CCI K K K K K K K 2k 2k 2k 2k 2k 2k 2k The previous step
426. rizontal coordinates Unstructured Dimension Name col The name of the horizontal spatial dimension assumed to delineate an unstruc tured grid is col nm which defaults to ncol number of columns Variables that contain the col_nm dimension on an unstructured input grid will be re gridded and regridded variables written to an unstructured output grid will all contain the col nm dimension To treat a different dimension as unstructured use the option rgr col col nm Note Often there is no coordinate variable for the col nm dimension i e there is no variable named col nm although such a coordinate could contain useful information about the unstructured grid Structured Grid Standard Names and Units Longitude and latitude coordinates both regular and auxiliary i e for unstruc tured grids receive CF standard name values of latitude and longitude CF axes attributes with values X and Y and units attributes with values degrees east and degrees north respectively Unstructured Grid Auxiliary Coordinates Unstructured grid auxiliary coordinates for longitude and latitude receive CF coordinates attributes with values lon and lat respectively Structured Grid Bounds Variables bnd lat bnd lon bnd Structured grids with 1D coordinates use the dimension bnd nm which de faults to nbnd with the spatial bounds variables in lat_bnd_nm and lon bnd_nm which default to lon bnds and lat bnds respectively By default spatial bou
427. roceedings IEEE Lecture Notes in Computer Science vol 4459 edited by C Cerin and K C Li pp 744 750 Springer Verlag Berlin Heidelberg doi 10 1007 978 3 540 72360 8_67 e WZJ074 Wang D L C S Zender and S F Jenks 2007 A System for Scripted Data Analysis at Remote Data Centers Eos Trans AGU 88 52 Fall Meet Suppl Abstract IN11B 0469 e WZJ081 Wang D L C S Zender and S F Jenks 2008 Cluster Workflow Execu tion of Retargeted Data Analysis Scripts Proceedings of the 8th IEEE Int Symposium on Cluster Computing and the Grid IEEE CCGRID 08 pp 449 458 Lyon France May 2008 e WZJ091 Wang D L C S Zender and S F Jenks 2009 Efficient Clustered Server side Data Analysis Workflows using SWAMP Earth Sci Inform 2 3 141 155 doi 10 1007 s12145 009 0021 z e PFT88 Press Flannery Teukolsky and Vetterling 1988 Numerical Recipes in C Cambridge Univ Press New York NY General Index General Index double quote 00 0c cece eee 186 fSinclude s e me XP p mE ex 131 wildcard character 0 0000 46 76 VAMO US Yee sced arid ceed c ene 178 9 end quote st exi E eet ct de 186 O MEDIO E E 189 filename expansion 005 46 multiplication 0 0 0 0020 178 wildcard character 0 0 2 005 46 de das ty Navas A EE E Wises one Serer ats later 189 t addition ta LED YS 178
428. rom the shell is uavg u avg This causes ncap2 to average over all dimensions rather than just the time dimension and unintended consequence We recommend using single quotes to protect ncap2 command line scripts from the shell even when such protection is not strictly necessary Expert users may violate this rule to exploit the ability to use shell variables in ncap2 command line scripts see Chapter 9 CCSM Example page 275 In such cases it may be necessary to use the shell backslash character to protect the ncap2 meta character A dimension of size one is said to be degenerate Whether a degenerate record dimension is desirable or not depends on the application Often a degenerate time dimension is useful e g for concatentating but it may cause problems with arithmetic Such is the case in the above example where the first step employs ncwa rather than ncra for the time averaging Of course the numerical results are the same with both operators The difference is that unless b is specified ncwa writes no time dimension to the output file while ncra defaults to keeping time as a degenerate size 1 dimension Appending u and v to the output file would cause ncks to try to expand the degenerate time axis of uavg and vavg to the size of the non degenerate time dimension in the input file Thus the append ncks A command would be undefined and should fail in this case Equally important is the C argument see Secti
429. roup in flat netCDF3 format WARNING Using the default non CF Udunits database cesm cesm O1 nc INFO INIT running CFchecker version 1 5 15 INFO INIT checking compliance with convention CF 1 5 INFO INIT using standard name table version 25 last modified 2013 07 05T05 40 INFO INIT using area type table version 2 date 10 July 2013 INFO 2 4 no axis information found in dimension variables not checking dimensi WARNING 3 variable tasi contains neither long_name nor standard name attribute 16 CFchecker is developed by Michael Decker and Martin Schultz at Forschungszentrum J lich and dis tributed at https bitbucket org mde cfchecker Chapter 3 Shared Features 55 WARNING 3 variable tas2 contains neither long_name nor standard name attribute INFO 3 1 variable tasi does not contain units attribute INFO 3 1 variable tas2 does not contain units attribute cesm cesm 02 nc By default the CF version checked is determined automatically by cf checker The user can override this default by supplying a supported CF version e g 1 3 as an optional fourth argument to ncdismember Current valid CF options are 1 0 1 1 1 2 1 3 1 4 and 1 5 Our development and testing of ncdismember is funded by our involvement in NASA s Dataset Interoperability Working Group DIWG https wiki earthdata nasa gov display ESDSWG Dataset Interoperability Working Group though ou
430. rpolation As such it also performs file subtraction addition multiplication albeit in a more convoluted way than ncbo Beginning with NCO version 4 3 1 May 2013 ncbo supports group broadcasting Group broadcasting means processing data based on group patterns in the input file s and automatically transferring or transforming groups to the output file Consider the case where file 1 contains multiple groups each with the variable v1 while file 2 contains v1 only in its top level i e root group Then ncbo will replicate the group structure of file 1 in the output file file 3 Each group in file 3 contains the output of the corresponding group in file 1 operating on the data in the single group in file 2 An example is provided below EXAMPLES Say files 85 0112 nc and 86 0112 nc each contain 12 months of data Compute the change in the monthly averages from 1985 to 1986 5 Prior to NCO version 4 3 1 May 2013 ncbo would only broadcast variables in file 2 to conform to file 1 Variables in file 1 were never broadcast to conform to the dimensions in file 2 6 This is because ncra collapses the record dimension to a size of 1 making it a degenerate dimension but does not remove it while unless b is given ncwa removes all averaged dimensions In other words by default ncra changes variable size though not rank while ncwa changes both variable size and rank 192 NCO 4 5 3 alpha03 User Guide ncbo 86 0112 n
431. rslab arguments MUST equal the variable s rank Hyperslabs on the Right Hand Side of an assign A simple 1D example time size 10 od time 20 22 24 26 28 30 32 34 36 38 od 7 34 od 7 34 36 38 138 NCO 4 5 3 alpha03 User Guide od 7 20 22 24 26 28 30 32 34 od 4 20 28 36 od 1 6 2 22 26 30 od 20 22 24 26 28 30 32 34 36 38 A more complex three dimensional example 1at size 2 10n size 4 th time 1at 10n 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 99 99 99 99 99 99 99 99 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 99 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 99 74 75 76 77 78 79 99 th 1 1 3 16 th 2 0 17 18 19 20 th 1 3 3 16 24 99 40 48 56 64 72 99 th 5 0 3 2 1 3 5 7 41 43 45 47 If hyperslab arguments collapse to a single value a cross section has been specified then that dimension is removed from the returned variable If all the values collapse then a scalar variable is returned So for example the following is valid th_nw th 0 th 9 th nw has dimensions lon lat NB the time dimension has become degenerate The following is invalid th nw th 0 0 1 th 9 0 1 because the lon dimension now only has two elements The above can be calculated by using a LHS cast with lon_nw as replacement dim for lon defdim
432. rslabbing capabilities Compare ncks traditional with XML printing zender roulee ncks v one nco data in nc one type NC FLOAT O dimensions 1 attribute chunked no compressed no one size RAM 1 sizeof NC FLOAT 1 4 4 bytes one attribute 0 long name size 3 NC CHAR value one one 1 zender roulee ncks xml v one nco data in nc lt xml version 1 0 encoding UTF 8 gt lt netcdf xmlns http www unidata ucar edu namespaces netcdf ncm1 2 2 locat lt variable name one type float shape gt lt attribute name long_name separator value one gt lt values gt 1 lt values gt lt variable gt lt netcdf gt XML mode prints variable metadata and as of NCO version 4 3 7 October 2013 variable data and as of NCO version 4 4 0 January 2014 hidden at tributes That ncks produces correct NcML translations of CDM files for all sup ported datatypes is verified by comparison to output from Unidata s toolsUI Java program Please let us know how to improve XML NcML features ncks provides additional options to configure NcML output xml no location xml_spr_chr and xml_spr_nmr Every NcML configuration option automatically triggers NcML printing so that specifying xml in addition to a configuration option is redundant and unnecessary The xml_no_location switch prevents output of the NcML location el ement By default the location element is printed
433. rve system memory use ncra rather than nces where possible e g if each input file is one record long The file output from nces will have the same dimensions meaning dimension names as well as sizes as the input hyperslabs see Section 4 4 nces netCDF Ensemble Statistics page 194 for a complete description of nces The file out put from ncra will have the same dimensions as the input hyperslabs except for the record dimension which will have a size of 1 see Section 4 9 ncra netCDF Record Averager page 225 for a complete description of ncra 2 6 3 Interpolator ncflint ncflint can interpolate data between or two files Since no other operators have this ability the description of interpolation is given fully on the ncflint reference page see Section 4 6 ncflint netCDF File Interpolator page 200 Note that this capability also allows ncflint to linearly rescale any data in a netCDF file e g to convert between differing units 2 7 Large Numbers of Files Occasionally one desires to digest i e concatenate or average hundreds or thousands of input files Unfortunately data archives e g NASA EOSDIS may not name netCDF files in a format understood by the n loop switch see Section 3 5 Specifying Input Files page 30 that automagically generates arbitrary numbers of input filenames The n loop switch has the virtue of being concise and of minimizing the command line This helps keeps output file small sin
434. ry Mangalam improved regression testing and benchmarking Daniel Wang de veloped the server side capability SWAMP and Henry Butowsky a long time contributor developed ncap2 This support also led NCO to debut in professional journals and meetings The personal and professional contacts made during this evolution have been immensely rewarding Charlie Zender March 2008 Grenoble France The end of the NSF SEI grant in August 2008 curtailed NCO development Fortunately we could justify supporting Henry Butowsky on other research grants until May 2010 while he developed the key ncap2 features used in our climate research And recently the NASA ACCESS program commenced funding us to support netCDF4 group functionality Thus NCO will grow and evade bit rot for the foreseeable future I continue to receive with gratitude the thanks of NCO users at nearly every scientific meeting I attend People introduce themselves shake my hand and extol NCO often effu sively while I grin in stupid embarassment These exchanges lighten me like anti gravity Sometimes I daydream how many hours NCO has turned from grunt work to productive research for researchers world wide or from research into early happy hours It s a cool feeling Charlie Zender April 2012 Irvine California The NASA ACCESS 2011 program generously supported Cooperative Agreement NNX12AF48A NCO from 2012 2015 This allowed us to produce the first iteration of a Group oriented Da
435. ry and March by interpolating the existing data as follows Combine 85 01 nc and 85 04 nc ina 2 1 ratio to make 85 02 nc 202 NCO 4 5 3 alpha03 User Guide ncflint w 0 667 85 01 nc 85 04 nc 85 02 nc ncflint w 0 667 0 333 85 01 nc 85 04 nc 85 02 nc Multiply 85 nc by 3 and by 2 and add them together to make tst nc ncflint w 3 2 85 nc 85 nc tst nc This is an example of a null operation so tst nc should be identical within machine precision to 85 nc Multiply all the variables except the coordinate variables in the file emissions nc by by 0 8 ncflint fix rec crd w 0 8 0 0 emissions nc emissions nc scaled emissions nc The use of fix_rec_crd ensures e g that the time coordinate if any is not scaled i e multiplied Add 85 nc to 86 nc to obtain 85p86 nc then subtract 86 nc from 85 nc to obtain 85m86 nc ncflint w 1 1 85 nc 86 nc 85p86 nc ncflint w 1 1 85 nc 86 nc 85m86 nc ncdiff 85 nc 86 nc 85m86 nc Thus ncflint can be used to mimic some ncbo operations However this is not a good idea in practice because ncflint does not broadcast see Section 4 3 ncbo netCDF Binary Operator page 189 conforming variables during arithmetic Thus the final two commands would produce identical results except that ncflint would fail if any variables needed to be broadcast Rescale the dimensional units of the surface pressure prs sfc from Pascals to hectopas cals millibars ncflint C v prs sfc w 0 01 0 0 i
436. s The right thing according to my gut feelings Now do you feel more comfortable using it Let D x represent the dimensionality of the variable x Dimensionality describes the order and sizes of dimensions If x has rank N then we may write D x as the N element vector D x D1 D2 Dai Di Dn Dic Dto Di D where D is the size of the n th dimension The dimension re order list specified with a is the R element vector R Ri Ra Ra Po Has Rr Ry Rr 2 Rr 1 Rp 1 This limitation imposed by the netCDF storage layer may be relaxed in the future with netCDF4 Chapter 4 Reference Manual 221 There need be no relation between N and R Let the S element vector S be the intersection i e the ordered set of unique shared dimensions of D and R Then S RND 91 Sa 5 gt S51 85 S541 EN 298 2 68 1 Ss S is empty if R D Re ordering or re shaping a variable means mapping the input state with dimension ality D x to the output state with dimensionality D z In practice mapping occurs in three logically distinct steps First we tranlate the user input to a one to one mapping M between input and output dimensions D D This tentative map is final unless external constraints typically netCDF restrictions impose themselves Second we check and if necessary refine the tentative mapping so that the re shaped variables will co exist in the same file without violating netCDF imposed stor
437. s D 2 M foo 4 nc Summary of foo 4 nc filetype NC FORMAT NETCDFA4 representation of extended underlying filetype NC FORMAT HDF5 O groups The extended filetype determines some of the capabilities that netCDF has to alter the file Second query the file with ncdump k ncdump k foo 3c nc classic ncdump k foo 364 nc 64 bit offset ncdump k foo 4c nc netCDF 4 classic model ncdump k foo 4 nc netCDF 4 This method requires a netCDF4 enabled netCDF 3 6 2 i e from 2007 or later The third option uses the POSIX standard od octal dump command od An c N4 foo 3c nc C D F 001 4 od An c N4 foo 364 nc C D F 002 od An c N4 foo 4c nc 211 H D F 4 od An c N4 foo 4 nc 211 H D F This option works without NCO and ncdump Values of C D F 001 and C D F 002 indicate 32 bit classic and 64 bit netCDF3 formats respectively while values of 211 H D F indicate either of the newer netCDF4 file formats Chapter 3 Shared Features 41 3 9 3 File Conversion Let us demonstrate converting a file from any netCDF supported input format into any netCDF output format subject to limits of the output format Here the input file in nc may be in any of these formats netCDF3 classic and 64bit netCDF4 classic and ex tended HDF4 HDF5 HDF EOS version 2 or 5 and DAP The switch determines the output format written in the comment ncks fl fmt classic in nc foo 3c nc netCDF
438. s Quickly page 217 for other methods of changing variable dimensionality in cluding the record dimension Turn on printing to screen or turn off copying data not metadata Also acti vated using print or prn By default ncks prints all metadata and data to screen if no netCDF output file is specified Use H to print data to screen if a netCDF output is specified or to restrict printing to data no metadata when no netCDF output is specified Also use H to turn off copying data not metadata to an output file Unless otherwise specified with s each element of the data hyperslab prints on a separate line containing the names indices and values if any of all of the variables dimensions The dimension and variable indices refer to the location of the corresponding data element with respect to the variable as stored on disk i e not the hyperslab 4 ncks C v three dmn var in nc lat 0 2 90 lev 0 100 lon 0 0 three dmn var 0 0 lat 0 2 90 lev 0 100 lon 1 90 three dmn var 1 1 lat 0 2 90 lev 0 100 lon 2 180 three dmn var 2 2 lat 1 90 lev 2 1000 lon 1 90 three dmn var 21 21 lat 1 90 lev 2 1000 lon 2 180 three dmn var 22 22 lat 1 90 lev 2 1000 lon 3 270 three dmn var 23 23 Printing the same variable with the F option shows the same variable indexed with Fortran conventions ncks F C v three dmn var in nc lon 1 0 lev 1 100 lat 1 90 three dmn var 1 20 lon 2
439. s a convenience abbreviation for C H M m u Also activated using print or prn This set of switches is useful for exploring file contents g Print quietly meaning omit dimension names indices and coordinate values when printing arrays Variable not dimension indices are printed Variable names appear flush left in the output zender roulee ncks Q v three dmn rec var C H nco data in nc three dmn rec var 0 1 This helps locate specific variables in lists with many variables and different dimensions See also the V option which omits all names and indices and prints only variable values q Quench turn off all printing to screen This overrides the setting of all print related switches equivalent to H M m when in single file printing mode When invoked with R see Section 3 8 Retaining Retrieved Files page 37 ncks automatically sets q This allows ncks to retrieve remote files without auto matically trying to print them Also quench rad Retain all dimensions When invoked with rad Retain All Dimensions ncks copies each dimension in the input file to the output file regardless of whether the dimension is utilized by any variables Normally ncks discards orphan dimensions i e dimensions not referenced by any variables This switch allows users to keep non referenced dimensions in the workflow When invoked in printing mode causes orphaned dimensions t
440. s functions However functions are not and cannot be used as methods Methods can be daisy chained d and their syntax is cleaner than functions Method names are reserved words and CANNOT be used as variable names The command ncap2 f shows the complete list of methods available on your build n2 sin theta 146 NCO 4 5 3 alpha03 User Guide n2 theta sin n2 sin theta 2 cos theta 2 n2 theta sin pow 2 theta cos 2 This statement chains together methods to convert three dmn var sht to type double average it then convert this back to type short three_avg three_dmn_var_sht double avg short Aggregate Methods These methods mirror the averaging types available in ncwa The arguments to the methods are the dimensions to average over Specifying no dimensions is equivalent to specifying all dimensions i e averaging over all dimensions A masking variable and a weighting variable can be manually created and applied as needed avg Mean value sqravgO Square of the mean avgsqr Mean of sum of squares max Maximum value min Minimum value mabs Maximum absolute value mebs Mean absolute value mibs Minimum absolute value rms Root mean square normalize by N rmssdn Root mean square normalize by N 1 tt1O0 or total Sum of values Average a variable over time four time avg four dmn rec var time Packing Methods For more information see see Section 3 34 Packed data page
441. s match Here the script illustrates how to regrid MODIS data Key steps include 1 2 Regrid using bilinear interpolation see Section 4 1 18 Bilinear interpolation page 159 Rename variables dimensions and attributions using ncrename see Section 4 11 ncre name netCDF Renamer page 230 Main Script bin bash Chapter 7 CMIP5 Example 267 include bi_interp nco foe SS SS a SS See LLL ee a ee a a eu eee Example for regrid using bi_interp nco the spatial resolution of MODIS data is much finer than those of CMIP5 models In order to compare the two we can regrid MODIS data to comform to CMIP5 Input files Note the hdf files downloaded have to be converted to nc at the present modis mcd43c3 MCD43C3 A2000049 005 2006271205532 nc Output files modis mcd43c3 cesm grid MCD43C3 2000049 regrid nc Online http nco sourceforge net nco html Regrid MODIS Data Execute this script bash rgr sh var MCD43C3 Variable fld in monthly Folder of input files fld out cesm grid Folder of output files drc_in media grele_data wenshan modis mcd43c3 Directory of input files for fn in 1s drc_in fld_in var nc do Loop over files sfx echo fn cut d f 8 cut d f 2 Part of file names Regrid ncap2 0 S bi interp nco fn drc_in fld_out var sfx regrid nc Keep only the new v
442. s of specifying large numbers of input file names pass these names via pipes encodings or argument transfer programs see Section 2 7 Large Numbers of Files page 21 When these methods are used the input file list is not explicitly passed on the command line This results in a loss of information since the history attribute no longer contains the exact command by which the file was created NCO solves this dilemma by archiving input file list attributes When the input file list to a multi file operator is specified via stdin the operator by default attaches two global 35 Note that before version 4 5 0 NCO could in append A mode only inadvertently overwrite the global metadata including history of the output file with that of the input file This is opposite the behavior most would want 124 NCO 4 5 3 alpha03 User Guide attributes to any file they create or modify The nco input file number global attribute contains the number of input files and nco input file list contains the file names specified as standard input to the multi file operator This information helps to verify that all input files the user thinks were piped through stdin actually arrived Without the nco_ input file list attribute the information is lost forever and the chain of evidence would be broken The H switch overrides turns off the default behavior of writing the input file list global attributes when input is from stdin The
443. s regular expressions see Section 3 11 Subsetting Files page 43 for at 184 NCO 4 5 3 alpha03 User Guide tribute names it has always accepted regular expressions for variable names Regular expressions will select all matching attribute names var nm Variable name Example pressure H20 Regular expressions see Section 3 11 Subsetting Files page 43 are accepted and will select all match ing variable and or group names The names global and group have special meaning mode Edit mode abbreviation Example a See below for complete listing of valid values of mode att type Attribute type abbreviation Example c See below for complete listing of valid values of att type att val Attribute value Example pascal There should be no empty space between these five consecutive arguments The description of these arguments follows in their order of appearance The value of att_nm is the name of the attribute to edit This meaning of this should be clear to all ncatted users Both att nm and var nm may be specified as regular expres sions If att nm is omitted i e left blank and Delete mode is selected then all attributes associated with the specified variable will be deleted The value of var um is the name of the variable containing the attribute named att nm that you want to edit There are three very important and useful exceptions to this rule The value of var nm can also be used to direct ncatted to
444. s_crd 2 Variable will be type code NC_INT var gds crd 3 Variable will be type code NC_INT var gds crd 2s Variable will be type OcodeiNC SHORT var gds crd 3s Variable will be type code NC_SHORT var att 41 Attribute will be type code NC_DOUBLE var att 41 f Attribute will be type code NC_FLOAT var att 41 Attribute will be type code NC_INT var att 21s Attribute will be type code NC_SHORT var units kelvin Attribute will be type code NC_CHAR There is no postfix for characters use a quoted string instead for NC_CHAR ncap2 interprets a standard double quoted string as a value of type NC_CHAR In this case any receiving variable must be dimensioned as an array of NC_CHAR long enough to hold the value To use the newer netCDF4 types NCO must be compiled linked to the netCDF4 library and the output file must be of type NETCDF4 var time 1UL Variable will be type code NC_UINT var lon 4b Variable will be type code NC_BYTE var lat 5u11 Variable will be type code NC_UINT64 var lat 511 Variable will be type code NC_INT64 var att 6 0d Attribute will be type code NC_DOUBLE varQatt 666L Attribute will be type code NC_INT var att kelvin s Attribute will be type code NC_STRING note the s Use a post quote s for NC STRING Place the letter s immediately following the double quoted string to indicate that the value
445. scribed here http cfconventions org 1 6 html coordinate system This con vention allows variables to specify additional coordinates including mult idimensional coor dinates in a space separated string attribute named coordinates NCO attaches any such coordinates to the extraction list along with the variable and its usual one dimensional coordinates if any These auxiliary coordinates are subject to the user specified override switches including c and C described in Section 3 12 Subsetting Coordinate Vari ables page 47 Elimination of reduced dimensions from the coordinates attribute helps ensure that rank reduced variables become completely independent from their former dimensions As of NCO version 4 4 9 May 2015 NCO may modify the coordinates attribute to assist this In particular ncwa eliminates from the coordinates attribute any dimension that it collapses e g by averaging The former presence of this dimension will usually be indicated by the CF cell methods convention described here http cfconventions org 1 6 html cell methods Hence the CF cell_methods and coordinates conventions can be said to work in tandem to characterize the state and history of a variable s analysis As of NCO version 4 4 2 February 2014 NCO supports some of the CF cell_methods convention http cfconventions org 1 6 html cell methods to describe the anal ysis procedures that have been applied to data The convention c
446. sd 3 and dsd 2 The only justifiable NSD for a given value depends on intrinsic accuracy and error char acteristics of the model or measurements and not on the units with which the value is stored The appropriate DSD for a given value depends on these intrinsic characteristics and in addition the units of storage This is the fundamental difference between the NSD and DSD approaches The eight hundred pound gorilla always has nsd 1 regardless of whether the value is store in pounds or in some other unit DSD corresponding to this weight is dsd 2 if the value is store in pounds dsd 4 if stored in megapounds Users may wish to express the precision to be preserved as either NSD or DSD Invoke PPC with the long option ppc var prc or give the same arguments to the synonyms precision preserving compression or to quantize Here var is the variable to quantize and prc is the precision The default algorithm assumes prc specifies NSD precision e g T 2 means nsd 2 Prepend prc with a decimal point to specify DSD precision e g T 2 means dsd 2 NSD precision must be specified as a positive integer DSD precision may be a positive or negative integer and is specified as the negative base 10 logarithm of the desired precision in accord with common usage For example specifying T 3 or T 2 tells the DSD algorithm to store only enough bits to preserve the value of T rounded to the near
447. set of comparable size Models often represent Earth on a spherical grid where global averages must be created by weighting each gridcell by its latitude dependent weight e g a Gaussian weight stored in gw or by the surface area of each contributing gridpoint stored in area Like many geophysical models and most GCMs CAM3 runs completely in double precision yet stores its archival output in single precision to save space In prac tice such models usually save multi dimensional prognostic and diagnostic fields like TREFHT lat lon as single precision while saving all one dimensional coordinates and weights here lat lon and gw 10n as double precision The gridcell area area lat lon is an extensive grid property that should be but often is not stored as double precision To obtain pure double precision arithmetic and storage of the globla mean temperature we first create and store double precision versions of the single precision fields ncap2 0 s TREFHT_dbl double TREFHT area_dbl double area in nc in nc Chapter 3 Shared Features 119 The single and double precision temperatures may each be averaged globally using four permutations for the precision of the weight and of the intermediate arithmetic representa tion 1 Single precision weight area single precision arithmetic 2 Double precision weight gw single precision arithmetic 3 Single precision weight area double precision arithmetic 4 Double pr
448. sfc avg time 2 total time time size 1 in nc out nc The operation may of course be broken into multiple steps in order to archive interme diate quantities such as the time anomalies ncap2 s prs sfc anm prs sfc prs sfc avg time s prs sfc sdn sqrt prs sfc anm 2 total time time size 1 in nc out nc ncap2 supports intrinsic standard deviation functions see Section 3 35 Operation Types page 107 which simplify the above expression to ncap2 s prs sfc sdn prs sfc prs sfc avg time rmssdn time in nc out nc These instrinsic functions compute the answer quickly and concisely The procedure to compute the spatial standard deviation of a field in a single file in nc involves three steps ncwa 0 v prs sfc gw a lat lon w gw in nc out nc ncbo 0 v prs sfc gw in nc out nc out nc ncwa 0 y rmssdn v prs sfc a lat lon w gw out nc out nc First the appropriately weighted with w gw spatial mean values are written to the output file This example includes the use of a weighted variable specified with w gw When using weights to compute standard deviations one must remember to include the weights in the initial output files so that they may be used again in the final step The initial output file is then overwritten with the gridpoint deviations from the spatial mean Finally the root mean square of the appropriately weighted spatial deviations is taken The ncap2 solution to the spatiall
449. sion matches any string matching either subexpression Chapter 3 Shared Features 4T To illustrate the use of these operators in extracting variables and groups consider file in grp nc with groups g0 g9 and subgroups s0 s9 in each of those groups and file in nc with variables Q Q01 099 Q100 QAA QZZ Q_H20 X H20 Q C02 X C02 All variables default Variables that contain Q Variables that start with Q Q Q0 Q9 Q01 Q99 QAA QZZ etc Q01 Q99 QAA QZZ etc Q01 Q99 Q100 Q01 Q99 ncks v in nc necks v Q in nc necks v Q in nc ncks v Q 7 in nc ncks v Q in nc ncks v Q 0 9 0 9 in nc ncks v Q digit 1142 in nc ncks v H20 in nc Q H20 X H20 ncks v H20 CO2 in nc Q_H20 X H20 Q_C02 X CO2 ncks v Q 0 9 0 9 in nc Q01 Q99 ncks v Q 0 6 0 9 I7 0 3 in nc Q01 Q73 Q100 ncks v QLO 6 0 9 7 0 3 1 in nc Q01 Q73 ncks v a z a z 3 in nc Q H20 X H20 Q_C02 X C02 ncks g g in grp nc 10 Groups g0 g9 ncks g s in grp nc 100 sub groups g0 s0 g0 sl g9 s9 ncks g g v v in grp nc All variables v in groups g dk HH cb Gb Gb Gb OH OF Beware two of the most frequently used repetition pattern matching operators and are also valid pattern matching operators for filename expansion globbing at the shell level Confusingly their meanings in extended reg
450. sional variable to define a hyperslab is quite different cat gt ncap2 foo nco lt lt EOF Purpose Save irregular 1 D regions based on variable values Included in NCO User Guide at http nco sf net nco html sort NB Single quotes around EOF above turn off shell parameter expansion in here documents This in turn prevents the need for protecting dollarsign characters in NCO scripts with backslashes when the script is cut and pasted aka moused from an editor or e mail into a shell console window Copy coordinates and variable s of interest into RAM variable s Benefits 1 ncap2 defines writes all variables on LHS of expression to disk Only exception is RAM variables which are stored in RAM only Repeated operations on regular variables takes more time because changes are written to disk copy after every change RAM variables are only changed in RAM so script works faster RAM variables can be written to disk at end with ram write 2 Script permutes variables of interest during processing Safer to work with copies that have different names This discourages accidental mistaken use of permuted versions 3 Makes this script a more generic template var in instead of specific variable names everywhere var in one dmn rec var crd in time dmn in sz time size nbr Size of input arrays Create all other intermediate variables as RAM variables to prevent them from cluttering the output file
451. sn t within the input X range GSL Interpolation Types All the interpolation functions have been implemented These are gsl_interp_linear gsl_interp_polynomial gsl_interp_cspline gsl_interp_cspline_periodic esl_interp_akima gsl interp akima periodic Evaluation of Interpolating Types Implemented gsl_spline_eval Unimplemented esl_spline_deriv gsl_spline_deriv2 esl_spline_integ 4 1 21 GSL least squares fitting Least Squares fitting is a method of calculating a straight line through a set of experimental data points in the XY plane The data maybe weighted or unweighted For more information please refer to the GSL manual These GSL functions fall into three categories A Fitting data to Y c0 c1 X B Fitting data through the origin Y c1 X C Multi parameter fitting not yet implemented Section A status gsl fit linear data x stride x data y stride y n amp co amp ci amp covOO amp covO1 amp covii amp sumsq Input variables data x stride x data y stride y n From the above variables an X and Y vector both of length n are derived If data x or data y is less than type double then it is converted to type double It is up to you to do Chapter 4 Reference Manual 171 bounds checking on the input data For example if stride x 3 and n 8 then the size of data x must be at least 24 Output variables cO cl cov00 cov01 cov11 sumsq The amp prefix indicates that these are
452. specify the interpolation option with i or with the ntp or interpolate long options This is the inverse of the first method in the following sense When the user specifies the weights directly ncflint has no work to do besides multiplying the input values by their respective weights and adding together the results to produce the output values It makes sense to use this when the weights are known a priori Another class of problems has the arrival value i e val3 of a particular variable var known a priori In this case the implied weights can always be inferred by examining the values of var in the input files This results in one equation in two unknowns wgtl and wgt2 val3 wgtl x vall wgt2 x val2 Unique determination of the weights requires imposing the additional constraint of normalization on the weights wgtl wgt2 1 Thus to use the interpolation option the user specifies var and val3 with the i option ncflint then computes wetl and wgt2 and uses these weights on all variables to generate the output file Although var may have any number of dimensions in the input files it must represent a single scalar value Thus any dimensions associated with var must be degenerate i e of size one If neither i nor w is specified on the command line ncflint defaults to weighting each input file equally in the output file This is equivalent to specifying w 0 5 or w Chapter 4 R
453. st gridcell Many global models use the grn_ctr longitude grid as their scalar grid where e g temperature humidity and other scalars are defined The staggered or offset grid where often the dynamics variables are defined then must have the grn wst longitude convention That way the centers of the scalar grid are the vertices of the offset grid and visa versa Grid Resolution lat_nbr lon nbr The number of gridcells in the horizontal spatial dimensions are lat nbr and lon nbr respectively There are no restrictions on lon nbr for any gridtype Latitude grids do place some restrictions on lat_nbr see above Grid Edges lon wst lon_est lat sth lat nrt The outer edges of a regional rectangular grid are specified by the North lat_nrt South lat_sth East lat_est and West lat_nrt sides Latitudes and longigudes must be specified in degrees not radians Latitude edges must be between 90 and 90 Longitude edges may be positive or negative and sep arated by no more than 360 degrees The edges may be specified individually with four arguments or together in one list to the options wesn or snwe These two specifications are equivalent ncks ncks rgr lat sth 30 0 rgr lat nrt 70 0 rgr lon wst 120 0 rgr lo rgr snwe 30 0 70 0 120 0 90 0 Generating common grids 180x360 1x1 degree Equi Angular grid first longitude centered at Greenwich ncks rgr grd_ttl
454. t d _ f 3 sort uniq c awk print 2 for mdl in mdl set do Loop over models Average all the ensemble members of each model ncea 0 4 d time 1956 01 01 00 00 0 0 2005 12 31 23 59 9 9 N drc_in var var_id _ rlm _ md1 _ xpt 0 _ nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm nc Greenland Geographical weight ncap2 0 s gw cos 1at 3 1415926 180 gw long_name geographical weight gw units ratio drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm nc ncwa 0 w gw d lat 60 0 75 0 d lon 300 0 340 0 a lat lon drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN nc HcAnOmal yes se ess ST eee ee for moy in 1 12 do Loop over months mm printf 02d moy Change to 2 digit format for yr in 1956 2005 do Loop over years If January calculate the annual average if moy eq 1 then ncra 0 d time yr 01 01 00 00 0 0 yr 12 31 23 59 9 9 drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN nc drc_out fld_out var_id var var_id _ rlm _ md1 _ xpt 0 _all nsm_GN_ yr nc fi The specific month ncks 0 d time yr mm 01 00 00 0 0 yr
455. t p nco data in nc out nc ls 1 bnr dat cut d f 5 200 B contains time and one dmn rec va 200 ls 1 bnr dat ncks C 0 v one dmn rec var b bnr dat p nco data in nc out nc ls 1 bnr dat cut d f 40 B contains one dmn rec var only 40 fix rec dmn Change record dimension dim in the input file into a fixed dimension in the output file Also no_rec_dmn Before NCO version 4 2 5 January 2013 the syntax for fix rec dmn did not permit or require the specification of the dimension name dim This is because the feature only worked on netCDF3 files which support only one record dimension so specifying its name was not necessary netCDF4 files allow an arbitrary number of record dimensions so the user must specify which record dimension to fix The decision was made that starting with NCO version 4 2 5 January 2013 it is always required to specify the dimension name to fix regardless of the netCDF file type This keeps the code simple and is symmetric with the syntax for mk rec dmn described next As of NCO version 4 4 0 January 2014 the argument all may be given to fix rec dmn to convert all record dimensions to fixed dimensions in the output file Previously fix rec dmn only allowed one option the name of a single record dimension to be fixed Now it is simple to simultaneously fix all record dimensions This is useful and nearly mandatory when flattening netCDF
456. t 0 11490 0 0012 0 00498 0 011165 This computes the Bessel function of order n 2 for every value in 1on in The Bessel order argument an integer can also be a non scalar variable i e an array n_in lon 0 1 2 3 lon_out gsl_sf_bessel_Jn n_in 0 5 lon_out 0 93846 0 24226 0 03060 0 00256 Arguments to GSL wrapper functions in ncap2 must conform to one another i e they must share the same sub set of dimensions For example three_out gsl_sf_bessel_ Jn n_in three_dmn_var_db1 is valid because the variable three dmn var db1 has a lon dimension so n in in can be broadcast to conform to three dmn var dbl However time out gsl sf bessel Jn n in time is invalid Consider the elliptical integral with prototype int gsl sf ellint RD e double x double y double z gsl mode t mode gsl sf result result three out gsl sf ellint RD 0 5 time three dmn var dbl The three arguments are all conformable so the above ncap2 call is valid The mode argument in the function prototype controls the convergence of the algorithm It also appears in the Airy Function prototypes It can be set by defining the environment variable GSL PREC MODE If unset it defaults to the value GSL PREC DOUBLE See the GSL manual for more details export GSL_PREC_MODE 0 GSL PREC DOUBLE export GSL_PREC_MODE 1 GSL PREC SINGLE export GSL_PREC_MODE 2 GSL PREC APPROX The ncap2 wrappers to the array functions are slightly different Consider the
457. t and equivalent results nc3tonc4 records the precision of its DSD algorithm in the attribute least significant digit and NCO does the same for consistency The Unidata blog here http www unidata ucar edu blogs developer en entry compression by bit shaving also 96 NCO 4 5 3 alpha03 User Guide It is safe to attempt PPC on input that has already been rounded Variables can be made rounder not sharper i e variables cannot be un rounded Thus PPC attempted on an input variable with an existing PPC attribute proceeds only if the new rounding level exceeds the old otherwise no new rounding occurs i e a no op and the original PPC attribute is retained rather than replaced with the newer value of prc To request say five significant digits nsd 5 for all fields except say wind speeds which are only known to integer values dsd 0 in the supplied units requires ppc twice ncks 4 ppc default 5 ppc u v 0 in nc out nc To preserve five digits in all variables except coordinate variables and u and v use the default option and separately specify the exceptions ncks ppc default 5 ppc u v 20 in nc out nc The ppc option may be specified any number of times to support varying precision types and levels and each option may aggregate all the variables with the same precision ncks ppc p w z 5 ppc q RH 4 ppc T u v 3 in nc out nc Any var argument may be a regular expression This sim
458. t are in order the names of the attribute variable and file to examine Global attributes are indicated by using a variable name of global This definition yields the following results ncattget purpose z in nc Height stored with a monotonically increasing coordinate ncattget Purpose Z in nc Height stored with a monotonically increasing coordinate ncattget history z in nc ncattget history global in nc History global attribute Note that case sensitivity has been turned off for the variable and attribute names and could be turned on by removing the i switch to grep Furthermore extended regular expressions may be used for both the variable and attribute names The next two commands illustrate this by searching for the values of attribute purpose in all variables and then for all attributes of the variable z ncattget purpose in nc 1 D latitude coordinate referred to by geodesic grid variables 1 D longitude coordinate referred to by geodesic grid variables ncattget Z in nc Height Height stored with a monotonically increasing coordinate meter Extended filters are best stored as shell commands if they are used frequently Shell commands may be re used when they are defined in shell configuration files These files are usually named bashrc cshrc and profile for the Bash Csh and Sh shells respec tively NB Untested on Csh Ksh Sh Zsh Send us feedback Bash shell bin bash users place th
459. t attribute name if any assumed to specify the value of data to ignore is _FillValue with version 3 9 2 August 2007 Prior to that the missing value attribute if any was assumed to specify the value of data to ignore Supporting both of these attributes simultaneously is not practical Hence the behavior NCO once applied to missing_value it now applies to any _FillValue NCO now treats any missing value as normal data It has been and remains most advisable to create both _FillValue and missing_ value attributes with identical values in datasets Many legacy datasets contain only missing_value attributes NCO can help migrating datasets between these conven tions One may use ncrename see Section 4 11 ncrename netCDF Renamer page 230 to rename all missing_value attributes to _FillValue ncrename a missing_value _FillValue inout nc Alternatively one may use ncatted see Section 4 2 ncatted netCDF Attribute Editor page 183 to add a _FillValue attribute to all variables ncatted 0 a _FillValue o f 1 0e36 inout nc 19 The old functionality i e where the ignored values are indicated by missing value not _FillValue may still be selected at NCO build time by compiling NCO with the token definition CPPFLAGS UNCO_USE_FILL_ VALUE Chapter 3 Shared Features 85 2 Converting the _FillValue to the type of the variable if neccessary Consider a variable var of type var_type with a FillValue attribute of type att
460. t bits The author John Caron writes that the technique has been called bit shaving When combined with our bias elimination procedure that rounds up to one a more suitable terminology might be bit grooming Chapter 3 Shared Features 97 Subs min x qi m min max min Typically eii 0 for quantization since many exact values need no quantization Bit shifting zeros into the least significant bits LSBs always underestimates true values so that max 0 Conversely bit shifting ones into the LSBs always overestimates true values so that Emin 0 Our NSD algorithm is balanced because it alternates bit shifting zeroes and ones Balanced algorithms should yield max Y miny Gras Y Shing and z 0 The three most important error metrics for quantization are and e The upper bound worst case quantization performance is f ps measures the absolute mean accuracy of quantization and does not allow positive and negative offsets to compensate eachother and conceal poor performance The difference bewtween e and indicates how much of an outlier the worst case is The mean accuracy indicates whether statistical properties of quantized numbers will accurately reflect the true values All three metrics are expressed in terms of the fraction of the ten s place occupied by the LSD If the LSD is the hundreds digit or the thousandths digit then the metrics are frac tions of
461. t is similar to its C counterpart The Conditional Operator ternary operator has also been implemented if exp1 stmt1 else if exp2 stmt2 else stmt3 Can use code blocks as well if exp1 stmt1 stmt a stmtib Jelse if exp2 stmt2 else stmt3 stmt3a stmt3b For a variable or attribute expression to be logically true all its non missing value elements must be logically true i e non zero The expression can be of any type Unlike C there is no short circuiting of an expression with the OR and AND amp amp operators The whole expression is evaluated regardless if one of the AND OR operands are True False Chapter 4 Reference Manual 143 Simple example if time gt 0 print All values of time are greater than zero n else if time lt 0 print All values of time are less than zero n else time max time max time min time min print min value of time print time_min f print max value of time print time max 4f Example from ddra nco if fl typ f1l typ gcm 1 var nbr apx 32 lmn nbr 1 0 var nbr apx varsz gcm 4D nbr Variable size if nco op typ nco op typ avg 1 lmn nbr avg 1 0 var nbr apx varsz gcm 4D Block size lmn nbr wgt dmnsz gcm lat nbr Weight size nco op typ avg else if fl typ fl typ st1 var nbr apx 8 lmn nbr 1 0 var nbr apx varsz stl 2D nbr Variable size if nco op typ 7nco op typ avg 1 lmn nbr
462. t least on netCDF3 files if judicious use of header padding see Section 3 2 Metadata Optimization page 27 was made when producing the input file Similarly using the hdr_pad option with ncatted helps ensure that future metadata changes to output file occur as swiftly as possible When ncatted is used to change the _FillValue attribute it changes the associated missing data self consistently If the internal floating point representation of a missing value e g 1 0e36 differs between two machines then netCDF files produced on those machines will have incompatible missing values This allows ncatted to change the missing values in files from different machines to a single value so that the files may then be concatenated e g by ncrcat without losing information See Section 3 27 Missing Values page 84 for more information To master ncatted one must understand the meaning of the structure that describes the attribute modification att_dsc specified by the required option a or attribute This option is repeatable and may be used multiple time in a single ncatted invocation to increase the efficiency of altering multiple attributes Each att_dsc contains five elements This makes using ncatted somewhat complicated though powerful The att dsc fields are in the following order att_dsc att_nm var nm mode att type att val att nm Attribute name Example units As of NCO 4 5 1 July 2015 ncatted ac cept
463. t the operator is actually doing it is useful to activate a more verbose output mode This is accomplished by supplying a number greater than 0 to the D debug level or debug level or dbg_1v1 switch When the debug level is nonzero the operators report their current status to the terminal through the stderr facility Using D does not slow the operators down Choose a debug level between 1 and 3 for most situations e g nces D 2 85 nc 86 nc 8586 nc A full description of how to estimate the actual amount of memory the multi file NCO operators consume is given in Section 2 9 Memory Requirements page 24 2 9 Memory Requirements Many people use NCO on gargantuan files which dwarf the memory available free RAM plus swap space even on today s powerful machines These users want NCO to consume the least memory possible so that their scripts do not have to tediously cut files into smaller pieces that fit into memory We commend these greedy users for pushing NCO to its limits This section describes the memory NCO requires during operation The required memory is based on the underlying algorithms The description below is the memory usage per thread Users with shared memory machines may use the threaded NCO operators see Section 3 3 OpenMP Threading page 27 The peak and sustained memory usage will scale accordingly i e by the number of threads Memory consumption patterns of all operators are similar with the ex
464. ta Analysis and Distribution GODAD software ecosystem Shifting more geoscience data analysis to GODAD is a long term plan Then the NASA ACCESS 2013 program agreed to support Cooperative Agreement NNX14AH55A NCO from 2014 2016 This support permits us to implement support for Swath like Data SLD Most recently the DOE has funded me to implement NCO re gridding and parallelization in support of their ACME program After many years of crafting NCO as an after hours hobby I finally have the cushion necessary to give it some real attention And I m looking forward to this next and most intense yet phase of NCO development Foreword Charlie Zender June 2015 Irvine California Summary 5 Summary This manual describes NCO which stands for netCDF Operators NCO is a suite of programs known as operators Each operator is a standalone command line program executed at the shell level like e g 1s or mkdir The operators take netCDF files including HDF5 files constructed using the netCDF API as input perform an operation e g averaging or hyperslabbing and produce a netCDF file as output The operators are primarily designed to aid manipulation and analysis of data The examples in this documentation are typical applications of the operators for processing climate model output This stems from their origin though the operators are as general as netCDF itself Chapter 1 Introduction 7 1 Introduction 1 1 Availability The c
465. tain a units attribute then NCO aborts Fourth NCO passes the specified and desired dimension strings microns are specified by the user meters are required by NCO to the UDUnits library Fifth the UDUnits library that these dimension are commensurate and it returns the appropriate linear scaling factors to convert from microns to meters to NCO If the units are incommensurate i e not expressible in the same fundamental MKS units or are not listed in the UDUnits database then NCO aborts since it cannot determine the user s intent Finally NCO uses the scaling information to convert the user specified hyperslab limits into the same physical dimensions as those of the corresponding cooridinate variable on disk At this point NCO can perform a coordinate hyperslab using the same algorithm as if the user had specified the hyperslab without requesting units conversion The translation and dimensional innterpretation of time coordinates shows a more pow erful and probably more common UDUnits application In this example the user prints all data between 4 PM and 7 PM on December 8 1999 from a variable whose time dimension is hours since the year 1900 4 ncks u H C v time udunits d time udunits 1999 12 08 16 00 0 0 1999 12 08 19 00 0 0 in nc time udunits 1 2876018 hours since 1900 01 01 00 00 0 0 Here the user invokes the stride see Section 3 16 Stride page 60 capability to obtain every other timeslice This is possible b
466. tats max var data data stride n double gsl stats mean var data data stride n double gsl stats sd with fixed mean var data data stride n var mean double gsl stats wmean var weight weight stride var data data stride n var mean double gsl stats quantile from sorted data var sorted data data stride n var f GSL has no notion of missing values or dimensionality beyond one If your data has missing values which you want ignored in the calculations then use the ncap2 built in aggregate functions Section 4 1 11 Methods and functions page 145 The GSL functions operate on a vector of values created from the var data stride n arguments The ncap wrappers check that there is no bounding error with regard to the size of the data and the final value in the vector Some examples al time 1 2 3 4 5 6 7 8 9 10 al_avg gsl_stats_mean al 1 10 print al_avg 5 5 Chapter 4 Reference Manual 173 ai var gsl stats variance a1 4 3 print ai var 16 0 bounding error vector attempts to access element a1 10 ai sd gsl stats sd ai1 5 3 For functions with the signature func nm var data data stride n one may omit the sec ond or third arguments The default value for stride is 1 The default value for n is 1 data size 1 stride Following statements are equvalent n2 gsl stats max a1 1 10 n2 gsl stats max ai1 1 n2 gsl stats max ai Following statements are equvalent n3 gsl stats median fr
467. tch Mode eoo ta iaa 121 3 38 Global Attribute Addition 0 00 eee eee 122 3 39 History Attributes ce ces eerte am wll Ehre qu ped 122 3 40 File List Attributes 0 00 cece III 123 SAL VCE Conventions Ed 124 3 42 ARM Conventions ssseseeee en 127 3 43 Operator Version sssssesssesse e en aes 127 4 Reference Manual 129 4 1 ncap2 netCDF Arithmetic Processor oooooooooocooo 130 4 1 1 Syntax of ncap2 statements 0 eee 131 4 1 2 Expressions noe ri faces HRM ad aoe ad te E Ero 131 ALS Dimensions rem EE A EI EMI 135 4 1 4 Left hand casting sessssseeseeee e 136 4 1 5 Arrays and hyperslabs 0 cee cece eee eee ees 136 ALO Attributes ste casita eres ets Reo pO RH Reds 139 4 1 7 Number literals 0 00 cece cece eects 140 4 1 8 At statement A O Sood OE y eee tn a 142 4 1 9 print statement cec ee i ice eee aya eed 143 4 1 10 Missing values ncap2 sseeeessse e 144 4 1 11 Methods and functions 00 e cece ee eee 145 4 1 12 RAM v ria bles set pent eter aed dint NE PT 149 4 1 13 Where statement 0 0 cece cee ee 149 2144 LOOPS a ere ae ee Se 151 4 1 15 Include files oer ete eee i ae 152 A16 sort methods REESE 152 4 1 17 Irregular Grids eeraa eee eee ees 157 4 1 18 Bilinear interpolation 0 0 cece eee eee eee 159 4 1 19 GSL special functions
468. te Editor page 183 for more details on string formatting and special characters As of NCO version 4 2 2 October 2012 NCO prints missing values as blanks i e the underscore character _ by default 4 ncks C H v mss val in nc lon 0 0 mss val 0 73 lon 1 90 mss_val 1 _ lon 2 180 mss_val 2 73 lon 3 270 mss_val 3 _ ncks s A4 5b 1f H C v mss val in nc Chapter 4 Reference Manual 215 73 0 _ 73 0 _ To print the numeric value of the missing value instead of a blank use the no_blank option ncks prints in a verbose fashion by default and supplies a number of switches to pare down or even spruce up the output The interplay of the Q V and otherwise un documented no_nm_prn switches yields most desired verbosities 4 ncks v three dmn rec var C H nco data in nc time 0 1 lat 0 90 lon 0 0 three dmn rec var 0 1 4 ncks Q v three dmn rec var C H nco data in nc three dmn rec var 0 1 4 ncks V v three dmn rec var C H nco data in nc 1 ncks Q no nm prn v three dmn rec var C H nco data in nc 1 ncks no nm prn v three dmn rec var C H nco data in nc 1 90 0 1 One dimensional arrays of characters stored as netCDF variables are automatically printed as strings whether or not they are NUL terminated e g necks v fl nm in nc The 4c formatting code is useful for printing multidimensional arrays of characters repre s
469. te mode is selected the att type and att val arguments are superfluous and may be left blank Chapter 4 Reference Manual 185 m Modify Change value of current var nm attribute att nm to value att val If var nm does not have an attribute att nm there is no effect o Overwrite Write attribute att nm with value att_val to variable var_nm over writing existing attribute att nm if any This is the default mode The value of att type is a single character abbreviation f d 1 i s c b u or a short string standing for one of the twelve primitive netCDF data types f Float Value s specified in att_val will be stored as netCDF intrinsic type NC FLOAT d Double Value s specified in att_val will be stored as netCDF intrinsic type NC DOUBLE id Integer or Long Value s specified in att val will be stored as netCDF intrinsic type NC INT s Short Value s specified in att val will be stored as netCDF intrinsic type NC_SHORT Char Value s specified in att val will be stored as netCDF intrinsic type NC CHAR b Byte Value s specified in att_val will be stored as netCDF intrinsic type NC BYTE ub Unsigned Byte Value s specified in att val will be stored as netCDF intrinsic type NC UBYTE us Unsigned Short Value s specified in att val will be stored as netCDF intrinsic type NC USHORT u ui ul Unsigned Int Value s specified in att_val will be stored as netCDF intrinsic type NC UINT 11 int64 Int64
470. ted variables ncks 4 L 1 t 8 v FS T map map nc in nc out nc OpenMP threading works well with regridding large datasets Threading improves throughput of regridding 1 10 GB files by factors of 2 5 Options specific to regridding are described below NCO supports 1D gt 1D 1D gt 2D 2D gt 1D and 2D 2D regridding for any unstructured 1D grid and any rectangular 2D grid This has been tested by converting among and between Gaussian equiangular FV unstructured cubed sphere grids and regionally refined grids Support for irregular 2D and regional grids e g swath like data is planned First order accurate data regridding is a straightforward procedure of identifying gridcell overlap and apportioning values correctly from source to destination The presence of missing values forces a decision on how to handle destination gridcells where some but not all source cells are valid NCO allows the user to choose between two distinct algorithms conservative and renormalized The conservative algorithm uses all valid data from the input grid on the output grid once and only once Destination cells receive the weighted valid values of the source cells This is conservative because the global integrals of the source and destination fields are equal The renormalized algorithm divides the destination value by the sum of the valid weights This produces values equal to the mean of the valid input values but extended to the en
471. th netCDF Operators NCO Environ Modell Softw doi 10 1016 j envsoft 2008 03 004 This paper is also available at http dust ess uci edu ppr ppr_Zen08 pdf NCO performance and scaling for arithmetic operations is described in Zender C S and H J Mangalam 2007 Scaling Properties of Common Statistical Opera tors for Gridded Datasets Int J High Perform Comput Appl 21 4 485 498 doi 10 1177 1094342007083802 This paper is also available at http dust ess uci edu ppr ppr_ZeM07 pdf It is helpful to be aware of the aspects of NCO design that can limit its performance 1 No data buffering is performed during nc_get_var and nc_put_var operations Hy perslabs too large too hold in core memory will suffer substantial performance penalties because of this 2 Since coordinate variables are assumed to be monotonic the search for bracketing the user specified limits should employ a quicker algorithm like bisection than the two sided incremental search currently implemented 3 C format FORTRAN_format signedness scale format and add offset attributes are ignored by ncks when printing variables to screen 4 In the late 1990s it was discovered that some random access operations on large files on certain architectures e g UNICOS were much slower with NCO than with similar operations performed using languages that bypass the netCDF interface e g Yorick This may have been a penalty of unnecessary byte swapp
472. th netCDF Operators NCO Environ Modell Softw 23 10 1338 1342 doi 10 1016 j envsoft 2008 03 004 e WZJ07 Wang D L C S Zender and S F Jenks 2007 DAP enabled Server side Data Reduction and Analysis Proceedings of the 23rd AMS Conference on Interac tive Information and Processing Systems IIPS for Meteorology Oceanography and Hydrology Paper 3B 2 January 14 18 San Antonio TX American Meteorological Society AMS Press Boston MA e ZMWO06 Zender C S H Mangalam and D L Wang 2006 Improving Scaling Properties of Common Statistical Operators for Gridded Geoscience Datasets Eos Trans AGU 87 52 Fall Meet Suppl Abstract IN53B 0827 e ZeW07 Zender C S and D L Wang 2007 High performance distributed data reduction and analysis with the netCDF Operators NCO Proceedings of the 23rd AMS Conference on Interactive Information and Processing Systems IIPS for Mete orology Oceanography and Hydrology Paper 3B 4 January 14 18 San Antonio TX American Meteorological Society AMS Press Boston MA e WZJ06 Wang D L C S Zender and S F Jenks 2006 Server side netCDF Data Reduction and Analysis Eos Trans AGU 87 52 Fall Meet Suppl Abstract IN53B 0826 e WZJ073 Wang D L C S Zender and S F Jenks 2007 Server side parallel data reduction and analysis in Advances in Grid and Pervasive Computing Second International Conference GPC 2007 Paris France May 2 4 2007 P
473. th the variable FSNT from typical un structured grid CAM SE cubed sphere output before and after an identity regridding Raw model output before regridding netcdf ne30_FSNT dimensions nbnd 2 ncol 48602 time UNLIMITED 1 currently variables float FSNT time ncol FSNT long_name Net solar flux at top of model double time time time long_name time time bounds time_bnds double time_bnds time nbnd time_bnds long_name time interval endpoints group Same model output after identity regridding netcdf dogfood dimensions nbnd 2 ncol 48602 1 nv time Io I variables float FSNT time ncol FSNT long_name Net solar flux at top of model FSNT coordinates lat lon double lat ncol lat long name latitude lat standard name latitude lat units degrees north 78 NCO 4 5 3 alpha03 User Guide lat axis Y lat bounds lat_vertices lat coordinates lat lon double lat_vertices ncol nv lat vertices long name gridcell latitude vertices double lon ncol lon long name longitude lon standard name longitude lon units degrees east lon axis X lon bounds lon vertices lon coordinates lat lon double lon vertices ncol nv lon vertices long name gridcell longitude vertices double time time time long name time time bounds time bnds double time bnds time nbnd time bnds
474. that climate scientists are concerned about global temperature variations of a tenth of a degree or less this difference is large Global mean temperature changes significant to climate scientists are comparable in size to the numerical artifacts produced by the averaging procedure Why are the single precision numerical artifacts so large Each global average is the result of multiplying almost 15 000 elements each by its weight summing those and then dividing by the summed weights Thus about 50 000 single precision floating point opera tions caused the loss of two to three significant digits of precision The net error of a series of independent rounding errors is a random walk phenomena Successive rounding errors Thanks to Michael J Prather for explaining this to me 120 NCO 4 5 3 alpha03 User Guide displace the answer further from the truth An ensemble of such averages will on average have no net bias In other words the expectation value of a series of IEEE rounding errors is zero And the error of any given sequence of rounding errors obeys for large series a Gaussian distribution centered on zero Single precision numbers use three of their four eight bit bytes to represent the mantissa so the smallest representable single precision mantissa is e 27 1 19209 x 1077 This e is the smallest x such that 1 0 4 x Z 1 0 This is the rounding error for non exact precision numbers Applying random walk theory to rounding
475. the following ncap2 invocations produce identical results ncap2 v S tst nco in nc out nc ncap2 v s a 3 s b 4 s c sqrt a 2 b 2 in nc out nc ncap2 v s a 3 b 4 c sqrt a 2 b 2 in nc out nc The second and third examples show that ncap2 does not require that a trailing semi colon be placed at the end of a s argument although a trailing semi colon is always allowed However semi colons are required to separate individual assignment statements chained together as a single s argument ncap2 may be used to grow dimensions i e to increase dimension sizes without al tering existing data Say in nc has ORO lat lon and the user wishes a new file with new ORO new lat new lon that contains zeros in the undefined portions of the new grid defdim new lat 1at size 1 Define new dimension sizes defdim new lon 10n size 1 new ORO new lat new 1on 20 0f Initialize to zero new ORO 0 1at size 1 0 10n size 1 0R0 Fill valid data The commands to define new coordinate variables new lat and new lon in the output file follow a similar pattern One would might store these commands in a script grow nco and then execute the script with ncap2 v S grow nco in nc out nc Chapter 4 Reference Manual 177 Imagine you wish to create a binary flag based on the value of an array The flag should have value 1 0 where the array exceeds 1 0 and value 0 0 elsewhere This example cr
476. the monthly data instead of subtracting the long term average This is a little more complicated in coding since we need to loop over years Flat files example it bin bash d222222222222 2 22 2 2 2 2 2 2 2 2 2 2 2 2 22 2 2 2 2222222222 2 2 2 2 2 2 22 2 2 2 After cmb fl sh Example Monthly cycle of each model in Greenland Input files data cmip5 snc_LImon_bcc csm1 1_historical_riiip1_185001 200512 nc Output files data cmip5 snc snc_LImon__all mdl_historical_all nsm_GN_mthly anm nc Online http nco sourceforge net nco html Monthly Cycle Execute this script bash mcc sh d222222222 22222 2 2 22 22 2 2 2 2 2 2 2 222 2 2 22222222 2 2 2 2 2 2 2 2 2 2 2 2 2 a ee Ln nn nn Parameters drc_in home wenshanw data cmip5 Directory of input files drc_out home wenshanw data cmip5 output Directory of output files var snc snd Variables 264 NCO 4 5 3 alpha03 User Guide rlm LImon Realm xpt historical Experiment could be more fld_out snc snd Folders of output files l M for var_id in 0 1 do Loop over two variables names of all models 1s get file names cut get the part for model names sort uniq remove duplicates awk print mdl set ls drc_in var var_id _ rlm _ _ xpt 0 _ nc cu
477. the ncra executable The ncra executable behaves slightly differently based on its invocation name i e argv 0 which can be nces ncra or ncrcat Logically these are three different operators that happen to share the same executable For historical reasons and to be more user friendly multiple synonyms or pseudonyms may refer to the same operator invoked with different switches For example ncdiff is the same as ncbo and ncpack is the same as ncpdq We implement the symbolic links and synonyms by the executing the following UNIX commands in the directory where the NCO executables are installed ln s f ncbo ncdiff ncbo op typ ln s f ncra nces ncra pseudonym nces The Cygwin package is available from http sourceware redhat com cygwin Currently Cygwin 20 x comes with the GNU C C compilers gcc g These GNU compilers may be used to build the netCDF distribution itself Chapter 1 Introduction 11 ln s f ncra ncrcat ncra pseudonym ncrcat ln s f ncbo ncadd ncbo op_typ ln s f ncbo ncsubtract ncbo op_typ ln s f ncbo ncmultiply ncbo op_typ ln s f ncbo ncdivide ncbo op typ ln s f ncpdq ncpack ncpdq ln s f ncpdq ncunpack ncpdq unpack NB Windows Cygwin executable link names have exe suffix e g ln s f ncbo exe ncdiff exe The imputed command called by the link is given after the comment As can be seen some thes
478. thin machine precision This information is also useful for benchmarking 3 4 Command Line Options peras All operators NCO achieves flexibility by using command line options These options are implemented in all traditional UNIX commands as single letter switches e g 1s 1 For many years NCO used only single letter option names In late 2002 we implemented GNU POSIX extended or long option names for all options This was done in a backward compatible way such that the full functionality of NCO is still available through the familiar single letter options In the future however some features of NCO may require the use of long options simply because we have nearly run out of single letter options More importantly mnemonics for single letter options are often non intuitive so that long options provide a more natural way of expressing intent Extended options also called long options are implemented using the system supplied getopt h header file if possible This provides the getopt long function to NCO The syntax of short options single letter options is key value dash key space value Here key is the single letter option name e g D 2 The syntax of long options multi letter options is 1ong name value dash dash key space value e g amp dbg lvl 2 or long name value dash dash key equal value e g dbg lv1 2 Thus the following are all valid for the D short version or dbg_1v1
479. this situation multi file operators store the number of input files in the nco input file number global attribute and the input file list itself in the nco input file list global attribute see Section 3 40 File List Attributes page 123 Although this does not preserve the exact command used to generate the file it does retains all the information required to reconstruct the command and determine the data provenance A second option is to use the UNIX xargs command This simple example selects as input to xargs all the filenames in the current directory that match a given pattern For illustration consider a user trying to average millions of files which each have a six character filename If the shell buffer cannot hold the results of the corresponding globbing operator pattern as an extended regular expression X nc see Section 3 11 Subsetting Files page 43 We use grep to filter the directory listing for this pattern and to pipe the results to xargs which in turn passes the matching filenames to an NCO multi file operator e g ncecat Use xargs to transfer filenames on the command line bin ls grep CASEID _ nc xargs x ncecat o foo nc The single quotes protect the only sensitive parts of the extended regular expression the grep argument and allow shell interpolation the CASEID variable substitution to proceed unhindered on the rest of the command xargs uses the UNIX pipe feature to append th
480. tire destination gridcell Thus renormalization is equivalent to extrapolating valid data to missing regions Input and output integrals are unequal and renormalized regridding is not conservative Both algorithms produce identical answers when no missing data maps to the destination gridcell By default NCO implements the conservative algorithm because it has useful prop erties is simpler to understand and requires no additional parameters To employ the renormalized algorithm instead use the rnr rgr_rnr or renormalize op tions to supply wgt_thr the threshold weight for valid destination values Valid values must cover at least the fraction wgt_thr of the destination gridcell to meet the threshold for a non missing destination value When wgt_thr is exceeded the mean valid value is renor malized by the valid area and placed in the destination gridcell If the valid area covers less than wgt_thr then the destination gridcell is assigned the missing value Valid values of wgt_thr range from zero to one Keep in mind though that this threshold is potentially a divisor and values of zero or very near to zero can lead to floating point underflow and divide by zero errors For convenience NCO permits users to specify a wgt_thr 0 0 thresh old weight This indicates that any valid data should be represented and renormalized on the output grid ncks map map nc in nc out nc Conservative regridding ncks rn
481. tmask to alter the significand of IEEE 754 floating point data The bitmask is one for all bits to be retained and zero or one for all bits to be ignored The algorithm assumes that the number of binary digits i e bits necessary to represent a single base 10 digit is In 10 In 2 3 32 The exact numbers of bits Nbit retained for single and double precision values are ceil 3 32 x nsd 1 and ceil 3 32 x nsd 2 respectively Once these reach 23 and 53 respectively bitmasking is completely ineffective This occurs at nsd 6 3 and 15 4 respectively 94 NCO 4 5 3 alpha03 User Guide The DSD algorithm by contrast uses rounding to remove undesired precision The rounding zeroes the greatest number of significand bits consistent with the desired preci sion To demonstrate the change in IEEE representation caused by PPC rounding algorithms consider again the case of pi represented as an NC FLOAT The IEEE 754 single precision representations of the exact value 3 141592 the value with only three significant digits treated as exact 3 140000 and the value as stored 3 140625 after PPC rounding with either the NSD prc 3 or DSD prc 2 algorithm are respectively S Exponent Fraction Significand Decimal Notes O 100000001 0010010000111111011011 3 14159265 Exact O 100000001 0010001111010111000011 3 14000000 O 100000001 0010010000000000000000 3 14062500 NSD O 100000001 0010010000000000000000 3 14062500 DSD
482. tml climatological statistics This convention allows coordinate variables including mul tidimensional coordinates to describe the possibly nested periods and statistical methods of their associated statistics T his is done by naming the variable which contains the periods and methods in the climatology attribute Note that coordinates of rank N have climatol 126 NCO 4 5 3 alpha03 User Guide ogy bounds of rank N 1 NCO generated subsets of CF compliant files with climatology attributes will include the variables specified by the climatology attribute if any Hence the subsets will themselves be CF compliant Climatology variables are subject to the user specified override switches including c and C described in Section 3 12 Subsetting Coordinate Variables page 47 As of NCO version 4 4 5 July 2014 NCO supports the CF ancillary variables con vention for described here http cfconventions org 1 6 html ancillary data This convention allows ancillary variables to be associated with one or more primary vari ables NCO attaches any such variables to the extraction list along with the primary variable and its usual one dimensional coordinates if any Ancillary variables are subject to the user specified override switches including c and C described in Section 3 12 Subset ting Coordinate Variables page 47 As of NCO version 3 9 6 January 2009 NCO supports the CF coordinates convention de
483. to hold the gridcell area in steradians To store the area in a different variable use the option rgr area area nm The area_nm variable receives a standard name attribute of cell area a units attribute of steradian the SI unit of solid angle and a cell methods attribute with value lat lon sum which indicates that area nm is extensive meaning that its value depends on the gridcell boundaries Since area_nm is a property of the grid it is read directly from the map file rather than regridded itself Latitude weights lat wgt Rectangular 2D grids use the variable lat_wgt_nm which defaults to gw orig inally for Gaussian weight to store the 1D weight appropriate for area weighting the latitude grid To store the latitude weight in a different variable use the option rgr lat wgt lat wgt nm The lat wgt nm variable will not appear in 1D grid output Weighting statistics by latitude i e by lat_wgt_nm will produce the same answers up to round off error as weighting by area i e by area_nm in grids that have both variables The former requires less memory because lat_wgt_nm is 1D whereas the latter is more general because area_nm works on any grid Provenance Attributes The map file and input file names are stored in the output file global attributes mapping file and source file respectively One may supply muliple rgr key value options to simultaneously customize multi ple grid field names The following ex
484. too many interactive questions slows productivity Therefore NCO also implements two ways to override its own safety features the 0 and A switches Specifying 0 tells the operator to overwrite any existing output file without prompting the user interactively Specifying A tells the 2 The OS specific system move command is used This is mv for UNIX and move for Windows Chapter 2 Operator Strategies 19 operator to attempt to append to any existing output file without prompting the user inter actively T hese switches are useful in batch environments because they suppress interactive keyboard input 2 4 Appending Variables Adding variables from one file to another is often desirable l his is referred to as appending although some prefer the terminology merging or pasting Appending is often confused with what NCO calls concatenation In NCO concatenation refers to splicing a variable along the record dimension The length along the record dimension of the output is the sum of the lengths of the input files Appending on the other hand refers to copying a variable from one file to another file which may or may not already contain the variable NCO can append or concatenate just one variable or all the variables in a file at the same time In this sense ncks can append variables from one file to another file This capability is invoked by naming two files on the command line input file and output file When
485. tput However the intermediate results where running sums are accumulated may suffer from too much round ing or from underflow unless computed in double precision The order of operations matters to floating point math even when the analytic expres sions are equal Cautious users feel disquieted when results from equally valid analyses differ in the final bits instead of agreeing bit for bit For example averaging arrays in multiple stages produces different answers than averaging them in one step This is easily seen in the computation of ensemble averages by two different methods The NCO test file in nc contains single and double precision representations of the same temperature timeseries as tpt flt and tpt_dbl Pretend each datapoint in this timeseries represents a monthly mean temperature We will mimic the derivation of a fifteen year ensemble mean January temperature by concatenating the input file five times and then averaging the datapoints representing January two different ways In Method 1 we derive the 15 year ensemble January average in two steps as the average of three five year averages This method is naturally used when each input file contains multiple years and multiple input files are needed In Method 2 we obtain 15 year ensemble January average in a single step by averaging all 15 Januaries at one time tpt flt and tpt dbl are identical except for precision ncks cdl C v tpt flt tpt dbl nco data in nc tpt dbl
486. tude in the output file depends on the weights in lat wgt and is not likely to be 45 0 the midpoint latitude of the hyperslab Op tion I overrides this default behavior and causes ncwa not to weight or mask coordinate variables In the above case this causes the value of latitude in the output file to be 45 0 an appealing result Thus I specifies simple arithmetic averages for the coordinate vari ables In the case of latitude I specifies that you prefer to archive the arithmetic mean latitude of the averaged hyperslabs rather than the area weighted mean latitude As explained in See Section 3 35 Operation Types page 107 ncwa always averages coordinate variables regardless of the arithmetic operation type performed on the non coordinate variables This is independent of the setting of the I option The mathematical definition of operations involving rank reduction is given above see Section 3 35 Operation Types page 107 4 12 1 Mask condition Each z also has an associated masking weight m whose value is 0 or 1 false or true The value of m is always 1 unless a mask var is specified with m As noted above mask var is broadcast if possible to conform to the variable being averaged In this case the value of m depends on the mask condition also known as the truth condition As expected m 1 when the mask condition is true and m 0 otherwise The mask condition has the syntax mask_var
487. two dimensional variable It must be of size grid out x size grid out y size O grid out x X output values grid out y Y output values grid in x X input values values Must be monotonic increasing or decreasing grid in y Y input values values Must be monotonic increasing or decreasing Prior to calculations all arguments are converted to type NC DOUBLE After calculations var out is converted to the input type of grid in Suppose the first part of an ncap2 script is 160 NCO 4 5 3 alpha03 User Guide defdim X 4 defdim Y 5 Temperature T in X Y 1100 200 300 400 500 101 202 303 404 505 102 204 306 408 510 103 206 309 412 515 0 Coordinate variables x_in X 0 0 1 0 2 0 3 01 y_in Y 1 0 2 0 3 0 4 0 5 Now we interpolate with the following variables defdim Xn 3 defdim Yn 4 T_out Xn Yn 0 0 x_out Xn 0 0 0 02 3 01 y_out Yn 1 1 2 0 3 4 var out bilinear interp T in T out x out y out x in y in print var out 110 200 300 400 110 022 200 04 300 06 400 08 113 3 206 309 412 It is possible to interpolate a single point var out bilinear interp T in 0 0 3 0 4 99 x in y in print var out 513 920594059406 Wrapping and Extrapolation The function bilinear_interp_wrap takes the same arguments as bilinear_interp but performs wrapping Y and extrapolation X for points off the edge of the grid If the gi
488. type containing the value _FillValue As a guideline the type of the _FillValue attribute should be the same as the type of the variable it is attached to If var type equals att type then NCO straightforwardly compares each value of var to _FillValue to de termine which elements of var are to be treated as missing data If not then NCO converts _FillValue from att type to var type by using the implicit conversion rules of C or if att type is NC_CHAR by typecasting the results of the C function strtod _ FillValue You may use the NCO operator ncatted to change the _FillValue at tribute and all data whose data is _FillValue to a new value see Section 4 2 ncatted netCDF Attribute Editor page 183 Identifying missing data during arithmetic operations When an NCO arithmetic operator processes a variable var with a _FillValue at tribute it compares each value of var to _FillValue before performing an operation Note the _Fill Value comparison imposes a performance penalty on the operator Arith metic processing of variables which contain the FillValue attribute always incurs this penalty even when none of the data are missing Conversely arithmetic processing of variables which do not contain the FillValue attribute never incurs this penalty In other words do not attach a FillValue attribute to a variable which does not contain missing data This exhortation can usually be obeyed for model generated data but it may be harder
489. u amp item_name NCO development amp item_number nco_dnt_dvl amp no_note 1 amp tax 0 amp currency_code USD NCO has been shared for over 10 years yet only two users have contributed any money to the developers So you could be the third 5 1 Contributors NCO would not exist without the dedicated efforts of the remarkable software engineers who conceive develop and maintain netCDF UDUnits and OPeNDAP Since 1995 NCO has received support from I believe the entire staff of all these projects including Russ Rew John Caron Glenn Davis Steve Emmerson James Gallagher Ed Hartnett and Dennis Heimbigner In addition to their roles in maintaining the software stack on which NCO perches Yertl like some of these gentlemen have advised or contributed to NCO specifically That support is acknowledged separately below The primary contributors to NCO development have been Charlie Zender All concept design and implementation from 1995 2000 Since then auto tools bug squashing CDL chunking documentation anchoring recursion GPE packing NCO library redesign ncap2 features ncbo ncpdq SMP thread ing and MPI parallelization netCDF4 integration external funding project management science research releases Henry Butowsky Non linear operations and min max total support in ncra and ncwa Type conversion for arithmetic Migration to netCDF3 API ncap2 parser lexer GSL support and I O Multislabbing algorithm Var
490. ually multiple scripting commands are pre stored ncap2 was written to perform arbi trary algebraic transformations of data and archive the results as easily as possible See Section 3 27 Missing Values page 84 for treatment of missing values The results of the algebraic manipulations are called derived fields e Unlike the other operators ncap2 does not accept a list of variables to be operated on as an argument to v see Section 3 11 Subsetting Files page 43 Rather the v switch takes no arguments and indicates that ncap2 should output only user defined variables ncap2 neither accepts nor understands the x switch NB As of 20120515 ncap2 is unable to append to files that already contain the appended dimensions Defining new variables in terms of existing variables is a powerful feature of ncap2 De rived fields inherit the metadata i e attributes of their ancestors if any in the script or input file When the derived field is completely new no identically named ancestors exist then it inherits the metadata if any of the left most variable on the right hand side of the defining expression This metadata inheritance is called attribute propagation At tribute propagation is intended to facilitate well documented data analysis and we welcome suggestions to improve this feature The only exception to this rule of attribute propagation is in cases of left hand casting see Section 4 1 4 Left hand casting
491. uch names may cause the DAP service to fail because DAP interprets the period as structure delimiter in an HTTP query string 38 NCO 4 5 3 alpha03 User Guide Note that the remote retrieval features of NCO can always be used to retrieve any file including non netCDF files via SSH anonymous FTP or msrcp Often this method is quicker than using a browser or running an FTP session from a shell window yourself For example say you want to obtain a JPEG file from a weather server ncks R p ftp weather edu pub pix jpeg 1 storm jpg In this example ncks automatically performs an anonymous FTP login to the remote machine and retrieves the specified file When ncks attempts to read the local copy of storm jpg as a netCDF file it fails and exits leaving storm jpg in the current directory If your NCO is DAP enabled see Section 3 7 1 OPeNDAP page 35 then you may use NCO to retrieve any files including netCDF HDF etc served by an OPeNDAP server to your local machine For example ncks R 1 p http www esrl noaa gov psd thredds dodsC Datasets ncep reanalysis dailyavgs surface pres sfc 1969 nc It may occasionally be useful to use NCO to transfer files when your other preferred methods are not available locally 3 9 File Formats and Conversion Availability ncap2 nces ncecat ncflint ncks ncpdq ncra ncrcat ncwa Short options 3 4 6 7 Long options 3 4 6
492. ular expressions and in shell level filename expansion are significantly different In an extended regular expression matches zero or more occurences of the preceding regular expression Thus Q selects all variables and Q selects all variables containing Q the ensures the preceding item matches at least once To match zero or one occurence of the preceding regular expression use 7 Documentation for the UNIX egrep command details the extended regular expressions which NCO supports One must be careful to protect any special characters in the regular expression specifica tion from being interpreted globbed by the shell This is accomplish by enclosing special characters within single or double quotes ncra v Q in nc out nc Error Shell attempts to glob wildcards ncra v Q in nc out nc Correct NCO interprets wildcards ncra v Q in nc out nc Correct NCO interprets Shell globs The final example shows that commands may use a combination of variable wildcarding and shell filename expansion globbing For globbing and have nothing to do with the preceding regular expression In shell level filename expansion matches any string including the null string and matches any single character Documentation for bash and csh describe the rules of filename expansion globbing 3 12 Subsetting Coordinate Variables 48 NCO 4 5 3 alpha03 User Guide Avail
493. uments 0 is the first dimension of a variable 1 is the second dimension of a variable n is the n 1 dimension of a variable float four dmn rec var time lat lev lon double three dmn var dbl time lat lon four nw four dmn rev var reverse time 10n four nw four dmn rec var reverse 0 3 four avg four dmn rec var avg 1at 1ev four avg four dmn rec var avg 1 2 three mw three dmn var dbl permute time 10n 1at three mw three dmn var dbl permute 0 2 1 ID Quoting If the dimension name contains non regular characters use ID quoting See see Section 4 1 27 ID Quoting page 181 defdim a list A 10 Ai a list A 230 0 136 NCO 4 5 3 alpha03 User Guide GOTCHA It is not possible to manually define in Output any dimensions that exist in Input When a variable from Input appears in an expression or statement its dimensions in Input are automagically copied to Output if they are not already present 4 1 4 Left hand casting The following examples demonstrate the utility of the left hand casting ability of ncap2 Consider first this simple artificial example If lat and lon are one dimensional coordinates of dimensions lat and lon respectively then addition of these two one dimensional arrays is intrinsically ill defined because whether lat_lon should be dimensioned lat by lon or lon by lat is ambiguous assuming that addition is to remain a commutative procedure i e one that does not depend on the
494. ummarizes quantization accuracy based on the three metrics NSD Number of Significant Digits Emabs Maximum absolute error Emebs Mean absolute error Emean Mean error Artificial Data N 1000000 values in 1 0 2 0 in steps of 1 0e 6 Single Precision Double Precision Single Precision NSD Emabs Emebs Emean Emabs Emebs Emean DSD Emabs Emebs Emean 1 0 31 0 11 4 1e 4 0 31 0 11 4 0e 4 1 0 30 0 11 8 1e 4 24 The artificial dataset employed is one million evenly spaced values from 1 0 2 0 The analysis data are N 13934592 values of the temperature field from the NASA MERRA analysis of 20130601 98 NCO 4 5 3 alpha03 User Guide 2 0 39 0 14 6 8e 5 0 39 0 14 5 5e 5 2 0 39 0 14 1 3e 4 3 0 49 0 17 1 0e 6 0 49 0 17 5 5e 7 3 0 49 0 17 2 0e 5 4 0 30 0 11 3 2e 7 0 30 0 11 6 1e 6 4 0 30 0 11 5 1e 8 5 0 37 0 13 3 1e 7 0 38 0 13 5 6e 6 5 0 38 0 13 2 6e 6 6 0 36 0 12 4 4e 7 0 48 0 17 4 1e 7 6 0 48 0 17 7 2e 6 7 0 00 0 00 0 0 0 30 0 10 1 5e 7 7 0 00 0 00 0 0 Observational Analysis N 13934592 values MERRA Temperature 20130601 Single Precision NSD Emabs Emebs Emean 1 0 31 0 11 2 4e 3 2 0 39 0 14 3 8e 4 3 0 49 0 17 9 6e 5 4 0 30 0 11 2 3e 3 5 0 37 0 13 2 2e 3 6 0 36 0 13 1 7e 2 7 0 00 0 00 0 0 All results show that PPC quantization performs as expected Absolute maximum errors Emabs 0 5 for all prc For 1 lt prc lt 6 quantization results in comparable maximum absolute and mean absolute errors Emabs and Emebs respectively Me
495. units In the following example the user extracts all wavelengths in the visible portion of the spectrum in terms of the units very frequently used in visible spectroscopy microns 4 ncks C H v wvl d wvl 0 4 micron 0 7 micron in nc wvl 0 25e 07 meter The hyperslab returns the correct values because the wvl variable is stored on disk with a length dimension that UDUnits recognizes in the units attribute The automagical algorithm that implements this functionality is worth describing since understanding it 80 NCO 4 5 3 alpha03 User Guide helps one avoid some potential pitfalls First the user includes the physical units of the hyperslab dimensions she supplies separated by a simple space from the numerical values of the hyperslab limits She encloses each coordinate specifications in quotes so that the shell does not break the value space unit string into separate arguments before passing them to NCO Double quotes foo or single quotes foo are equally valid for this purpose Second NCO recognizes that units translation is requested because each hyperslab argument contains text characters and non initial spaces Third NCO determines whether the wvl is dimensioned with a coordinate variable that has a units attribute In this case wvl itself is a coordinate variable The value of its units attribute is meter Thus wvl passes this test so UDUnits conversion is attempted If the coordinate associated with the variable does not con
496. unt able items For example time weighting monthly averages for March April and May 226 NCO 4 5 3 alpha03 User Guide to obtain a spring seasonal average can be done with w 31 30 31 instead of w 0 33695652173913043478 0 32608695652173913043 0 33695652173913043478 However sometimes one wishes to use weights in dot product mode i e multiply by the non normalized weights As of NCO version 4 5 2 released in July 2015 ncra accepts the N or long option equivalent no nrm by wgt switch that prevents auto matic weight normalization When this switch is used the weights will not be normalized unless the user provides them as normalized and the numerator of the weighted average will not be divided by the sum of the weights which is one for normalized weights Bear these two exceptions in mind when weighting input First ncra only applies weights if the arithmetic operation type is averaging see Section 3 35 Operation Types page 107 i e for timeseries mean and for timeseries mean absolute value Second weights are never applied for minimization square roots etc ncra never weights coordinate variables e g time regardless of the weighting performed on non coordinate variables EXAMPLES Average files 85 nc 86 nc 89 nc along the record dimension and store the results in 8589 nc ncra 85 nc 86 nc 87 nc 88 nc 89 nc 8589 nc ncra 8 56789 nc 8589 nc ncra n 5 2 1 85 nc 8589 nc These thr
497. ups of records These groups may be connected to physical sub cycles of a periodic nature e g months of a year or hours of a day Or they may be thought of as groups of a specifed duration The feature and the Chapter 3 Shared Features 63 terminology to describe it are new For now we call this the subcycle feature sometimes abbreviated SSC The subcycle feature allows processing of groups of records separated by regular intervals of records It is perhaps best illustrated by an extended example which describes how to solve the same problem both with and without the SSC feature The first task in climate data processing is often creating seasonal cycles Suppose a 150 year climate simulation produces 150 output files each comprising 12 records each record a monthly mean 1850 nc 1851 nc 1999 nc Our goal is to create a single file containing the summertime June July and August aka JJA mean Traditionally we would first compute the climatological monthly mean for each month of summer Each of these is a 150 year mean i e Step 1 Create climatological monthly files clm06 nc clm08 nc for mth in 6 8 do mm printf 02d mth ncra 0 F d time mm 12 n 150 4 1 1850 nc clm mm nc done Step 2 Average climatological monthly files into summertime mean ncra 0 clm06 clm07 nc clm08 nc cim JJA nc So far nothing is unusual and this task can be performed by any NCO version The SSC feature makes obsolete the n
498. use the mask within ncap2 for variable specific processing and or with other operators e g ncwa ncdiff for entire file processing Before describing the construction of masks let us review how irregularly gridded geo science data are described Say that latitude and longitude are stored as R dimensional arrays and the product of the dimension sizes is the total number of elements N in the other variables Geoscience applications tend to use R 1 R 2 and R 3 If the grid is has no simple representation e g discontinuous then it makes sense to store all coordinates as 1D arrays with the same size as the number of grid points These gridpoints can be completely independent of all the other own weight area etc R 1 lat number of gridpoints and lon number of gridpoints If the horizontal grid is time invariant then R 2 is common R 2 lat south north east west and lon south north east west The Weather and Research Forecast WRF model uses R 3 R 3 lat time south north east west lon time south north east west and so supports grids that change with time Grids with R 1 often use missing values to indicated empty points For example so called staggered grids will use fewer east west points near the poles and more near the equator netCDF only accepts rectangular arrays so space must be allocated for the maximum number of east west points at all latitudes Then the application writes missing values into
499. users most of whom I have never met have encouraged the development of NCO Thanks espcially to Jan Polcher Keith Lindsay Arlindo da Silva John Sheldon and William Weibel for stimulating suggestions and correspondence Your encouragment motivated me to complete the NCO User Guide So if you like NCO send me a note I should mention that NCO is not connected to or officially endorsed by Unidata ACD ASP CGD or Nike Charlie Zender May 1997 Boulder Colorado Major feature improvements entitle me to write another Foreword In the last five years a lot of work has been done to refine NCO NCO is now an open source project and appears to be much healthier for it The list of illustrious institutions that do not endorse NCO continues to grow and now includes UCI Charlie Zender October 2000 Irvine California The most remarkable advances in NCO capabilities in the last few years are due to con tributions from the Open Source community Especially noteworthy are the contributions of Henry Butowsky and Rorik Peterson Charlie Zender January 2003 Irvine California NCO was generously supported from 2004 2008 by US National Science Founda tion NSF grant IIS 0431203 http www nsf gov awardsearch showAward do AwardNumber 0431203 This support allowed me to maintain and extend core NCO code 2 NCO 4 5 3 alpha03 User Guide and others to advance NCO in new directions Gayathri Venkitachalam helped implement MPI Har
500. v vi Yes g1 v1 No g2 g1 v1 vi g2 v1 ncks v g1 v1 Yes g1 v1 No g2 g1 v1 vi g2 v1 dk db Gb Gb H Gt Union mode subsetting invoke with unn or union ncks g gi v vi unn All variables in gi or progeny or named vi ncks g gl v vi unn All variables in g1 or progeny or named v1 ncks g gl v vi unn All variables in gl or named v1 ncks g g1 v vi unn All variables in g1 or named v1 The first command g g1 v v1 extracts the variable v1 from any group named g1 or descendent gi The second command extracts v1 from any root group named g1 and any descendent groups as well The third and fourth commands are equivalent ways of extracting v1 only from the root group named g1 not its descendents The fifth and sixth commands are equivalent ways of extracting the variable v1 only from the root group named g1 Subsetting in union mode with unn causes all variables to be extracted which meet either one or both of the specifications of the variable and group specifications Union mode subsetting is simply the logical OR of intersection mode subsetting As discussed below the group and variable specifications may be comma separated lists of regular expressions for added control over subsetting Remember if averaging or concatenating large files stresses your systems memory or disk resources then the easiest solution is often to subset with g and or v to retain only t
501. valents nintap or pth and path respectively Techniques to augment these methods to specify ar bitrary numbers e g thousands and patterns of filenames are discussed separately see Section 2 7 Large Numbers of Files page 21 To illustrate these methods consider the simple problem of using ncra to average five input files 85 nc 86 nc 89 nc and store the results in 8589 nc Here are the four methods in order They produce identical answers ncra 85 nc 86 nc 87 nc 88 nc 89 nc 8589 nc ncra 8 56789 nc 8589 nc ncra 8 nc 8589 nc ncra p input path 85 nc 86 nc 87 nc 88 nc 89 nc 8589 nc ncra n 5 2 1 85 nc 8589 nc The first method explicitly specifying all filenames works by brute force The sec ond method relies on the operating system shell to glob expand the regular expression 8 56789 nc The shell then passes the valid filenames those which match the regular expansion to ncra In this case ncra never knows that a regular expression was used Chapter 3 Shared Features 31 because the shell intercepts and expands and matches the regular expression before ncra is actually invoked The third method is uses globbing with a different regular expression that is less safe it will also match unwanted files such as 81 nc and 8Z nc if present The fourth method uses the p input path argument to specify the directory where all the input files reside NCO prepends input path e g data username mo
502. ven range of longitude is say 25 335 and we have a point at 20 degrees then the endpoints of the range are used for the interpolation This is what wrapping means For wrapping to occur Y must be longitude and must be in the range 0 360 or 180 180 There are no restrictions on the longitude X values though typically these are in the range 90 90 This ncap2 script illustrates both wrapping and extrapolation of end points defdim lat_in 6 defdim lon_in 5 Coordinate input vars lat in 1at in 1 80 40 0 30 60 0 85 0 lon_in lon_in 30 110 190 270 350 0 Chapter 4 Reference Manual 161 T in 1at in 10on in 10 40 50 30 15 12 43 52 31 16 14 46 54 32 17 16 49 56 33 18 18 52 58 34 19 20 55 60 35 20 0 defdim lat_out 4 defdim lon_out 3 Coordinate variables lat out 1at out 1 90 0 70 88 0 lon out lon_out 0 190 355 0 T_out lat_out lon_out 0 0 T out bilinear interp wrap T in T out lat out lon out lat in lon in print T_out 13 4375 49 5 14 09375 16 25 54 16 625 19 25 58 8 19 325 20 15 60 24 20 135 4 1 19 GSL special functions As of version 3 9 6 released January 2009 NCO can link to the GNU Scientific Li brary GSL ncap2 can access most GSL special functions including Airy Bessel error gamma beta hypergeometric and Legendre functions and elliptical integrals GSL must be version 1 4 or later To list the GSL functions
503. vert to NC FLOAT a single precision 4 byte floating point number double convert to NC DOUBLE a double precision 8 byte floating point number netCDF4 Types ubyte convert to NC UBYTE an unsigned 1 byte integer ushort convert to NC USHORT an unsigned 2 byte integer uint convert to NC_UINT an unsigned 4 byte integer int64 convert to NC INT64 a signed 8 byte integer uint64 convert to NC_UINT64 an unsigned 8 byte integer Intrinsic Mathematical Methods The list of mathematical methods is system dependant For the full list see Section 4 1 25 Intrinsic mathematical methods page 178 All the mathematical methods take a single argument except atan2 and pow which take two If the operand type is less than float then the result will be of type float Argu ments of type double yield results of type double Like the other methods you are free to use the mathematical methods as functions ni pow 2 3 0f ni type float n2 atan2 2 3 0 n2 type double n3 1 three dmn var dbl cos pow 2 tan three dmn var db1 2 n3 type double Chapter 4 Reference Manual 149 4 1 12 RAM variables Unlike regular variables RAM variables are never written to disk Hence using RAM vari ables in place of regular variables especially within loops significantly increases execution speed Variables that are frequently accessed within for or where clauses provide the great est opportunities for optimization To declare and de
504. vg lat avg y x uwnd msk 0 0 uvwnd where lat gt 35 6 amp amp lat lt 37 0 amp amp lon gt 100 5 amp amp lon lt 99 0 uwnd_msk uwnd elsewhere uwnd_msk uwnd _FillValue Average only over horizontal dimensions x and y preserve level and time uwnd avg uwnd msk avg y x Stripped of comments and formatting this example is a three statement script executed by a one line command NCO needs only this meagre input to unpack and copy the input data and attributes compute the statistics and then define and write the output file Unless the comments pointed out that wind variable uwnd was four dimensional and the latitude longitude grid variables were both two dimensional there would be no way to tell This shows how NCO hides from the user the complexity of analyzing multi dimensional SLD We plan to extend such SLD features to more operators soon 4 1 14 Loops ncap2 supplies for loops and while loops They are completely unoptimized so use them only with RAM variables unless you want thrash your disk to death To break out of a loop use the break command To iterate to the next cycle use the continue command Set elements in variable double temp time lat If element 0 set to 0 if element gt 100 set to 100 sz idx time size sz jdx 1at size for idx 0 idx sz idx idx for jdx 0 jdx sz jdx jdxt if temp idx jdx gt 100 temp idx jdx 100 0 else if temp idx jdx lt
505. voked this way Users should specify fout using either not both method If fl out is specified twice once with the switch and once as the last positional argument then the positional argu ment takes precedence 3 7 Accessing Remote Files Availability All operators Short options p 1 Long options pth path 1c1 local All NCO operators can retrieve files from remote sites as well as from the local file system A remote site can be an anonymous FTP server a machine on which the user has rcp scp or sftp privileges NCAR s Mass Storage System MSS or an OPeNDAP server Examples of each are given below following a brief description of the particular access protocol To access a file via an anonymous FTP server simply supply the remote file s URL Anonymous FTP usually requires no further credentials e g no netrc file is necessary FTP is an intrinsically insecure protocol because it transfers passwords in plain text format Users should access sites using anonymous FTP or better yet secure FTP SFTP see below when possible Some FTP servers require a login password combination for a valid user account NCO allows transactions that require additional credentials so long as the required information is stored in the netrc file Usually this information is the remote machine name login and password in plain text separated by those very keywords e g machine dust ess uci edu
506. w gw d 1at 30 0 40 0 d lon 80 0 100 0 a lat lon drc pfx _ tms nc drc pfx _ tms _tbt nc Aggregate 2 regions together ncecat 0 u rgn drc pfx _ tms _ 7 nc drc pfx _ tms _rgn2 nc Change dimensions order ncpdq 0 a time rgn drc pfx _ tms _rgn2 nc drc pfx _ tms _rgn2 nc Remove temporary files optional rm drc pfx _ tms _ nc Annual average ncra 0 mro d time 12 12 drc pfx _ tms _rgn2 nc drc pfx _ tms _rgn2_ann nc Anomaly Long term average ncwa 0 a time drc pfx _ tms _rgn2_ann nc drc pfx _ tms _rgn2_clm nc Subtract ncbo 0 op_typ drc pfx _ tms _rgn2_ann nc drc pfx _ tms _rgn2_clm nc drc pfx _ tms _rgn2_anm nc Standard Deviation inter annual variability RMS of the above anomaly ncra 0 y rmssdn drc pfx _ tms _rgn2_anm nc drc pfx _ tms _rgn2_stddev nc Chapter 7 CMIP5 Example 263 7 4 Monthly Cycle Monthly Snow Cover Anomaly 96 1 10 3 0 E E 10 4 E J S GISEAR CC MU e E 20 GESMI CAMS E ew od C OSMA l 2 gt bcc cemi 1 F d L Jan Feb Mar Apr May June July Aug Sept Oct Nov Dec Figure 7 3 Monthly Cycle This script illustrates how to calculate the monthly anomaly from the annual average see Figure 7 3 In order to keep only the monthly cycle we will subtract the annual average of each year from
507. want to use loop if you need the average of each month for moy in 1 12 do Loop over months mm printf 402d moy Change to 2 digit format ncra 0 snd mm nc out mm nc done 6 4 Multiple files with multiple time points Similar as the last one it s more about shell scripts Suppose you have daily data with one month of them in one data file The monthly average is simply to apply ncra on the specific data file And for seasonal averages you can specify the three months by shell scripts Chapter 7 CMIP5 Example 247 7 CMIP5 Example The fifth phase of the Coupled Model Intercomparison Project CMIP5 http cmip pcemdi 11nl gov cmip5 index html submenuheader 0 provides a multi model framework for comparing the mechanisms and responses of climate models from around the world However it is a tremendous workload to retrieve a single climate statistic from all these models each of which includes several ensemble members Not only that it is too often a tedious process that impedes new research and hypothesis testing Our NASA ACCESS 2011 project simplified and accelerated this process Traditional geoscience data analysis requires users to work with numerous flat data in one level or namespace files In that paradigm instruments or models produce and then repositories archive and distribute and then researchers request and analyze collections of flat files NCO works well with that paradigm yet it also em
508. with ncrename also built with all versions of the netCDF4 library until 4 3 1 rc5 released in December 2013 This bug caused renaming either a dimension or its associated coordinate variable not both which would fail as above in a netCDF4 file to inadvertently rename both Demonstate bug in netCDF4 HDF5 library prior to netCDF 4 3 1 rc5 ncks O h m M 4 v lat T42 nco data in nc foo nc ncrename 0 v lat T42 1at foo nc foo2 nc Also renames dimension ncrename 0 d lat T42 1at foo nc foo2 nc Also renames variable To avoid this faulty behavior either build NCO with netCDF version 4 3 1 rc5 or later or convert the file to netCDF3 first then rename as intended then convert back Unforunately while this bug and the related coordinate renaming bug were fixed in 4 3 1 rc5 released in December 2013 a new and related bug was discovered in October 2014 Another netCDF bug that causes unintended side effects with ncrename affects at least versions 4 3 1 4 3 2 and all snapshots of the netCDF4 library until January 2015 This bug fixed in 4 3 3 in February 2015 corrupts values or renamed netCDF4 coordinate variables i e variables with underlying dimensions of the same name and other non coordinate variables that include an underlying dimension that was renamed In other words renaming coordinate variables and dimensions succeeds yet it corrupts the values contained by the Chapter 4 Reference Manual 233 affecte
509. wledge the netCDF library calls for renaming always work well on netCDF3 files so one workaround to some netCDF4 issues is convert to netCDF3 rename then convert back To understand the renaming limitations associated with particular netCDF versions read the ncrename documentation below in its entirety y Although ncrename supports full pathnames for both old_name and new_name this is really window dressing The full path to new_name must be identical to the full path to old_name in all classes of objects attributes variables dimensions or groups In other words ncrename can change only the local names of objects it cannot change the location of the object in the group hierarchy within the file Hence using a full path in new_name is redundant The object name is the terminal path component of new_name and this object must already exist in the group specified by the old_name path ncrename is an exception to the normal NCO rule that the user will be interactively prompted before an existing file is changed and that a temporary copy of an output file is constructed during the operation If only input file is specified then ncrename changes object names in the input file in place without prompting and without creating a temporary copy of input file This is because the renaming operation is considered reversible if the user makes a mistake The new_name can easily be changed back to old_name by using ncrename one more time
510. y weighted standard deviation problem is ncap2 s prs sfc sdn prs sfc gw prs sfc gw avg 1at 10n rmssdn 1at 10n in nc out nc Be sure to multiply the variable by the weight prior to computing the the anomalies and the standard deviation The procedure to compute the standard deviation of a time series across multiple files involves one extra step since all the input must first be collected into one file Chapter 3 Shared Features 113 ncrcat 0 v tpt in nc in nc fool nc ncwa 0 a time foo1 nc foo2 nc ncbo 0 v tpt fool nc foo2 nc foo3 nc ncra 0 y rmssdn foo3 nc out nc The first step assembles all the data into a single file Though this may consume a lot of temporary disk space it is more or less required by the ncbo operation in the third step 3 36 Type Conversion Availability automatic type conversion ncap2 ncbo nces ncflint ncra ncwa Short options None it s automatic Availability manual type conversion nces ncra ncwa Short options None Long options dbl f1t rth_dbl rth_f1t Type conversion refers to the casting or coercion of one fundamental or atomic data type to another e g converting NC SHORT two bytes to NC DOUBLE eight bytes Type conversion always promotes or demotes the range and or precision of the values a variable can hold Type conversion is automatic when the language carries out this promotion according to an internal set of rules w
511. ye bale a eS 76 SOT wae von ted tate tee asad atl tet m oa abate eather sates 152 297 SOUTE Code a2 so ste zer ele ee 7 source fille anos whales ueni bare bodes das 76 spatial distribution 0 0 e eee eee 253 special attributes 0 0 eee eee ee eee 205 special characters 0 cece eee eee eee eee 186 speed 11 18 24 26 27 85 183 231 SAA cirisensscb pedcs y s ERU hurd tase 107 SdravE iE elLi elf EE RADER NE 146 A aud iacet etel ede Re nls 107 178 square root function 000 e eee 178 DODD RAs cerdo biete dtu Ede stets 243 short dass aca heath 148 SS bim idu is 10 37 standard deviation 107 111 256 standard input 22 195 198 225 228 standard name e lees 67 75 stat system call 00000 103 statement wt ne is 131 static Mk oe ee ete eA 11 Stdinw ncicecsaeintes 22 123 195 198 225 228 steradian iia da 76 Steve Emmerson eee eee eee eee 241 strides iiid e 59 60 64 80 225 226 228 229 SUTIN GS ida ie wl pir B tex neonate she 186 Stu Muller isc oot GRIPE bb testes 242 Subirats DE 35 s b cycle tia gel Merge ESL 62 subcyele uirnnieebiUrRRems Da imet 62 subsetting 0 43 47 126 213 215 S btr ct c sse ee leiRichli esq 189 subtracting datai er igatsed atietan 189 subtractlioM citar o st 178 189 Summary ites hc eee WEISS e tek ae E
512. you are trying to do and include the complete commands run with D 5 error messages and version of NCO with r Post your help request to the NCO Help forum http s f net p nco discussion 9830 16 NCO 4 5 3 alpha03 User Guide If you think you used the right command when NCO misbehaves then you might have found a bug Incorrect numerical answers are the highest priority We usually fix those within one or two days Core dumps and sementation violations receive lower priority They are always fixed eventually How do you simplify a problem that reveal a bug Cut out extraneous variables di mensions and metadata from the offending files and re run the command until it no longer breaks Then back up one step and report the problem Usually the file s will be very small i e one variable with one or two small dimensions ought to suffice Run the operator with r and then run the command with D 5 to increase the verbosity of the debug ging output It is very important that your report contain the exact error messages and compile time environment Include a copy of your sample input file or place one on a pub licly accessible location of the file s If you are sure it is a bug post the full report to the NCO Project buglist http sf net p nco bugs Otherwise post all the information to NCO Help forum http sf net p nco discussion 9830 Build failures count as bugs Our limited machine access me
513. ys Hence results may be ambiguous to NcML parsers If problems arise use xml_spr_chr to specify a multi character separator that does not appear in the string array and that does not include an NcML formatting characters e g commas angles quotes 4 7 2 Filters for ncks We encourage the use of standard UNIX pipes and filters to narrow the verbose output of ncks into more precise targets For example to obtain an uncluttered listing of the variables in a file try ncks m in nc grep E type cut f 1 d sed s sort A Bash user could alias the previous filter to the shell command nclist as shown below More complex examples could involve command line arguments For example a user may frequently be interested in obtaining the value of an attribute e g for textual file examination or for passing to another shell command Say the attribute is purpose the variable is z and the file is in nc In this example ncks m v z is too verbose so a robust grep and cut filter is desirable such as ncks M m in nc grep E i z attribute 0 9 purpose cut f 11 d sor The filters are clearly too complex to remember on the fly so the entire procedure could be implemented as a shell command or function called say ncattget function ncattget ncks M m 3 grep E i 12 attribute 0 9 1 cut 212 NCO 4 5 3 alpha03 User Guide The shell ncattget is invoked with three arugments tha

NCO User Guide

Contents

Download Pdf Manuals

Related Search

Related Contents