Home

CDF User's Guide (V3.5.0) - CDAWeb

1. Usage VMS CDFCONVERT SKELETON lt skt cdf path gt NO LOG NO PERCENT REPORT lt types gt CACHE lt sizes gt NO PAGE NO STATISTICS ABOUT lt src cdf spec gt ZMODE lt mode gt NO NEG2POSFPO lt dst cdf spec gt SINGLE MULTI ROW COLUMN NO DELETE ENCODING lt encoding gt HOST NETWORK COMPRESSION lt types gt SPARSENESS lt types gt BACKAWRD CHECKSUM lt mode gt EPOCH2TT2000 TT20002EPOCH TT20002DEPOCH UNIX including Mac OS X cdfconvert skeleton lt skt cdf path gt no log no percent report lt types gt cache lt sizes gt no page no statistics about lt src cdf spec gt zmode lt mode gt no neg2posfp0 lt dst cdf spec gt single multi row column no delete encoding lt encoding gt host network compression lt types gt sparseness lt types gt backward checksum lt mode gt epoch2tt2000 tt20002epoch tt20002depoch Parameter s lt src cdf spec gt 77 The source CDF s This can be either a single CDF file name or a directory wildcard path in which case all CDFs that match the specification will be converted Wildcards are allowed in the CDF name but not in the directory path In either case do not specify an extension lt dst cdf spec gt The destination of the converted CDF s This may be a single CDF file
2. UNIX ole cdfedit samples cdfedit zmode 2 noformat cache 10d 100s 200c giss wetl cdfedit browse prompt report errors ole ol 3 2 4 Interaction with CDFedit Interaction with CDFedit is through a series of menus and windows Extensive online help is provided and will not be repeated here The online help does refer to the sections of a window by name Figure 3 1 illustrates the various sections of the possible types of windows Item Window Prompt Window Edit Window E Label a 1 i Label 7 E PromptField il EditSection TrailerSection TrailerSection TrailerSection ItemSection Figure 3 1 Window Sections CDFedit 48 Tt is our intention that the use of CDFedit be as intuitive as possible You may not even need the online help We re sure you ll let us know 69 Item Windows are used when a choice is to be made from a list of one or more items e g functions to perform CDFs to edit variable names etc In some cases the entire list of items may not fit on the screen at once When this occurs the ItemSection may be scrolled to display hidden items Some ItemWindows have a percentage indicator at the bottom right portion of the ItemSection The percentage indicator shows which part of the ItemSection is being displayed PromptWindows are used when a textual response is reguired e g a CDF specification a new attribute name a variable value etc If the text is too long to fit i
3. UNIX including Mac OS X cdfedit no browse zmode lt mode gt no format no prompt no neg2posfp0 report lt types gt cache lt sizes gt nolstatistics no gwithentries no vwithentries about lt cdf spec gt Parameter s lt cdf spec gt The specification of the CDF s to edit Do not specify an extension This may be either a single CDF file name or a directory wildcard path Wildcards are allowed in the CDF name but not in the directory path If the prompt qualifier is used this will appear as the initial specification at the prompt If this parameter is omitted the prompt qualifier must be specified and the initial specification at the prompt will be the default current directory Qualifier s NO BROWSE VMS no browse UNIX Specifies whether or not a browsing mode is desired In browsing mode the creation modification or deletion of a CDF is not allowed ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies which zMode should be used The zMode may be one of the following 0 Indicates that zMode should be disabled Running CDFedit in a browse only mode provides the same functionality as CDFbrowse once did 67 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of NOVARY false are removed NOJFORMAT
4. CDFCONVERT CDFSSMPL TEMPLATEO TEMPLATEOX CDFCONVERT LOG REPORT ERRORS CDFSSMPL USER DISK USER CDF CDFCONVERT CAC SST BLENDED CAC SST BLENDEDX SINGLE NETWORK CDFCONVERT SKELETON CDF SMPL TEMPLATE3 CAC SST BLENDED USER CDF CDFCONVERT SOURCE DESTINATION BACKWARD CDFCONVERT FILEIN FILEOUT CHECKSUM MD5 UN Ur UY A A Ur UNIX cdfconvert samples template0 template0x cdfconvert log report errors samples disk4 user cdf cdfconvert cac sst blended cac sst 1 single network cdfconvert skeleton template3 cdf cac sst user cdf e cdfconvert source destination backward cdfconvert fileIn fileOut checksum md5 oe VMS UNIX Command line help is displayed when CDFconvert is executed without any arguments 3 4 3 Output from the CDF convert Program As CDFconvert executes the name of each CDF being converted is displayed If message logging is enabled the progress of each conversion is also displayed 3 5 CDF compare 3 5 1 Introduction The CDFcompare program displays the differences between two CDFs More than one pair of CDFs can be compared This program would be used to verify changes made to a CDF comparing it with the saved original or to verify the conversions performed by CDFconvert see Section 3 4 3 5 2 Executing the CDFcompare Program Usage VMS
5. CDFCOMPARE NO LOG NO ATTR NO VAR NO NUMBER NO NEG2POSFP0 ZMODES lt model gt lt mode2 gt REPORT lt types gt CACHE lt sizes gt NO PAG NO STATISTICS NO PERCENT NO VALUE TOLERANCE lt F tolerancel gt lt D tolerancel gt ABOUT lt cdf spec 1 gt lt cdf spec 2 gt NOJETC NO LOCATION E UNIX including Mac OS X 82 o cdfcompare nollog no Jattr no var no number no etc Lal no neg2posfp0 zmodes lt mode1 gt lt mode2 gt no location report lt types gt cache lt sizes gt no page no statistics no percent no value tolerance lt f tolerancel gt lt d tolerance2 gt about lt cdf spec 1 gt lt cdf spec 2 gt Parameter s lt cdf spec 1 gt lt cdf spec 2 gt The specifications of the CDFs to be compared Do not enter extensions These can be either a file name specifying a single CDF or a directory wildcard path specifying more than one CDF Wildcards are allowed in the CDF name but not in the directory path If two directory wildcard paths are specified all of the CDFs with matching names will be compared If a CDF file name and a directory wildcard path are specified the CDF specified will be compared with the CDF in the directory wildcard path having the same name If two CDF file names are specified the CDFs are compared
6. lt value gt lt rec num gt lt indices gt lt value gt lt rec num gt lt indices gt lt value gt Each field is defined as follows Record Dimension Variance Variances lt sizes gt lt rec vary gt lt dim varys gt Note the lt var name gt lt var data type gt lt n elems gt lt dims gt lt sizes gt The name of the zVariable The name must be delimited with a character not appearing in the name itself e g EPOCH or Temperature The delimiting characters are not part of the zVariable name in the CDF The data type for the zVariable The data type must be one of the following CDF_BYTE CDF INT1 CDF_UINT1 CDF_INT2 CDF_UINT2 CDF_INT4 CDF_UINTA CDF_INT8 CDF_REAL4 CDF_FLOAT CDF_REAL8 CDF_DOUBLE CDF_EPOCH CDF_EPOCH16 CDF_TIME_TT2000 CDF_CHAR or CDF_UCHAR The number of elements of the data type For character data types CDF_CHAR and CDF UCHAR this is the number of characters in each string For non character data types this value must be one 1 The number of dimensions for the zVariable The dimension sizes one value per dimension If the zVariable has zero 0 dimensions this field would be left blank 118 lt rec vary gt lt dim varys gt lt attr name gt lt entry data type gt lt entry value gt lt rec num gt lt indices gt lt value gt The record variance of the zVariable This must be either T the values vary from record
7. backward about lt skeleton path gt Parameter s lt skeleton path gt The file name of the skeleton table from which a skeleton CDF will be created Do not specify an extension Qualifier s CDF lt cdf path gt VMS cdf lt cdf path gt UNIX The file name of the CDF that will be created overriding the file name in the skeleton table If this qualifier is not specified the CDF file name in the skeleton table is used Do not specify an extension in the file name NOJLOG VMS no log UNIX Specifies whether or not messages are displayed as the program executes NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS NO DELETE VMS no delete UNIX 96 Specifies whether or not the CDF will be deleted first if it already exists essentially overwriting it NO FILLVAL VMS no fillval UNIX Specifies whether or not entries of the FILLVAL vAttribute are used to set the pad values for the corresponding variables If this qualifier is specified the FILLVAL vAttribute must exist and only those variables with an entry for the FILLVAL vAttribute will be affected CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The
8. 37 open CDFs 28 variable name length 37 little endian 34 majority variable 48 MONOTON attribute 66 70 87 multi file format 32 multiple variable access 53 network encoding 34 pad values variable 54 performance considerations decoding 32 37 encoding 35 format 32 majority 49 gualifier special 66 read only mode 26 reserve percentage compression 48 Run Length encoding compression 60 SCALEMAX attribute 66 87 SCALEMIN attribute 66 87 scope attribute 56 scratch files 28 sequential access variable 49 52 single file format 31 Skeleton CDF 95 skeleton table 91 95 creating 91 98 example 120 file extension 98 SkeletonCDF 20 22 95 executing 96 SkeletonTable 20 91 109 executing 92 output 95 sparseness arrays 14 47 records 14 44 Standard Interface 25 status codes 127 categories 127 trailing blanks attribute name 56 CDF file name 31 variable name 39 TT2000 59 syntax 60 VALIDMAX attribute 66 70 87 VALIDMIN attribute 66 70 86 variables 8 15 37 accessing 38 hyper read write 50 example 50 reading 49 writing 49 multiple variable 53 sequential values 52 example 52 single values 49 arrays 40 closing 38 149 compression 14 47 algorithms 60 reserve percentage 48 data specification 40 changing 40 data type 40 number of elements 40 deleting 39 dimensionality 39 majority 48 changing 49 example 49 naming 39 case sensi
9. ALPHAVMSg DECODING DEC Alpha running OpenVMS data representation Double precision floating point values will be in Digital s G FLOAT representation ALPHAVMSi_DECODING DEC Alpha running OpenVMS data representation Double precision floating point values will be in IEEE representation ALPHAOSF1_DECODING DEC Alpha running OSF 1 data representation SUN_DECODING Sun data representation SGi_DECODING Silicon Graphics Iris and Power Series data representation DECSTATION_DECODING DECstation data representation IBMRS_ DECODING IBM RS6000 series data representation HP_DECODING HP 9000 series data representation PC_DECODING PC data representation NeXT_DECODING NeXT data representation MAC_DECODING Macintosh data representation 36 Performance The best performance when reading a CDF will occur when the CDF s decoding is the same as the CDF s encoding since no conversion will have to be performed by the CDF library Since host decoding is the only directly usable decoding by an application CDFs with the host s encoding will provide the best performance Care should be taken when selecting the encoding for a CDF 2 2 10 Compression A compression may be specified for individual variables and or a single file CDF that is performed when the CDF is closed and written to disk When compression is specified for a CDF the CDF library maintains an uncompressed version of the dotCDF file in a scratch file When the CDF is closed the uncom
10. CDFcreate CDFopen etc that are functionally eguivalent to the ones that are built into IDL The CDF office had to supply its own routines hereafter referred to as the CDF IDL interface for manipulating CDF files in IDL because IDL originally didn t include support for CDF Research Systems Inc the developers of IDL later implemented an interface to CDF as part of the IDL product It differs from the interface provided with the CDF IDL interface distribution in that it is intended more for the non programmer and is functionally similar to other interfaces IDL provide The CDF IDL interface was always included as part of the CDF standard distribution package for Unix albeit they are redundant with IDL s built in CDF routines up until CDF 2 7 2 to support legacy applications that utilities the CDF IDL interface Those users who must run applications that are based on the old CDF IDL interface SHOULD NOT upgrade to CDF 3 0 or a later version For those IDL applications that utilize the CDF IDL interface it s highly recommended to port these applications to use the IDL s built in CDF interface The migration is relatively easy since IDL s built in CDF functions are very similar to the ones in the CDF IDL interface B 2 CDF Version 3 x and IDL The advent of CDF 3 0 introduced among many other things an ability to create files bigger than 2G bytes and a new data type CDF EPOCH16 to address the limitation of the highest timestamp res
11. The directory of software consisting of the CDF library include files and toolkit The software library that is used to access a CDF A set of utility programs which ease the creation modification and verification of CDFs A CDF toolkit program that allows the display and modification of a CDF s contents A CDF toolkit program that allows the possibly filtered contents of a CDF to be exported to the terminal screen a text file or another CDF A CDF toolkit program that generates a report containing various statistics about a CDF s variables A CDF toolkit program that reports any differences between two CDFs A CDF toolkit program that allows various overall properties of a CDF to be changed in a newly created CDF A CDF toolkit program that displays the version of the CDF distribution being used many of the configurable parameters and the default CDF toolkit qualifiers options A completion status code indicating unqualified success An include file used in C applications An include file used in Fortran applications An include file used in Digital Visual Fortran applications An include file used in Digital Visual Fortran applications An include file used in Digital Visual Fortran applications An include file used in Digital Visual Fortran applications An include file used in Microsoft Fortran applications The variable majority where the first index of a multidimensional array of values increments the f
12. appropriate Wildcard characters may be used in the CDF name but not the directory path portion of a specification The wildcard characters supported are similar to those available on the operating system being used UNIX Ifa CDF specification is to contain a wildcard character the entire specification must be enclosed in single quote marks e g disk3 sst 7 On VMS OpenVMS systems qualifiers begin with a slash On UNIX qualifiers begin with a hyphen NOTE You can override the default notation by specifying a slash or hyphen as the first parameter qualifier immediately after the program name When this is done you may have to adjust the syntax used as follows a When the slash notation is used on UNIX systems character string will be necessary in the file names e g specify disk1 CDFs rather than distl CDFs Also double quote marks are required around options enclosed in parenthesis b When the slash notation is used on MS DOS systems double quote marks may be needed around entire qualifier option combinations 8 On UNIX systems all parameters qualifiers entered at the command line are case sensitive On VMS OpenVMS and MS DOS systems parameters qualifiers are not case sensitive Note that variable names are always case sensitive regardless of the operating system being used 9 If an option contains blanks it will generally be necessary to enclose the entire option in double quote marks 10 On som
13. cdfexport simple batch text text flux out flux 1996 cdfexport batch cdf cdf mycdf1 include varl var2 test cdfexport batch cdf cdf mycdf2 exclude varl var2 test cdfexport batch text text mytext exclude varsfile home cdf varslist txt test cdfexport batch cdf cdf mycdf epochrange 01 10 2005 00 00 00 000 01 10 2005 10 00 00 000 test cdfexport batch text text myoutput controlsettings settings export set test 3 3 4 Interaction with CDFexport Interaction with CDFexport is through a 4 part SelectionWindow an ActionMenu an OptionMenu numerous prompt windows and several screen listing windows Detailed online help is available for each window so only a brief description of each will be given here After selecting a CDF from which to export part 1 of the SelectionWindow will be loaded with a line for the lt Record gt item the lt Indices gt item and each variable The lt Record gt item allows the record number to be included in a screen file listing and or filtering on the record number for any type of output The lt Indices gt item allows the dimension indices to be included in a screen file listing and or filtering on the dimension indices for any type of output Each variable line allows that variable to be included and or filtered when generating any type of output The KeyDefinitions window displays the available functions and their corresponding keys for a given window prompt The MessageBuffer display
14. e w and i CACHE lt sizes gt VMS cache lt sizes gt UNIX 84 Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library TOLERANCE lt F TOLERANCE1 D TOLERANCE2 gt VMS tolerance lt f tolerancel1 d tolerance2 gt UNIX Specifies the tolerance s that is used to check the equality between two single double precision floating point values The default option is no tolerance It means that two values are considered unegual if their data representations in common encoding are different If a tolerance s is provided it is used against the difference between the two unegual values If their difference is wit
15. gt ana Il Variable Name Coefficients Attribute Name FIELDNAM FORMAT 0 0254 14 2338 9 9444 1 2 3 Variable Name Data Type CDF_CHAR CDF INT4 CDF INT4 CDF INT4 CDF INT4 CDF CHAR CDF CHAR Data Type CDF INT4 Data Type CDF CHAR CDF INT4 CDF INT4 CDF CHAR CDF CHAR Data Type CDF REAL4 Data Type CDF CHAR CDF CHAR Data Type Value Temperature oy 0 50 0 10 Deg C y I2 mo Number Record Dimension Elements Dims Sizes Variance Variances 1 0 T Value Correction bias for temperature 5 5 deg C gt I2 y Number Record Dimension Elements Dims Sizes Variance Variances 1 1 3 F T Value Temperature model coefficients F9 1 Mina Number Record Dimension Elements Dims Sizes Variance Variances 123 TMP model end Attribute Name FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT CDF_REAL4 Data Type CDF_CHAR CDF REAL4 CDF REAL4 CDF REAL4 CDF REAL4 CDF CHAR CDF CHAR 1 2 360 180 Temperature model 20 0 50 0 0 0 30 0 deg C i F9 6 M eke AS i SS s AS 124 Appendix B IDL Support B 1 CDF IDL Interface and Legacy Applications In addition to the built in CDF functions e g CDF CREATE CDF OPEN etc in IDL the CDF distribution package prior to CDF 3 0 used to include its own set of IDL functions procedures e g
16. operation of the CDFlib function Internal Interface Hyper writes for zVariables must be performed using the lt PUT zZ VAR HYPERDATA gt operation of CDFlib Hyper reads for rVariables are performed using the CDFvarHyperGet function Standard Interface or the lt GET rVAR HYPERDATA gt operation of CDFlib Hyper reads for zVariables must be performed using the lt GET Z VAR HYPERDATA gt operation of CDFlib In a Fortran application hyper writes for rVariables are performed using the CDF var hyper put subroutine Standard Interface or the lt PUT rVAR HYPERDATA gt operation of the CDF lib function Internal Interface Hyper writes for zVariables must be performed using the lt PUT zZ VAR HYPERDATA gt operation of CDF lib Hyper reads for 51 rVariables are performed using the CDF var hyper get subroutine Standard Interface or the lt GET rVAR HYPERDATA gt operation of CDF lib Hyper reads for zVariables must be performed using the lt GET ZVAR HYPERDATA gt operation of CDF lib 2 3 18 Sequential Access Sequential access provides a way to sequentially read write the values physically stored for a variable To use sequential access a starting value must first be selected by specifying a record number and dimension indices This selects the current sequential value A sequential read will return the value at the current sequential value and then automatically increment the current sequential value to the next value
17. 165 40 20 0 2 0000 165 30 21 7 3 0000 150 40 19 2 4 0000 150 30 20 7 5 0100 165 40 18 2 6 0100 165 30 19 3 7 0100 150 40 22 0 8 0100 150 30 19 2 9 0200 165 40 19 9 10 0200 165 30 19 3 11 0200 150 40 19 6 12 0200 150 30 19 0 Although rVariables are described here first the trend among CDF users is toward CDFs containing only zVariables since zVariables can do everything rVariables can do and more zVariables are described in the next section 15 93 2300 165 40 21 0 94 2300 165 30 19 5 95 2300 150 40 18 4 96 2300 150 30 22 0 Table 1 1 Example Data Set Flat Representation 0 Dimensional Adding another independent rVariable for instance Pressure poses no difficulty for the example Temperature would then be dependent on a specific Longitude Latitude and Pressure a 3 dimensional array structure In this 3 dimensional example Longitude Latitude and Pressure define the number of dimensions for the rVariables in the CDF where the size of each dimension is determined by the number of discrete values contained in each of those rVariables Additional dependent rVariables would be stored in the same way as Temperature Although conceptually there is a 2 dimensional array structure for each rVariable in each record of the CDF this would not be an efficient way to store the data For instance the time for each record need only be stored once as opposed to being stored four ti
18. 2 0 2 1 Example s CDFDUMP my data cdf UNIX and Windows cdfdump my data cdf Help is displayed when CDFdump is executed without any arguments 104 3 13 CDFirsdump 3 13 1 Introduction The CDFirsdump program displays the statistics of CDF Internal Records IRs within a file to a screen default or text file This program can also dump in hex form each internal record CDFirsdump can be used to show how a CDF is constructed whether it s more compacted or very fragmented Compacted CDF files provide much better performance as less IRs are visited to acquire data If r ZVDR r zVaribale Descriptor Record counts are quite fewer than VXR Variable Index Record or and VVR Variabale Value Record a CDF file has become more fragmented CDFconvert can use be to reconstruct CDF files to make them compact See CDF Internal Format Description for information about various internal records in a CDF 3 13 2 Executing the CDFirsdump Program Usage VMS CDFirsDUMP OUTPUT lt file path gt BRIEF FULL NO PAGE NO SUMMARY ABOUT lt cdf path gt UNIX including Mac OS X Windows cdfirsdump output lt file path gt brief EALLI no page no summary about lt cdf path gt Parameter s lt cdf path gt The pathname of the CDF file to be dumped Qualifier s OUTPUT lt file path gt output lt file path gt Unix Windows By default the contents of file is displa
19. 58 Character data types are unigue for variables in that they are the only data types for which more than one element per value is allowed Each variable value consists of a character string with the number of elements being the number of characters More than one element is allowed for any of the data types when dealing with attribute entries Currently the character set supported by the CDF is limited to ASCII set of characters Non conforming characters are either rejected or will not be properly handled displayed by the CDF library 2 5 4 EPOCH Data Types CDF_EPOCH 8 byte double precision floating point CDF EPOCH16 two 8 byte double precision floating point The CDF EPOCH and CDF EPOCH16 data types are used to store date and time values referenced from a particular epoch For CDF that epoch is 01 Jan 0000 00 00 00 000 and 01 Jan 0000 00 00 00 000 000 000 000 respectively CDF_EPOCH values are the number of milliseconds since the epoch The standard format used to display a CDF_EPOCH value is dd mmm yyyy hh mm ss ccc where dd is the day of the month 01 31 mmm is the month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov or Dec yyyy is the year 0000 9999 hh is the hour 00 23 mm is the minute 00 59 ss is the second 00 59 and ccc is the millisecond 000 999 CDF_EPOCH16 values are the number of picoseconds since the epoch The standard format used to display a CDF_EPOCH 16 value is dd mmm yyyy hh mm ss ccc m
20. 9 1 3 92 3 END_OF_VAR where r i j is a physically stored value with r being the record number i the first dimension index and j the second dimension index r i and j are physical record numbers and dimension indices The next sequential read after the last physical value would cause a status code indicating the end of the variable to be returned END OF VAR Had the dimension variances been NOVARY VARY the values read would have been 7 1 2 74 3 8 1 1 8 1 2 8 1 3 91 1 91 2 91 3 END OF VAR Note that specifying the virtual value 7 2 2 as the current sequential value caused physical value 7 1 2 to actually be selected because the first dimension variance is NOVARY Sequential access for variables is performed using the lt GET 1 zZVAR SEODATA gt and lt PUT r ZVAR SEODATA gt operations of the Internal Interface 52 2 3 19 Multiple Variable Access Multiple variable access allows an application to read from or write to multiple variables in a single operation Multiple variable access works on either the rVariables or the zVariables of a CDF not a mixture of the two Up to all of the rVariables zVariables may be accessed with a single call to the CDF library For each variable specified in a multiple variable access a full physical record for that variable will be read written A full physical record consists of all of the values exactly as they are physically stored in each variable record the physical va
21. Because the records would be allocated as contiguously as possible within the CDF file the indexing scheme see Section 2 2 7 would reguire fewer entries making the access to that variable more efficient Note that this method is not as efficient as allocating records in those cases where all of the records are going to be written by the application This is because the records would be written twice once with the pad value and then again by the application The number of initial records specified would in most cases be the number of records planned for a variable Note that additional records may be added to a variable at any time For NRV variables the number of initial records must always be specified as one 1 This is because only one physical record will ever actually be written Initial records for a variable may be specified only once Initial records are written to variables using the lt PUT 1 Z VAR INITIALRECS gt operation of the Internal Interface Explicit pad values are specified using the lt PUT zZ VAR PADVALUE gt operation Blocking Factor A variable s blocking factor a number of records affects how its records are allocated in the CDF file s For NRV variables the blocking factor is not applicable because only one physical record will ever exist For variables in a multi file CDF the blocking factor is not used because only those records written by an application will exist in the variable files But for the ot
22. C and VB Application Programming Interfaces APIs are based on the core CDF library s extended Standard Interface which is a near complete set of wrappers around the Internal Interface functions Both APIs support zVariables and zVariables 1 12 How to create a CDF A CDF file can be created either by using the programming interface Standard Interface or Internal Interface or the CDFEdit and SkeletonTable toolkit programs included in the standard CDF distribution package 1 12 1 Sample C Fortran Java Perl and C Programs 21 Sample programs are included as part of the CDF standard distribution package Below describes where the sample programs can be found Unix including Mac OS X Windows C lt cdf_dir gt samples lt cdf_dir gt samples Fortran lt cdf_dir gt samples lt cdf_dir gt samples Java lt cdf_dir gt cdfjava examples lt cdf_dir gt samples Perl lt perl_dir gt lt perl_dir gt C lt csharp dir gt lt csharp dir gt 1 12 2 Creating a CDF with CDFEdit The CDF toolkit program CDFEdit is one of the utility programs included in the CDF standard distribution package and it allows users to create a CDF file without programming CDFEdit allow interactively creating a new empty CDF file reading modifying the data metadata in an existing file 1 12 3 Creating a CDF with SkeletonTable The CDF toolkit program SkeletonCDF is another one of the utility programs included in the CDF sta
23. CDF creating a new CDF with the desired format The default format for a created CDF is single file and it can be changed if needed In a user application the Internal Interface must be used to change the format of a CDF by using the lt PUT CDF FORMAT gt operation of the Internal Interface The format of an existing CDF can be changed only if no variables have been created in the CDF If the SkeletonCDF toolkit program is used to create a CDF the format is specified in the skeleton table see Appendix A Single File CDFs 6The file of a CDF having an extension of cdf is referred to as the dotCDF file 22 Actually the CDF library will check several possible extensions cdf cdf 1 CDF and CDF 1 These extensions are checked because some CD ROM drivers primarily on UNIX machines do peculiar things when making the files e g CDFs on a CD ROM visible 23 Or edf 1 or CDF or CDF 1 31 A single file CDF SINGLE FILE consists of only one file with extension cdf This file is referred to as the dotCDF file The dotCDF file contains the control information for the entire CDF the attribute entry data and all of the variable data An indexing scheme is used to provide efficient access to variable records Indexing Scheme In single file CDFs an indexing scheme is used to keep track of where a variable s records are located within the dotCDF file The order that variable and attribute entry values are written to a single
24. Factor descriptions in Section 2 3 12 It is possible to inquire the indexing statistics for a variable Using the Internal Interface an application may inquire the number of indexing levels in the hierarchy the number of index records and total number of entries for a variable using the lt GET 1 ZVAR nINDEXLEVELS gt lt GET 1 ZVAR nIiNDEXRECORDS gt and lt GET 1 ZVAR nINDEXENTRIES gt operations Malti File CDFs A multi file CDF MULTI FILE consists of one file with extension cdf referred to as the dotCDF file containing control information and attribute entry data and a separate file for each variable defined in the CDF with extensions v0 vl1 for rVariables and z0 z1 for zVariables Each variable file contains the data values for the corresponding variable The control information for each variable is stored in the dotCDF file Performance The most efficient access to CDF variables will usually occur when the CDF has the multi file format The extra overhead involved with the indexing scheme used in single file CDFs is small so the difference may not be significant especially if hyper reads writes are used The drawback to using the multi file format is that more than one file is associated with a CDF which may or may not be a problem for your system management There is a case in which the single file format may be more efficient If a CDF has a large number of variables larger than the number of f
25. NT 95 98 Only RV variable values should be put in the skeleton table All VMS All values radio button UNIX All radio button Macintosh Java UNIX amp Windows NT 95 98 All variable values should be put in the skeleton table lt named gt VMS Selected values radio button UNIX named radio button Macintosh Java UNIX amp Windows NT 95 98 Values of the named variables should be put in the skeleton table VMS UNIX lt values gt is a comma separated list of delimited variable names with the entire list enclosed in double quote marks NOTE Do not use double quote marks to delimit a variable name Ignore NRV data No values are placed in the skeleton table NRVTABLE VMS nrvtable UNIX Put NRV variable data values in the skeleton table VMS UNIX Note that only the values qualifier is actually needed The others are supported for compatibility with previous CDF distributions NOJLOG VMS 93 no log UNIX Specifies whether or not messages are displayed as the program executes ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies the zMode that should be used with the CDF The zMode may be one of the following O Indicates that zMode should be disabled 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of NOVARY false are removed NOJFORMAT V
26. Number of dimensions and dimension sizes for the rVariables 3 gAttribute definitions and gEntry values 4 rVariable and zVariable definitions and vAttribute definitions with rEntry zEntry values 91 5 Data values for all or a subset of the CDF s variables The above information is written in a format that can be understood by the SkeletonCDF program see Section 3 8 SkeletonCDF reads a skeleton table and creates a new CDF called a skeleton CDF 3 7 2 Special Attribute Usage The special attribute FORMAT is used by SkeletonTable depending on the setting of the format qualifier when writing variable values in a skeleton table 3 7 3 Executing the SkeletonTable Program Usage VMS SKELETONTABLE SKELETON lt skeleton path gt NO LOG ZMODE lt mode gt NONRV NRVTABLE VALUES lt values gt NO SCREEN NO NEG2POSFP0 NO FORMAT REPORT lt types gt IN CACHE lt sizes gt NO PAGE NO STATISTICS ABOUT lt cdf path gt UNIX including Mac OS X skeletontable skeleton lt skeleton path gt no log zmode lt mode gt nonrv nrvtable values lt values gt no screen no neg2posfp0 no format report lt types gt cache lt sizes gt no page no statistics E about lt cdf path gt Parameter s lt cdf path gt The file name of the CDF from which t
27. The number of records allocated for a variable can be inquired using the lt GET r ZVAR NUMallocRECS gt operation The maximum record allocated for a variable can be inquired using the lt GET r zZVAR MAXallocREC gt operation The exact records allocated for a variable can be determined using a combination of the lt GET 1 Z VAR ALLOCATEDTO gt and lt GET_ zVAR_ALLOCATEDFROM_ gt operations Initial Records The Internal Interface may be used to specify an initial number of records to be written for a variable The pad value for the variable is written at each record as if the application had done so itself The Internal Interface allows this to be done more conveniently with only one function call Note that the default pad value for the variable s data type will be used unless a pad value is explicitly specified for the variable If a specific pad value is desired for a variable then it must be specified before the number of initial records is specified Also any compression or sparseness for the There is no reason to allocate records for a variable in a multi file CDF 33 The use of allocated records would in most cases be more efficient than specifying initial records 45 variable must be specified before writing the initial records because those properties cannot be changed after records have been written Specifying a number of initial records for a variable would usually be done only for a CDF with the single file format
28. This is the only way to compare two CDFs having different names Qualifier s NO LOG VMS no log UNIX Specifies whether or not messages about the progress of each comparison are displayed NO PERCENT VMS no percent UNIX Specifies whether or not the percentage of a variable s values compared is displayed during the comparison of that variable Message logging must also be enabled NO ATTR VMS no attr UNIX Specifies whether or not attributes and their entries are to be compared NO VAR VMS no var UNIX Specifies whether or not variables are to be compared Note that an rVariable will never be compared with a zVariable NO NUMBER VMS no number UNIX Specifies whether or not numbering differences between attributes with the same names and between variables with the same names are to be displayed 83 NOJETC VMS no ete UNIX Specifies whether or not differences transparent to an application will be displayed These would consist of the version release increment of the creating CDF library format encoding etc ZMODES lt mode1 gt lt mode2 gt VMS zmodes lt model gt lt mode2 gt UNIX Specifies the zModes that should be used with the CDF s being compared Note that different zModes may be used for the two CDF s specifications The zModes may be one of the following 0 Indicates that zMode should be disabled 1 Indicates that zMode 1 should be used The
29. Up until CDF 2 7 2 if you wanted to create or access zVariables and zEntries you had to use the Internal Interface which might be harder to use The limitations of the Original Standard Interface were addressed with the introduction of the Extended Standard Interface in CDF 3 1 The new Extended Standard Interface allows many new operations that were only previously available through the Internal Interface 1 8 2 Internal Interface The Internal Interface consists of one routine CDFlib when called from C and CDF lib when called from Fortran The Internal Interface is used to perform all CDF operations In reality the Standard Interface is implemented via the Internal Interface 1 9 CDF Java Interface The CDF Java Application Programming Interfaces APIs are based on the core CDF library s Internal Interface and they support a near complete set of the Internal Interface functions The Java APIs only support zVariables and treats rVariables as zVariables This is not a problem since zVariable is a superset of rVariable In another words with zVariables you can do everything with rVariables and more but not vice versa 1 10 CDF Perl Interface The CDF Perl Application Programming Interfaces APIs are based on the core CDF library s Standard and Internal Interface and they support a near complete set of the Internal Interface functions The Perl APIs support both rVariables and zVariables 1 11 CDF C and Visual Basic Interface The CDF
30. VMS no format UNIX Specifies whether or not the FORMAT attribute is used when displaying variable values if the FORMAT attribute exists and an entry exists for the variable NO PROMPT VMS no prompt UNIX Specifies whether or not a prompt is issued for the CDF s specification When enabled the prompt will be issued both at program startup and after editing the current CDF s specification at which point a new CDF s specification may be specified If a CDF s specification was entered on the command line that CDF s specification will appear at the prompt Otherwise the current default directory will appear at the prompt NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and i CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt
31. a browse only mode 12 CDFexport has replaced CDFlist and CDFwalk 19 SkeletonCDF Reads a specially formatted text file called a skeleton table and creates a Skeleton CDF A skeleton CDF is complete except for record variant data SkeletonTable Reads a CDF and produces a specially formatted text file called a skeleton table The skeleton table may be modified and then input to SkeletonCDF to create a skeleton CDF CDFinquire Displays the version of your CDF distribution many of the configurable parameters and the default CDF toolkit qualifiers CDFstats Produces a report containing various statistics about the variables in a CDF CDFcompare Reports the differences between two CDFs CDFdir Produces a directory listing of a CDF s files For a multi file CDF the variable files are listed in ascending numerical order CDFDump Dumps the metadata data in a CDF CDFIrsDump Dumps the internal records IRs in a CDF This is a debugging tool to show the internal data structure of a CDF CDFMerge Merges several CDFs into a single one Most suitable for combining time sequenced CDFs CDF Validate Validates CDFs to see if CDFs will pass data checking for certain data fields 1 8 Library Interface Routines The core CDF library supports two programming interfaces the Standard Interface and the Internal Interface Standard Interface is easier to use and requires a much shorter learning curve than the Internal Interface but it s not
32. a multidimensional array of values increments the fastest Record variant variable A variable whose values change from record to record a record variance of VARY 143 rVariable scratch directory scratch files scope seguential access single file single value access skeleton CDF skeleton table SkeletonCDF SkeletonTable sparse arrays sparse records Standard Interface standard variable State status code R variable A CDF object in which data values are stored All rVariables have the same dimensionality The directory in which the CDF library creates scratch files This directory may be specified by a user or an application Temporary files used by the CDF library to minimize core memory usage The intended use for an attribute This may be global scope or variable scope A variable access method in which values are read written in the physical order in which they are stored in the CDF A CDF format Single file CDFs are entirely contained within one file A variable access method in which exactly one value is read written for a variable A CDF consisting of only control metadata and NRV variable values A text file containing the control metadata and traditionally only the NRV variable values of a CDF RV variable values may now also be included in a skeleton table A skeleton table is read by the SkeletonCDF toolkit program which then creates the corresponding skeleton CDF or com
33. aia e oe eee os te EA Ts Sia MARA EA ae 14 VAA E O O NN 14 1 4 5 Variable Data Access Options sx sc cccceivicoscecessivvsescstesstaceccacvedaesaceeodcuusacssaacdesssassavscosctuvscastaccsstveutececasesciuvecs 14 1 5 Organizing Your Data ina CDE cocococccinaci ncni n condado on kku paki a ENE AE EE A E ESEE EEEN EEE AET 15 ISto Variables nesunan NN 15 L6 A NN 18 LF EDE Toolbar olaaa 19 1 8 Library Interface Routines eccocccconcinnoneinacicncni n condado ak oih peksu keeks Ananda aire denia A inscoecesessncuaccs lieve kava even 20 1 8 1 Standard Interface sssini riene ni taoi ESEESE EE OE EEE sha ESEE lna cos NEEE AE 20 18 2 Internal AAA NN 21 L9 CDE Java Tintertaces 01511 s pakuti nosi auku NN 21 1 10 CDE Perl Interface nisi ondaa eRe aE E EE EE v ka AE EEE EEE EEEE AEE 21 1 11 CDF Cr and Visual Basic Interface ics ccccciscssccesciuvsccssteccesccssaeecceesechosdevacecuanactabaenssivecoecessssneebccestesndtecscaactcauvees 21 1 12 Howto createa CDE srt A Ai annikesekene ii ii entries 21 1 12 1 Sample C Fortran Java Perl and CH PrograMS ooooonoccnnoconocnconnnonnnconnconnncnn nono nonnnonnn cono nn ron cnn nc ere ennne enne 21 1 12 2 Cr ating a CDE with CDE Edi A oa 22 12 3 Creating a CDE with Skeleton Table occ eee tia 22 A M AT RE 2L MEDEL A SM SST A 25 Zell Internac na e ain A E TTE A iial sd 25 22 CEDE Mods a lalia 26 2 13 LS e on ndo ad 28 2 1 4 gt Scratch Fils ui e iia k va vv ju P tu Eva 28 2 15 NT
34. been reached Error An error occurred while deleting a variable file in a multi file CDF Check that sufficient privilege exist to delete the CDF files Error Named variable already exists cannot create or rename Each variable in a CDF must have a unique name rVariables and zVariables can not share names Note that trailing blanks are ignored by the CDF library when comparing variable names Error Variable name truncated to CDF VAR NAME LEN characters The variable was created but with a truncated name Warning An error occurred while opening variable file Check that sufficient privilege exists to open the variable file Also make sure that the associated variable file exists Error Failed to read variable as requested error from file system Check that the associated file is not corrupted Error Failed to write variable as requested error from file system Check that the associated file is not corrupted Error One or more of the records are virtual never actually written to the CDF Virtual records do not physically exist in the CDF file s but are part of the conceptual view of the data provided by the CDF library Virtual records are described in the Concepts chapter in the CDF User s Guide Informational 135 Appendix D Release Notes D 1 Supported Systems CDF V3 2 is currently supported on the following computers operating systems 1 2 8 9 VAX OpenVMS amp POSIX shell S
35. by IDL 6 2 or earlier or by CDF version 2 7 2 or earlier However IDL 6 3 or later versions can read CDF files by IDL 6 2 or earlier or by CDF version 2 7 2 or earlier The file incompatibility problem is due to the use of 64 bit file offset in CDF 3 0 and later versions to allow for creation of files bigger than 2G bytes Note that IDL 6 3 uses CDF 3 1 If you wish to create and share CDF files with colleagues who access CDF files using IDL 6 2 or earlier or CDF version 2 7 2 or earlier there s an IDL procedure called CDF_SET_CDF27_BACKWARD_FILE_COMPATIBLE that allow users of IDL version 6 3 or later to create a CDF file that can be read by IDL 6 2 or earlier or by CDF version 2 7 2 or earlier This procedure must be called prior to creating a CDF file with CDF_CREATE If a file is created in the CDF 2 7 format the maximum file size is 2G bytes If you can t wait until IDL 6 3 is released and now need to use any of the CDF 3 0 s new features or manipulate CDF files that were created with CDF 3 0 or later in IDL 6 2 or earlier please contact the CDF support office at gsfc cdf support lists nasa gov for a binary copy of the IDL CDF system routines 126 Appendix C Status Codes C 1 Introduction A status code is returned from most CDF functions The cdf h for C and CDF INC for Fortran include files contain the numerical values constants for each of the status codes and for any other constants referred to in the explanations Th
36. ce cc eeceesecesecesseceeeecseeeesceeeeeecseceececeseeesaeceaeecaeeeaaeeeeeeeaeeeeaeeeneees 95 HS Skeleton CDE Roa E 95 A ON 95 3 8 2 Executing the SkeletonCDF Program renteeneonteoeneeeneneeneeeeneenrneenaeeneneennenenaenneneeneeeene enne 96 3 9 3 Creating the Skeleton Table ota acre E E N en Kan anta cy a TUNES 98 JO RNA 98 IDA Introduction a SESE ea oy Sees ADAT eua Lu TNS ke AED E KU E Ta E SE Ea EA 98 3 9 2 Executmgthe CDEmgyuireProgtam sw civ seca Shu eed eE e A th laga US 98 3 9 3 Output from the CDFinquire Program renteeneneonneoeneennenoeneeeneenrneenaeennneenneeenaeeneneenaeeene enne 99 3210 SCD Git siete 14 eine el ick aie eee eevee i a ae ie See ls canta ae 99 JLO A NA 99 3 10 22 Executing the CDE dit Program siit cece MAN i vaevata atea 99 3 10 3 Output from the CDFdir Program anneni iir a E e E REEE EEE E N 100 3 11 EDEMEFSE cca eaves E TEN ie aie acted ie a ae a eects EE he pa Mis E E 100 32114 TAHOdUGUON NN 100 3 11 27 Executing the CDE merge Program secs a E 100 3 12 SC DE dumps ore nie ls ad Hee iol cadet eke Mee a ee ee tee ee nia aes 103 3 12 1 IO UC a eee tie aii ae NS ee oe es Mee ae scene ane Rien 103 3 12 2 Executing the CDF dump Program eeccescseessceececeeeeesaecencecaeeeaeeeeeeecseceeaecaeeesaeceeeecsaeseaeeeeeeecaeeeeneeaes 103 313 CDEISQUMip ES E EEEE SUR MT TS 105 BABA o NN 105 3 13 2 Executing the CDFirsdump Program eccececscees
37. data type and new data type are considered equivalent and the number of elements for the variable are the same Equivalent data types are described in Section 2 5 5 Data Type The supported data types are described in Section 2 5 Variables having any combination of data types may exist in the same CDF Number of Elements In addition to a data type each variable also has a number of elements This refers to the number of elements of the data type at each variable value For character data types CDF CHAR and CDF UCHAR this is the number of characters in each string A variable value consists of the entire character string The character string can be thought of as an array of characters For non character data types this must always be one 1 An array of elements per variable value is not allowed for non character data types 2 3 10 Record Variance A variable s record variance specifies whether or not the variable s values change from record to record The effect of a variable s record variance is defined as follows VARY The values do change from record to record Each variable record is physically written with no gaps between records 1 e if a record more than one beyond the maximum record is written the intervening records are also physically written and contain pad values If a 40 record is read beyond the maximum record written to a variable the pad value for the variable is returned Variables of this type are referred to
38. directory to be used for scratch files with the lt SELECT SCRATCHDIR gt operation of the Internal Interface which will override a scratch directory specified with CDF TMP CDF TMP The caching scheme used by the CDF library see Section 2 1 5 affects how these scratch files can impact performance On machines with large amounts of core memory available the cache size of a scratch files can be set high enough to result in no blocks actually being written paged out to that file In that case the scratch file is more like an allocated block of core memory 2 1 5 Caching Scheme 28 The CDF library reads and writes to open files in 10240 byte blocks The CDF library for each open file maintains a cache of 10240 byte memory buffers The CDF library attempts to keep in the cache the set of file blocks currently being accessed This results in fewer actual I O operations to the file if repeated accesses to these blocks would occur When the cache is completely full and a new block of the file is accessed one of the cache buffers is written back to the file Gf it was modified and the new block is read into that cache buffer unless the file is being extended in which case the cache buffer is simply cleared This process is known as paging By optimizing the number of cache buffers for a file improved performance can be achieved There is a tradeoff between having too few cache buffers and having too many Having too few cache buffers will cause
39. excessive paging while having too many cache buffers may slow performance because of the overhead involved in maintaining the cache although this is very rare Having too many cache buffers may also cause problems on machines having limited memory such as the PC and Macintosh The CDF library attempts to choose optimal default cache sizes based on a CDF s format and the operating system being used This is difficult because the CDF library does not know how an application will access a CDF For that reason an application may specify via the Internal Interface the number of cache buffers to be used for a file The number of cache buffers may be changed as many times as necessary while a file is open the first time will override the default used by the CDF library Default cache sizes may be configured for your CDF distribution when it is built and installed Consult your system manager for the values of these defaults or use the CDFinquire toolkit program The situations in which it will be necessary to specify a cache size will depend on how a CDF is accessed For example consider a variable in a multi file row major CDF having a dimensionality of 2 10 64 a data specification of CDF REAL8 1 and variances of T TT This variable definition results in each record of the variable being spread across 10 file blocks with the second dimension varying the fastest since the CDF s variable majority is row major If single value reads were used to acces
40. experiment with using larger cache sizes Significant gains in performance can be achieved with the proper cache sizes It is also important to allocate records for uncompressed variables This will reduce the fragmentation that can occur in the dotCDF file which degrades performance because of the increased indexing that occurs Allocating variable records is described in Section 2 3 12 29 File type Selecting Confirming dotCDF file lt SELECT_ CDF_CACHESIZE_ gt lt CONFIRM_ CDF_CACHESIZE_ gt rVariable file lt SELECT CrVAR CACHESIZE gt lt CONFIRM VAR CACHESIZE gt All rVariable files lt SELECT rVARS CACHESIZE gt lt CONFIRM rVARs CACHESIZE gt zVariable file lt SELECT zZ VAR CACHESIZE gt lt CONFIRM_ zVAR_CACHESIZE gt All zVariable files lt SELECT_ zVARs_CACHESIZE gt lt CONFIRM zZVARs CACHESIZE gt Staging scratch file lt SELECT STAGE CACHESIZE gt lt CONFIRM STAGE CACHESIZE gt Compression scratch file lt SELECT_ COMPRESS_CACHESIZE gt lt CONFIRM COMPRESS CACHESIZE gt Table 2 3 Cache Size Operations Internal Interface 2 2 CDFs The following sections describe various aspects of a CDF 2 2 1 Accessing All supported CDF operations are available using the Internal Interface The Extended Standard Interface is capable is doing almost all the CDF operations that are available with the Internal Interface The Original Standard Interface was developed in early 90 s an
41. file Indexing informational code Internal Interface Item libedf a libedf s1 libedf so LIBCDF LIB LIBCDF OLB little endian Majority Metadata Monotonicity multi file The encoding of the computer currently being used The Huffman compression algorithm A variable access method in which multiple records values are read written for a variable A set of functions callable from within IDL Interactive Data Language that allow access to CDFs The CDF distribution contains an IDL interface in addition to the CDF interface built into IDL by Research Systems Inc RSI the distributors of IDL The floating point representation of XDR A file included by a C or Fortran application that contains constants recognized by the CDF library pertaining to various aspects of CDF objects states The method used in a single file CDF to keep track of where each variable s records are located A status code indicating success but providing some additional information that may be of interest A set of routines in the CDF library callable from C and Fortran applications that provide all types of access to CDFs When the Internal Interface is used an object or state on which a function is performed The static CDF library on UNIX systems The dynamic CDF library on HP UX systems The dynamic CDF library on UNIX other than HP UX The static CDF library on MS DOS or Windows NT 95 98 systems The CDF library on VMS
42. file CDF by an application may result in a variable s records being noncontiguous There will be blocks of contiguous records but these blocks will not be contiguous in the dotCDF file For each variable in a single file CDF one or more index records will exist Each of these index records will contain one or more index entries Because the indexing scheme is now hierarchical each index entry will point to either another index record at a lower level in the hierarchy or to a block of contiguous variable records at the lowest level of the hierarchy An index entry consists of the following fields FirstRecord The number of the first record in a block of contiguous variable records or the first record indexed in a lower level index record LastRecord The number of the last record in a block of contiguous variable records or the last record indexed in a lower level index record ByteOffset The byte offset within the dotCDF file of the block of contiguous variable records or the byte offset of a lower level index record To find a particular variable record the CDF library must search through the index entries for that variable Improved performance will result if there are fewer index entries to search This can be achieved by having a larger number of records in each block of contiguous variable records resulting in fewer overall index entries Techniques used to achieve fewer index entries are outlined in the Allocated Records and Blocking
43. in terms of storage and offers more functionality than rVariable use of zVariable is strongly recommended As a matter of fact there s no reason to use rVariables at all if you are creating a CDF file from scratch One may wonder why there are rVariables and zVariables not just zVariables When CDF was first introduced only rVariables were available The inefficiencies with rVariables were quickly realized and addressed with the introduction of zVariables in later CDF releases 2 3 1 Types With the introduction of compression and sparseness for variables there now exist several different types of variables in addition to the distinction between rVariables and zVariables The various types of variables are as follows standard variable A variable in a single file CDF that is not compressed nor has sparse records or arrays compressed variable A variable in a single file CDF that is compressed and may or may not have sparse records but cannot have sparse arrays variable with sparse records A variable in a single file CDF that has sparse records and may be compressed have sparse arrays or have neither variable with sparse arrays A variable in a single file CDF that has sparse arrays and may or may not have sparse records but cannot be compressed multi file variable A variable in a multi file CDF It cannot be compressed have sparse records or have sparse arrays The term variable is used when a discussing a pro
44. lt indices gt lt entry data type gt lt entry value gt Note the lt value gt lt value gt lt value gt lt value gt Each field is defined as follows lt var name gt lt var data type gt lt n elems gt lt rec vary gt lt dim varys gt lt attr name gt lt entry value gt lt rec num gt The name of the rVariable The name must be delimited with a character not appearing in the name itself e g EPOCH or Temperature The delimiting characters are not part of the rVariable name in the CDF The data type for the rVariable The data type must be one of the following CDF_BYTE CDF INT1 CDF_UINT1 CDF_INT2 CDF_UINT2 CDF_INT4 CDF_UINTA CDF_INT8 CDF_REAL4 CDF_FLOAT CDF_REAL8 CDF_DOUBLE CDF_EPOCH CDF_EPOCH16 CDF_TIME_TT2000 CDF_CHAR or CDF_UCHAR The number of elements of the data type For character data types CDF_CHAR and CDF_UCHAR this is the number of characters in each string For non character data types this value must be one 1 The record variance of the rVariable This must be either T the values vary from record to record or F the values do not vary from record to record The dimension variances of the rVariable For each dimension there must be either a T the values vary along that dimension or F the values do not vary along that dimension Each dimension variance must be separated by at least one space If the rVariables have zero dimensions thi
45. lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies the zMode that should be used with the skeleton table If zMode is enabled zVariables will be created from the definitions in the rVariables section The zMode may be one of the following O Indicates that zMode should be disabled 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of F false are removed BACKWARD VMS backward UNIX Specifies whether an older backward compatible version of file in V2 7 is to be created The default will create a file of the same version as t
46. lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when a CDF is closed NO GWITHENTRIES VMS no gwithentries UNIX 68 Specifies whether or not gEntries are displayed with the gAttributes or on separate menus with one menu per gAttribute NO VWITHENTRIES VMS no vwithentries UNIX Specifies whether or not rEntries zEntries are displayed with the vAttributes or on separate menus with one menu per vAttribute ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS CDFEDIT SAMPLES CDFEDIT ZMODE 2 NOFORMAT CACHE 10D 100S 200C GISS WETLX CDFEDIT BROWSE PROMPT REPORT ERRORS
47. mm ss cccuuunnn where yyyy is the year 1707 2292 mm is the month 01 12 dd is the day of the month 01 31 hh is the hour 00 23 mm is the minute 00 59 ss is the second 00 59 or 00 60 if leap second ccc is the millisecond 000 999 uuu is microseconds 000 999 and nnn is nanoseconds 000 999 Functions exist that parse encode compute and decompose CDF_TIME_TT2000 values These functions similar to those for CDF EPOCH and CDF EPOCH16 data types are described in the CDF C Reference Manual for C applications and in the CDF Fortran Reference Manual for Fortran applications 2 5 6 Equivalent Data Types Certain data types are considered equivalent with respect to their representation in memory and in a CDF Table 2 9 shows the groups of equivalent data types CDF_CHAR CDF_INT2 CDF_INT4 CDF_INT8 CDF_REAL4 CDF_REAL8 CDF_UCHAR CDF_UINT2 CDF_UINT4 CDF_TIME_TT CDF_FLOAT CDF_DOUBLE 2000 CDF_INT1 CDF_EPOCH CDF_UINT1 CDF_EPOCH16 CDF_BYTE Table 2 9 Equivalent Data Types Note that while the signed and unsigned forms of a data type are considered equivalent by the CDF library they must be correctly interpreted by an application to produce the desired results 2 6 Compression Algorithms Several compression algorithms are supported by the CDF library Selecting the proper algorithm to use will depend on the characteristics of the data being compressed Experimentation with the available algorithms on th
48. neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS REPORT lt types gt VMS report lt types gt UNIX 73 Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and i CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching
49. of anything else in the CDF and have meaning only to the application Note that gEntries are sometimes referred to simply as entries Accessing The Standard Interface deals exclusively with rEntries for vAttributes and gEntries for gAttributes No access to zEntries is provided The Internal Interface may be used to access any type of attribute entry Numbering The rEntries and zEntries for a vAttribute and the gEntries for a gAttribute are numbered starting at one 1 for Fortran applications and starting at zero 0 for C applications For vAttributes the entry numbers are in fact the variable numbers of the variables being described rEntries correspond to rVariables and zEntries correspond to zVariables For gAttributes the gEntry numbers have meaning only to the application The entry numbers used need not be contiguous as are variable and attribute numbers An application may choose to write any combination of entries for a particular attribute keeping in mind that the entry numbers used for a vAttribute correspond to the existing variables Data Specification Each entry for an attribute has a data specification and an associated value A data specification consists of a data type and a number of elements of that data type The supported data types are described in Section 2 5 The entries for an attribute may have any combination of data specifications For character data types the number of elements is the number of characte
50. parameters and the default toolkit gualifiers 3 9 2 Executing the CDFinguire Program Usage VMS CDFINQUIRE ID NO PAGE ABOUT UNIX cdfinquire id nolpage about Parameter s None Qualifier s 98 ID VMS id UNIX Causes the version of your CDF distribution and the default toolkit qualifiers to be displayed This qualifier is required NO PAGE VMS no page UNIX Specifies whether or not the output is displayed a page at a time A prompt for the RETURN key will be issued after each page A page is generally 22 lines of output ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS CDFINQUIRE ID PAGE UNIX cdfinquire id page VMS UNIX Command line help is displayed when CDFinquire is executed without any arguments 3 9 3 Output from the CDFinquire Program The version of your CDF distribution is displayed first followed by the configurable parameters and then the default toolkit qualifiers in the style of the system being used 3 10 CDFdir 3 10 1 Introduction The CDFdir utility is used to display a directory listing of a CDF s files The dotCDF file is displayed first followed by the rVariable files and then the zVariable files if either exist in a multi file CDF in numerical order 3 10 2 Executing the CDFdir Program The command line syntax for CDFdir is as follows Usage
51. physical format It is important to state here that the use of a data abstraction in no way inhibits access to physical data or necessarily makes such access inefficient It merely provides a way of generalizing the data model and makes possible the specification of a uniform interface for manipulation of a data set The data abstraction allows future extensibility and provides for conceptual simplicity while isolating machine and device dependence The contents of a CDF fall into two categories The first is a series of records comprising a collection of variables consisting of scalars vectors and n dimensional arrays The second is a set of attribute entries metadata describing the CDF in global terms or specifically for a single variable This dual function of CDF is what provides its data set independence Both the metadata attributes and the data objects variables are combined into an integrated data set An important element of the CDF conceptual data abstraction is the virtual dimensional layer that allows data objects that share a subset of the overall CDF dimensionality to be projected into the full dimensional space This capability is made available through the use of logical dimensional variances that indicate the subset of CDF dimensions that are applicable How is CDF Used The origins of CDF date back to the development of the NASA Climate Data System at the National Space Science Data Center NSSDC As such it has had three
52. rVariables An even better example of how zVariables are preferred over rVariables in certain situations involves the storage of 1 dimensional arrays vectors Assume that five 1 dimensional arrays are being stored with dimension sizes of 2 3 5 7 and 25 Using rVariables with a dimensionality of 1 25 would waste considerable space while using rVariables with a dimensionality of 5 2 3 5 7 25 and dimension variances of T TFFFF T FTFFF T FFTFF T FFFTF and T FFFFT would be quite confusing to deal with zVariables with dimensionalities of 1 2 1 3 1 5 1 7 and 1 25 would be straight forward and efficient 1 6 Attributes The second component of a CDF is the metadata Metadata values consist of user supplied descriptive information about the CDF and the variables in the CDF by way of attributes and attribute entries Attributes can be divided into two categories attributes of global scope gAttributes and attributes of variable scope vAttributes gAttributes describe the CDF as a whole while vAttributes describe some property of each variable rVariables and zVariables in the CDF Any number of attributes may be stored in a single CDF The term attribute is used when describing a property that applies to both gAttributes and vAttributes The notation for dimensionality used here is lt num dims gt lt dim sizes gt where lt num dims gt is the number of dimensions and lt dim sizes gt is zero or more dimension sizes se
53. s values do not vary from record to record record variance of NOVARY all of that variable s records beyond the first one are virtual and have the same values as the first record only the first record is physically stored If a record has not yet been written to that variable then all of its records are virtual and contain the pad value for that variable 2 Ifa variable s values do vary from record to record record variance of VARY then the records beyond the last record actually written are virtual and contain the pad value for that variable 3 Ifa variable has sparse records then any unwritten records for that variable are virtual and contain either the pad value for that variable or the previous existing record s values depending on the type of sparse records Sparse records are described on page 48 Record variance is described in Section 2 3 10 Variable pad values are described in Section 2 3 20 The maximum record written is maintained by the CDF library for each variable in the CDF The maximum CDF record is simply the maximum rVariable record written of all the rVariables This quantity is available through the Standard Interface when inquiring about a CDF Because the Standard Interface does not allow access to zVariables zVariables are not considered when determining the maximum CDF record The maximum CDF record would be used by applications dealing only with rVariables The maximum record written for each rVariable and
54. the blocking factor becomes the maximum number of records per compressed block Depending on which records are written some of the compressed blocks may contain fewer A variable s blocking factor was previously called its extend records 46 records The blocking factor for a compressed variable may be explicitly specified by an application or a default may be used as determined by the CDF library the default blocking size for compressed variable 5120 bytes decided by the record size Using the default for the compressed variables may be too small and inefficient if writing a large number of records Once a record has been written to the variable however the blocking factor cannot be changed Uncompressed Variables With Sparse Records The CDF library uses a staging area scratch file for uncompressed variables with sparse records This is done in an attempt to minimize the indexing for the variable as described in Section 2 2 7 when the records being written are not first allocated by an application The blocking factor specifies the number of records to be maintained in the staging area for the variable which will be the maximum number of unallocated consecutive records that would be stored contiguously in a block when written by an application An explicit blocking factor can be specified or a default determined by the CDF library may be used Blocking factors are explicitly specified for variables using the lt PUT r ZVAR BLO
55. the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when a CDF is closed BACKWARD VMS backward UNIX Specifies whether or not to make the converted file as a backward compatible file i e V 2 7 instead of the default V 3 file If the environment variable CDF_FILEBACKWARD on Unix or Windows or CDF FILEBACKWARD on OpenVMS is set to TRUE the converted file is then automatically a V2 7 file whether this option is set If this environment variable is not set or set to anything other than TRUE then use this option to create a backward file CHECKSUM lt MODE gt VMS checksum lt mode gt UNIX By default whether to set the checksum mode on the converted file is based on the environment variable CDF_CHECKSUM or CDF CHECKSUM on OpenVMS or the source file If the environment variable is set to a valid method the checksum bit for the co
56. the output is displayed a page at a time A prompt for the RETURN key will be issued after each page A page is generally 22 lines of output NOJUPDATE VALIDS VMS no update valids UNIX Specifies whether or not the VALIDMIN and VALIDMAX attribute entry values are updated for each variable based on the actual minimum and maximum values found with fill values being ignored if requested If the VALIDMIN and VALIDMAX attributes do not exist they are created NO JUPDATE SCALES VMS noJupdate scales UNIX Specifies whether or not the SCALEMIN and SCALEMAX attribute entry values are updated for each variable based on the actual minimum and maximum values found with fill values being ignored if requested If the SCALEMIN and SCALEMAX attributes do not exist they are created NOJUPDATE MONOTONIC VMS noJupdate monotonic UNIX Specifies whether or not the MONOTONIC attribute entry values are updated for each variable based on the monotonicity found with fill values being ignored if requested If the MONOTONIC attribute does not exist it is created ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies the zMode that should be used with the CDF The zMode may be one of the following 88 0 Indicates that zMode should be disabled 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a
57. this tool program OG testl cdf test2 cdf outtest cdf CDFM ERGE EFIXES I1D1 1D2 1D3 cdffilel cdffile2 cdffile3 mergedFile CDFM UN 1N Y Ur ERGE O PREFIX cdffilel cdffile2 cdffile3 mergedFile CDFM UNIX and Windows ERGE OG FILE filelist DATAONLY 102 e cdfmerge log testl cdf test2 cdf outtest cdf cdfmerge prefixes 1D1 1D2 1D3 cdffilel cdffile2 cdffile3 mergedFile cdfmerge noprefix cdffilel cdffile2 cdffile3 mergedFile cdfmerge log file filelist dataonly ae Ae oe Help is displayed when CDFmerge is executed without any arguments 3 12 CDFdump 3 12 1 Introduction The CDFdump program displays or extracts the contents of a CDF file to a screen default or text file This program extracts and displays one variable at a time while the CDFexport programs extracts and displays the contents of a CDF file in a table format each column representing a variable CDFdump extracts the value of the variable attributes but CDFexport does not extract the value of the variable attributes 3 12 2 Executing the CDFdump Program Usage VMS CDFDUMP NO FORMAT DUMP lt option gt OUTPUT lt file path gt VARS lt varl var2 varN gt ABOUT NOHEADER RECORDRANGE lt STARTREC ENDREC gt COL2ROW cdf path gt UNIX including Mac OS X Windows cdfdump no format dump lt option gt output lt fil
58. used to specify a function to be performed on an item The syntax is lt function_ item_ gt 10 There are actually two types of zMode read on 17 In a future release of CDF support for rVariables will be eliminated zMode is provided to ease the transition from rVariables to the more extensible zVariables rVariables are essentially a subset of zVariables This notation is used throughout this document In this case there are two dimensions whose sizes are 180 and 360 A dimensionality of zero is represented as 0 26 rVariable Name Variances EPOCH T FF LATITUDE T TF LONGITUDE T FT HUMIDITY T TT If this CDF were to be placed into zMode 1 the following zVariables would replace the existing rVariables rVariable Name Dimensionality Variances EPOCH 2 180 360 T FF LATITUDE 2 180 360 T TF LONGITUDE 2 180 360 T FT HUMIDITY 2 180 360 T TT Note that the dimensionality of each zVariable is the same as it was for the rVariables in the CDF However if zMode 2 were used the following zVariables would replace the existing rValues rVariable Name Dimensionality Variances EPOCH 0 T LATITUDE 1 180 T T LONGITUDE 1 360 T T HUMIDITY 2 180 360 T TT In this case the false dimensional variances were removed which decreased the dimensionality in several of the variables A CDF can be placed into or taken out of zMode any number of times while it is open Each time the zMode is changed for a CDF it would
59. variable majority of the CDF This may be either ROW or COLUMN Variable majority is described in Section 2 3 15 The format of the CDF This may be either SINGLE or MULTI CDF formats are described in Section 2 2 7 Note that this line is optional Skeleton tables created by SkeletonTable in CDF V2 0 did not have this line because the single file option did not exist To allow SkeletonCDF to read skeleton tables created with SkeletonTable in CDF V2 0 this line was made optional If omitted SkeletonCDF will create a CDF with the default format for your CDF distribution Consult your system manager to determine this default SkeletonTable in CDF V2 1 and beyond always generates this line regardless of the version of the CDF being read The number of rVariables in the CDF SkeletonTable always places the correct number here However when SkeletonCDF reads a skeleton table this value is ignored but a place holder is necessary The number of rVariables created is determined by the number of rVariable definitions in the rVariable definitions section The number of zVariables in the CDF SkeletonTable always places the correct number here However when SkeletonCDF reads a skeleton table this value is ignored but a place holder is necessary The number of zVariables created is determined by the number of zVariable definitions in the zVariable definitions section The number of gAttributes in the CDF SkeletonTable always places the correct numb
60. write since full physical records are being accessed The majority of the values in the full physical records retrieved from placed into the memory buffer must be the same as the variable majority of the CDF For example consider a column major CDF containing the following three zVariables as well as others zVariable Name Data Specification Dimensionality Variances 53 zVarl CDF INT2 14 3 0 T zVar2 CDF CHAR 7 1 5 T T ZVar3 CDF REAL8 1 2 12 4 T TT If a Fortran application were to perform a multiple variable read on these three zVariables it could define a STRUCTURE to receive the physical records as follows STRUCTURE inputStruct REAL 8 zVar3values 2 4 INTEGER 2 zVarl value CHARACTER 7 zVar2values 5 END STRUCTURE Note that because a full physical record for the zVariable zVar2 is an odd number of bytes it would most likely cause a gap in the STRUCTURE if not placed at the end on some computers An approach that would work on all computers would be to use EQUIVALENCE statements as follows INTEGER 2 zVarl value CHARACTER 7 zVar2values 5 REAL 8 zVar3values 2 4 BYTE buffer 101 EQUIVALENCE zVar3values buffer 1 EQUIVALENCE ZVarl value buffer 65 EQUIVALENCE zVar2values buffer 67 The EQUIVALENCE statements ensure that the full physical records will be contiguous In each of the above examples the order of the zVariables would be zVar3 zVarl and zVar2 C applications must also be concerned with
61. 000 000 000 CDF TIME TT2000 9223372036854775807 as 0000 01 01T00 00 00 000000000 CDF_CHAR space character CDF_UCHAR space character Table 2 8 Default Pad Values Variable pad values are specified using the lt PUT_ r ZVAR_PADVALUE_ gt operation of the Internal Interface The pad value being used for a variable can be inquired with the lt GET_ t zVAR_PADVALUE gt operation If a pad value has not been explicitly specified for a variable the default pad value based on the variable s data type will be returned along with the NO_PADVALUE_SPECIFIED informational status code The existence of an explicitly specified pad value can be confirmed for a variable without actually inquiring the value using the lt CONFIRM_x ZVAR_PADVALUE_ gt operation 2 4 Attributes CDF attributes are the mechanism for storing metadata A new attribute may be created in a CDF at any time These default pad values can be changed by your system manager when the CDF distribution is built 4 It is a value of 9223372036854775807 one more than the minimum value for an 8 byte integer We present it a such date time an invalid date for CDF TIME TT2000 data type which is similar to CDF EPOCH EPOCH16 data type 55 2 4 1 Naming Each attribute in a CDF has a unique name Attribute names are case sensitive regardless of the operating system being used and may consist of up to CDF ATTR NAME LEN or CDF ATTR NAME LEN256 printable cha
62. 25 CDF s IDL Interface IDL has a built in support for CDF and CDF files can be created and manipulated in IDL By default CDF files created in IDL 6 3 or later can t be read by IDL 6 2 or earlier or CDF 2 7 2 or earlier However IDL 6 3 or later can read files that were generated with IDL 6 2 or earlier or CDF 2 7 2 or earlier If you create files in IDL 6 3 or later and want to share files with colleagues who access CDF files using IDL 6 2 or earlier or CDF 2 7 2 or earlier you can do so by calling the CDF SET CDF27 BACKWARD COMPATIBLE routine prior to creating CDF files i e CDF_CREATE Note that if this option is used the maximum file size is 2G bytes 2 1 2 CDF Modes Once a CDF has been opened or created and not yet closed the CDF library may be configured to act on that CDF in one or more modes These modes are specified independently for each open CDF Read Only Mode A CDF may be placed in read only mode via the Internal Interface using the lt SELECT_ CDF_READONLY_MODE_ gt operation Only read access will be allowed on the CDF all attempts to modify the CDF will fail A CDF may be toggled in and out of read only mode any number of times Note that attempts to modify a CDF may also fail if insufficient access privileges exist for the CDF the file system enforces this access However if the CDF is modified while not in read only mode subsequently setting read only mode in the same session will not prevent further
63. 28 selecting 68 74 81 84 89 94 97 interfaces 20 25 limits 28 open CDFs 28 modes 26 0 0 to 0 27 147 decoding 35 performance considerations 37 read only 26 zMode 26 67 73 80 84 88 94 97 example 26 selecting 26 zMode 1 26 zMode 2 26 scratch files 28 CDF toolkit 19 63 CDFcompare 82 CDFconvert 76 CDFdir 99 CDFdump 103 CDFedit 66 CDFexport 70 CDFinguire 98 CDFirsdump 105 CDFleapsecondsinfo 107 CDFmerge 100 CDFstats 86 CDF validate 106 command line syntax 63 default settings 64 Java version 65 Macintosh OS X 64 SkeletonCDF 95 SkeletonTable 91 Windows NT 95 98 2000 XP 65 CDF ATTR NAME LEN256 37 CDF EPOCH 59 CDF EPOCH16 59 CDF error 127 CDF TIME TT2000 59 CDF VAR NAME LEN256 37 CDFcompare 82 executing 82 output 86 CDFconvert 76 executing 77 output 82 CDFdir 99 executing 99 output 100 CDFdump 103 executing 103 cdfedit 22 66 executing 67 interaction with 69 CDFerror 127 CDFexport 70 executing 70 interaction with 76 CDFinguire 98 executing 98 output 99 CDFirsdump 105 executing 105 CDFleapsecondsinfo 107 executing 107 CDFmerge 100 executing 100 CDFs 30 accessing 30 backward compatible 77 browsing 66 checksum changing 77 closing 30 comparing 82 compression 14 37 algorithms 60 changing 77 conceptual organization 7 converting 76 creating 30 editing modifying 66 encoding 14 33 changing 33 77 eguiv
64. 4 Interaction with CDFexportiia cain ilge ais Arian aes hk agit wn lam Gla ee are hea 76 NN 76 SAL A AN 76 3 42 Executing the CDFconvert Program wemrennenneotteoeneenneneeneoeeneenrneenaeennneenaeeeneeennneeneeeene enne 77 3 4 3 Output from the CDFconvert Program sarres ienet EEE E pE AEAEE TEER E E EN 82 BD HCE COMPAL hen rA A E N e EEEE Ee EAREN NANS N REE a TAE RA EA 82 33510 E AMEOAUCUOM dada dad dae E OR TENE e 82 3 9 2 Executing the CDF compare Progra M weiss rv setas avers A ie 82 3 5 3 Output from the CDFcompare Program mieenteoneneonneooneennenoeneeneneonnneenaeeneneenneeenaeennneeneeeene enne 86 3 01 CIDE Stats ti A A A e A e paves ove an Ge E ls E 86 FOLIO UCM A ch r ibe aaa A Wea i ae Ta el s 86 3 02 Special Attribute Sages eis eceuccacs ienr noos A ES es ke Seek ongi k ha kPa ta ea 86 316 3 Executing the CDF stats PEO a chs envir sundae via kalde Soe vei let E e e cede 87 3 6 4 Output from the CDFstats Program terreenteeneneonneoeneeeneneoneeneneenrneenaeennneenneeeneeennneene een enne 90 Ske Skeleton Lables series oa Soe ea os 91 3421 KA OdUCUON s cies koik NN 91 3 2 Special Attribute Usage ieira ie a IET tuy danna Waa due vk places nasa ESA ap paid 92 3 7 3 Executing the SkeletonTable Program ieeonneneeerereeeereenrneenneeneneenneeneennnneeneeeeneenneeet 92 3 7 4 Output from the SkeletonTable Program
65. AA NN 28 AS C D NN 30 ASE E NS NA O NN 30 ARAN NN 30 22 3 O NN 30 A O O O NN 30 A SO NN 31 PAPA A NN 31 DQ O LA NN 31 A NN 33 22 9 O A NN NA 35 ZeD VOE A COMPLESS ION A A A SS A E E Sees 37 DR AS A A A E SS 37 23 Wal tr IAS A A A AA A 37 E TYPES O RN 38 232 ACCESSING O NN 38 E O DD 38 DES O O ON 38 DIO NAO AAA it 39 2 3707 NUMAN ea 39 E A UA PO A OA 39 3 PESA AD 4108 53 1153 980 1 A RNE 39 925 DAA PEC ACM A TD 40 2 3 10 Record VATICAN SS 40 2 3 11 Dimension VTA A A A Id E te de 41 ZINC Sr A AA A A A Org kka KE aa ese 42 E Sparse Aay ONO 47 DSA COMPLESSIOM sec ces IR AAA AT KETSU Ata t M 47 O A NN 48 ES E CAI OY 49 A TAY Per ACCESS O E E n buat eae es 50 23 18 SSEQuential ACCESS rre tn valet vana ak A UTA AEDU EL Leeda ads Suess Soe TT iia iD 52 2 3 19 Multiple Variable ACCESS 7116 sint varte attend even Peek nede tiv aces ke see kat ae ATR aa Tuan a Dante toe a 53 2 3 20 Variable Pad Valdes ia 54 2 4 AHTIDULES 1944023 an NA 55 O A A 56 ZAD INUNDA ni a A A As 56 ZAS ATGADBULE SCOPES incaico ini AO REA in ass 56 DAA A A 57 24 Attribute ENIES AA Ri 57 2 9 Data A NR NAS 58 2S S Integer Data A O NS 58 25 2 Floating Point Data Tips A ic 58 25 37 Character DAA DA DT TD 58 ZAR EPOCH Data Types ri sinki davai eE t aga 59 2529 112000 Data Types O arate 59 25 6 Equivalent Data Types rin videvikku kukke ra 60 2 6 Compression Algorithm S A ai de 60 2 6 Run Length Encoding 4 torti si
66. AL4 0 0 VALIDMAX CDF_REAL4 300 0 UNITS CDF_CHAR Knots FORMAT CDF_CHAR F9 1 A 7 End Section This section simply consists of the keyword end This section is required A 8 Example Skeleton Table An example skeleton table containing rVariables and zVariables follows Skeleton table for the example2 CDF Generated Thursday 17 Nov 1994 14 07 58 CDF created modified by CDF V2 4 10 Skeleton table created by CDF V2 5 0 120 Dimension Variances Dimension Variances header GLOBALattributes VARIABLEattributes Variables G Attributes V Attributes Records Attribute Name TITLE FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT variables Variable Name EPOCH Attribute Name FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT Variable Name CDF NAME example2 DATA ENCODING NETWORK MAJORITY ROW FORMAT SINGLE Entry Number Data Type CDF EPOCH Data Type CDF CHAR CDF EPOCH CDF EPOCH CDF EPOCH CDF EPOCH CDF CHAR CDF CHAR Data Type Dims Sizes 7 2 11 7 Data Type Value CDF CHAR Title for Number Record Dimension Elements Variance Variances 1 E F F Value Time since 0 A D CE 01 Jan 0000 00 00 00 000 01 Jan 2089 00 00 00 000 01 Apr 1986 07 00 00 000 01 Apr 1986 23 00 00 000 milliseconds UT 3 E14 0 Number Record Dimens
67. AR CDF UCHAR CDF BYTE CDF INT1 and CDF UINT1 are encoded in the same way on each Multiple Byte Integer Data Types The multiple byte integer data types CDF INT2 CDF UINT2 CDF INT4 CDF UINT4 CDF INT8 and CDF TIME TT2000 are encoded in one of two ways big Endian or little Endian Big Endian has the least significant byte LSB in the highest memory location while little Endian has the LSB in the lowest memory location The supported computers use big Endian or little Endian as shown in Table 2 4 Network XDR encoding uses big Endian encoding for multiple byte integer data types Big Endian Little Endian Sun VAX SGi Iris DECstation IBM RS6000 PC HP 9000 DEC Alpha OSF 1 NeXT DEC Alpha OpenVMS Macintosh PPC Mac OS X Network XDR Table 2 4 Equivalent Byte Orderings Single Precision Floating Point Data Types The single precision floating point encodings on the supported computers are either IEEE 754 floating point or Digital s F FLOAT floating point There are also two different byte orderings for the computers that use IEEE 754 big Endian and little Endian The single precision floating point encodings for each supported computer are shown in Table 2 5 Network XDR encoding uses IEEE 754 big Endian encoding for single precision floating point data types 34 TEEE 754 Big Endian IEEE 754 Little Endian Digital s F FLOAT Sun DECstation VAX SGi Iris DEC Alpha OSF 1 DEC Alpha OpenVM
68. BUT NOT LIMITED TO ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL CONFORM TO SPECIFICATIONS ANY IMPLIED WARRANTIES OF MERCHANTABILITY FITNESS FOR A PARTICULAR PURPOSE OR FREEDOM FROM INFRINGEMENT ANY WARRANTY THAT THE SUBJECT SOFTWARE WILL BE ERROR FREE OR ANY WARRANTY THAT DOCUMENTATION IF PROVIDED WILL CONFORM TO THE SUBJECT SOFTWARE THIS AGREEMENT DOES NOT IN ANY MANNER CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY OR ANY PRIOR RECIPIENT OF ANY RESULTS RESULTING DESIGNS HARDWARE SOFTWARE PRODUCTS OR ANY OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT SOFTWARE FURTHER GOVERNMENT AGENCY DISCLAIMS ALL WARRANTIES AND LIABILITIES REGARDING THIRD PARTY SOFTWARE IF PRESENT IN THE ORIGINAL SOFTWARE AND DISTRIBUTES IT AS IS Waiver and Indemnity RECIPIENT AGREES TO WAIVE ANY AND ALL CLAIMS AGAINST THE UNITED STATES GOVERNMENT ITS CONTRACTORS AND SUBCONTRACTORS AS WELL AS ANY PRIOR RECIPIENT IF RECIPIENT S USE OF THE SUBJECT SOFTWARE RESULTS IN ANY LIABILITIES DEMANDS DAMAGES EXPENSES OR LOSSES ARISING FROM SUCH USE INCLUDING ANY DAMAGES FROM PRODUCTS BASED ON OR RESULTING FROM RECIPIENT S USE OF THE SUBJECT SOFTWARE RECIPIENT SHALL INDEMNIFY AND HOLD HARMLESS THE UNITED STATES GOVERNMENT ITS CONTRACTORS AND SUBCONTRACTORS AS WELL AS ANY PRIOR RECIPIENT TO THE EXTENT PERMITTED BY LAW RECIPIENT S SOLE REMEDY FOR ANY SUCH MATTER SHALL BE THE IMMEDIATE UNILATERAL TERMINATION OF THIS AGREEMENT 5 GENERAL TERMS A Te
69. CDF User s Guide Version 3 5 September 4 2013 Space Physics Data Facility NASA Goddard Space Flight Center Copyright 2013 NASA GSFC Space Physics Data Facility NASA Goddard Space Flight Center Greenbelt Maryland 20771 U S A This software may be copied or redistributed as long as it is not sold for profit but it can be incorporated into any other substantive product with or without modifications for profit or non profit Ifthe software is modified it must include the following notices The software is not the original for protection of the original author s reputations from any problems introduced by others Change history e g date functionality etc This copyright notice must be reproduced on each copy made This software is provided as is without any express or implied warranties whatsoever Internet gsfc cdf support Olists nasa gov Permission is granted to make and distribute verbatim copies of this document provided this copyright and permission notice are preserved on all copies Contents 1 2 lA IITA PEETRE NN 1 1 Introductions sercan NN 7 1 2 INIA AAA NN 7 1 3 Conceptual Organi Zaion sesinin eiie ienie aE caste keha dv saves caccechasdevacacuanaeiasben devs EE E NE EE ih 7 1 4 Eeatures ofthe CD FIL Drary ss is orge cs vessels kreeni coun do eee Bv teeke OSE E SREE 8 1 4 1 CS bbs KU KUS seiuasvacedbac cases TEE AE 9 1 42 Data Encoding Options ati LIA he Ma ORE 14 1 43 COMPTE ach
70. CDF NAME sample0 DATA ENCODING NETWORK MAJORITY ROW FORMAT SINGLE Variables G Attributes V Attributes Records Dims A gAttributes Section The gAttributes section contains the definition of each gAttribute as well as any gEntries for those gAttributes The format of the gAttributes section is as follows GLOBAL attributes lt global scope attribute definition gt 111 180 360 lt global scope attribute definition gt lt global scope attribute definition gt lt global scope attribute definition gt Where lt global scope attribute definition gt needless to say is a gAttribute definition Zero or more gAttribute definitions are allowed There is no limit on the number of attributes that a CDF may have The format of each gAttribute definition is as follows Attribute Name lt attr name gt Entry Data Number Type Value lt entry n gt lt data type gt lt value gt lt entry n gt lt data type gt lt value gt lt entry n gt lt data type gt lt value gt lt entry n gt lt data type gt lt value gt Note the The fields are defined as follows lt attr name gt lt entry n gt lt data type gt lt value gt The name of the gAttribute The name must be delimited with a character not appearing in the name itself e g TITLE or History The delimiting characters are not part of the gAttribute name in the CDF The gEntry number Zero or more gEntries ma
71. CDFdir is not available on Windows systems It s also not available in Java version pf the CDF toolkit 99 VMS CDFDIR lt cdf path gt UNIX including Mac OS X cdfdir lt cdf path gt NOTE This tool is not supported by Windows Parameter s lt cdf path gt The file name of the CDF for which to display a directory listing do not specify an extension Example s VMS CDFDIR NCDSSDATA GISS WETL CLIMATOLOGY CDFDIR TEMP FGGE3B UNIX cdfdir cac sst blended oe cdfdir CDFs giss wetl climatology Help is displayed when CDFdir is executed without any arguments 3 10 3 Output from the CDFdir Program The format of the output from CDFdir is that of a directory listing on the operating system being used 3 11 CDFmerge 3 11 1 Introduction The CDFmerge utility merges two or more CDF files into a single file Input file names and output file name can be specified either from the command line or in a text file If there are many CDF files to merge it may be more convenient to specify the input file names and output file name in a text file with the file flag 3 11 2 Executing the CDFmerge Program The command line syntax for CDFmerge is as follows Usage 100 VMS CDFMERGE NOPREFIX PREFIXES lt prefix1 gt lt prefix2 gt lt prefixN gt NO LOG NO DATAONLY ABOUT lt lt cdf path1 gt lt cdf path2 gt lt cdf path
72. CDFexport by default exports variables in a sequence based on their order in the source CDF file no matter how settings file is defined This option requires the settings file to be present either specified at the command line or the default file export set or simple set for simple mode residing in the current directory Only those variables specified in settings file are considered for export The file also dictates the output sequence of variables in the export ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS 15 DFEXPORT SAMPLES DFEXPORT ZMODE 2 CACHE 50d 100s GISS WETLX DFEXPORT PROMPT REPORT W E INITIAL EXCLUSIVE NOFORMAT DFEXPORT SIMPLE BATCH TEXT TEXT FLUX OUT FLUX1996 DFEXPORT BATCH CDF CDF MYCDF1 INCLUDE varl var2 test DFEXPORT BATCH CDF CDF MYCDF2 EXCLUDE varl var2 test DFEXPORT BATCH TEXT TEXT MYTEXT EXCLUDE varsfile home cdf varslist txt test cdfexport BATCH CDF CDF MYCDF EPOCHRANGE 01 10 2005 00 00 00 000 01 10 2005 0 00 00 000 test CDFEXPORT BATCH TEXT TEXT MYOUTPUT txt CONTROLSETTINGS SETTINGS EXPORT set test QO GG 0 32 TO UN Er vr vn vn n UNIX cdfexport samples cdfexport zmode 2 cache 50d 100s giss_wetl cdfexport prompt report w e initial exclusive noformat
73. CKINGFACTOR gt operation of the Internal Interface The blocking factor may be inquired using the lt GET r zZ VAR BLOCKINGFACTOR gt operation If an explicit blocking factor has not been specified the default blocking factor for the variable will be returned Note the distinction between records allocated and records actually written The CDF library may allocate more records than are actually written by an application for the reasons stated above Both the number of records written to a variable and the number of records allocated for that variable may be inguired using the Internal Interface Deleting The records of a variable in a single file CDF may be deleted If the variable has sparse records the deleted records simply cease to exist A gap of one or more missing records will be formed But ifthe variable does not have sparse records the records following the block of deleted records are immediately renumbered to fill in the gap created The record numbers remain consecutive without a gap Variable records are deleted using the lt DELETE r ZVAR RECORDS gt operation of the Internal Interface 2 3 13 Sparse Arrays The idea being that only those values actually written to a variable array record will be physically stored Currently unwritten values in each variable array are physically stored using the variable s pad value Note that specifying a compression for a variable will in many cases result in a disk space savings si
74. DF_FILE BACKWARD environment variable See section 4 18 of the CDF C Reference Manual and the CDF Fortran Reference Manual for details on how to create CDF 2 7 compatible files The command line version of the CDFedit CDFexport and CDFconvert utility programs can now create CDF files that can be read by CDF 2 7 2 or earlier The lt GET_ CDF INFO gt routine now returns the data type of 64 bit off t or __int64 on Windows for the compressed file size cSize and uncompressed file size uSize parameters in V3 1 while they used to return as 32 bit long integer in V2 5 2 6 or 2 7 Thus if you have a legacy application that calls the lt GET_ CDF_INFO_ gt routine you MUST change the data type of the cSize and uSize parameters to off_t or __int64 on Windows from long to access files that were created with V3 If the file accessed was created with V2 5 V2 6 or V2 7 you should always use long instead of off_t to get the correct results Using off_t for non V3 files from V3 library may or may not return the correct results depending upon what operating system it is executed under D 3 Changes The following features have been added to from CDF V3 0 to V3 2 1 Add checksum feature to allow verification of data integrity in a CDF file upon its opening 2 Ability to create a CDF file that is compatible with CDF 2 7 2 or earlier 3 Addition of the easy to use Extended Standard Interface that allows almost all of the CDF operations w
75. ERROR CDF EXISTS CDF INTERNAL ERROR CDF NAME TRUNC CDF OK CDF OPEN ERROR 7 Writing initial records to a variable after a value excluding the pad value has already been written to that variable 8 Changing a variable s blocking factor when a compressed variable and a value excluding the pad value has been written or when a variable with sparse records and a value has been accessed 9 Changing an attribute entry s data specification where the new specification is not eguivalent to the old specification The CDF or variable cannot be compressed For CDFs this occurs if the CDF has the multi file format For variables this occurs if the variable is in a multi file CDF values have been written to the variable or if sparse arrays have already been specified for the variable Error Sparse arrays cannot be specified for the variable This occurs if the variable is in a multi file CDF values have been written to the variable records have been allocated for the variable or if compression has already been specified for the variable Error Sparse records cannot be specified for the variable This occurs if the variable is in a multi file CDF values have been written to the variable or records have been allocated for the variable Error Error detected while trying to close CDF Check that sufficient disk space exists for the dotCDF file and that it has not been corrupted Error Cannot create the CDF specifi
76. Error The time or date value supplied for the CDF_EPOCH or CDF_EPOCH 16 data type is invalid Error The operation is illegal for the attribute s scope For example only gEntries may be written for gAttributes not rEntries or zEntries Error The attempted operation is illegal while in zMode Most operations involving rVariables or rEntries will be illegal Error The specified operation 1 e opening is not allowed on Version 1 CDFs Error 132 ILLEGAL TT2000 VALUE MULTI FILE FORMAT NA FOR VARIABLE NEGATIVE FP ZERO NO ATTR SELECTED NO CDF SELECTED NO DELETE ACCESS NO ENTRY SELECTED NO MORE ACCESS NO PADVALUE SPECIFIED NO SU NO SU NO SU NO SU NO SU NO STATUS SELECTED CH ATTR CH CDF CH ENTRY CH RECORD CH VAR NO VAR SELECTED The time or date value supplied for the CDF TIME TT2000 data type is invalid Error The specified operation is not applicable to CDFs with the multi file format For example it does not make sense to inguire indexing statistics for a variable in a multi file CDF indexing is only used in single file CDFs Informational The attempted operation is not applicable to the given variable Warning One or more of the values read written are 0 0 an illegal value on VAXes and DEC Alphas running OpenVMS Warning An attribute has not yet been selected First select the attribute on which to perform the operation Error A CD
77. F has not yet been selected First select the CDF on which to perform the operation Error Deleting is not allowed read only access Make sure that delete access is allowed on the CDF file s Error An attribute entry has not yet been selected First select the entry number on which to perform the operation Error Further access to the CDF is not allowed because of a severe error If the CDF was being modified an attempt was made to save the changes made prior to the severe error In any event the CDF should still be closed Error A pad value has not yet been specified The default pad value is currently being used for the variable The default pad value was returned Informational A CDF status code has not yet been selected First select the status code on which to perform the operation Error The named attribute was not found Note that attribute names are case sensitive Error The specified CDF does not exist Check that the file name specified is correct Error No such entry for specified attribute Error The specified record does not exist for the given variable Error The named variable was not found Note that variable names are case sensitive Error A variable has not yet been selected First select the variable on which to perform the operation Error 133 NO VARSIN CDF NO WRITE ACCESS NOT A CDF PRECEEDING RECORDS ALLOCATED READ ONLY DISTRIBUTION READ ONLY MODE SCRATC
78. FORMAT A Fortran or C format specification that is used when displaying a variable value VALIDMIN The minimum valid value for a variable VALIDMAX The maximum valid value for a variable FILLVAL The value used for missing or invalid variable values MONOTON The monotonicity of a variable INCREASE strictly increasing values DECREASE strictly decreasing values or FALSE not monotonic Monotonicity only applies to NRV variables that vary along one dimension and RV variables that vary along no dimensions SCALEMIN The minimum value for scaling a variable when graphically displaying its values SCALEMAX The maximum value for scaling a variable when graphically displaying its values In the description of each CDF toolkit program the special attributes that may affect that program s operation are defined Note that most of the CDF toolkit programs can be instructed to ignore these special attributes 3 1 6 Special Qualifier There is a special qualifier applied to all toolkit programs This qualifier as about on all platforms except Macintosh will show version release and increment information of the distribution that the toolkit program is based on This special qualifier if present supersedes all other qualifiers and parameters 3 2 CDFedit 3 2 1 Introduction The CDFedit program allows the display and or modification of practically all of the contents of a CDF by way of a text mode full screen interface It is also possib
79. H CREATE ERROR SCRATCH DELETE ERROR SCRATCH READ ERROR SCRATCH WRITE ERROR SINGLE FILE FORMAT SOME ALREADY ALLOCATED TOO MANY PARMS TOO MANY VARS UNKNOWN COMPRESSION UNKNOWN SPARSENESS This CDF contains no rVariables The operation performed is not applicable to a CDF with no rVariables Informational Write access is not allowed on the CDF file s Make sure that the CDF file s have the proper file system privileges and ownership Error Named CDF is corrupted or not actually a CDF This can also occur if an older CDF distribution is being used to read a CDF created by a more recent CDF distribution Contact CDF User Support if you are sure that the specified file is a CDF that should be readable by the CDF distribution being used CDF is backward compatible but not forward compatible Error Because of the type of variable records preceding the range of records being allocated were automatically allocated as well Informational Your CDF distribution has been built to allow only read access to CDFs Check with your system manager if you require write access Error The CDF is in read only mode modifications are not allowed Error Cannot create a scratch file error from file system If a scratch directory has been specified ensure that it is writable Error Cannot delete a scratch file error from file system Error Cannot read from a scratch file error from file system Error Cannot
80. IDMAX CDF REAL4 NRV values follow alee 1 40 0 1 2 J 30 0 Number Elements Number Elements 180 0 180 0 Number Elements 23 Record Variance Record Variance Record Variance Dimension Variances Dimension Variances Dimension Variances Variable Name Temperature Attribute Name VALIDMIN VALIDMAX fend Data Type CDF REAL4 Data Type CDF REAL4 CDF REAL4 Number Record Elements Variance 1 T Value 40 0 500 A 24 Dimension Variances Chapter 2 2 Concepts 2 1 CDF Library The CDF library is the only way to access a CDF Various properties of the CDF library are described in the following sections 2 1 1 Interfaces Two types of interfaces to the CDF core library exist for C and Fortran programs the Standard Interface and the Internal Interface The Standard Interface is easier to use and requires a much shorter learning curve than the Internal Interface but it s not as efficient as the Internal Interface The Standard Interface is recommended if you are not familiar with the Internal Interface These interfaces are described in the following sections The CDF Java Interface is described in the CDF Java Reference Manual Standard Interface There are two types of the Standard Interfaces Original and New The Original Standard Interface was introduced in early 90 s and they only provide a very limited fu
81. Illegal variable number specified Variable numbers must be zero 0 or greater for C applications and one 1 or greater for Fortran applications Error Illegal zMode specified The CDF zModes are defined in cdf h for C applications and in cdf inc for Fortran applications Error Records cannot be allocated for the given type of variable e g a compressed variable Error Because of dependencies on the value it cannot be changed Some possible causes of this error follow 1 Changing a CDF s data encoding after a variable value including a pad value or an attribute entry has been written 2 Changing a CDF s format after a variable has been created or if a compressed single file CDF 3 Changing a CDF s variable majority after a variable value excluding a pad value has been written 4 Changing a variable s data specification after a value including the pad value has been written to that variable or after records have been allocated for that variable 5 Changing a variable s record variance after a value excluding the pad value has been written to that variable or after records have been allocated for that variable 6 Changing a variable s dimension variances after a value excluding the pad value has been written to that variable or after records have been allocated for that variable 130 CANNOT COMPRESS CANNOT SPARSEARRAYS CANNOT SPARSERECORDS CDF CLOSE ERROR CDF CREATE ERROR CDF DELETE
82. Likewise a sequential write will store a value at the current sequential value and then increment the current sequential value to the next value Sequential reads are allowed until the end of the physical records has been reached not the end of the virtual records they never end Sequential reading will increment to the beginning of the next physical record if necessary Sequential writing can be used to extend the physical records for a variable as well as to overwrite existing values If the variable has sparse records the virtual records in a gap of missing records are not skipped The type of sparse records see Section 2 3 12 will determine the values returned When a virtual record in a gap of missing records is read the informational status code VIRTUAL RECORD DATA is returned rather than END OF VARIABLE Sequential writes will create any necessary record in a gap of missing records i e sequential writes do not skip virtual records in a gap of missing records Example Fortran application Assume a 2 dimensional array with sizes 2 3 column majority a record variance of VARY dimension variances of VARY VARY nine 9 physical records written and that the current sequential value has been set to record number 7 and indices 2 2 Consecutive sequential reads would cause the following values to be read and returned to the application 7 2 2 7 1 3 7 2 3 8 1 1 8 2 1 8 1 2 8 2 2 8 1 3 8 2 3 9 1 1 9 2 1 9 1 2 9 2 2
83. MODIFIES OR REDISTRIBUTES THE SUBJECT SOFTWARE AS DEFINED HEREIN OR ANY PART THEREOF IS BY THAT ACTION ACCEPTING IN FULL THE RESPONSIBILITIES AND OBLIGATIONS CONTAINED IN THIS AGREEMENT Government Agency National Aeronautics and Space Administration NASA Government Agency Original Software Designation GSC 14272 Government Agency Original Software Title Coordinated Data Analysis workshop Web CDA Web User Registration Requested Please email the following person Government Agency Point of Contact for Original Software Robert E McGuire nasa gov 1 DEFINITIONS A Contributor means Government Agency as the developer of the Original Software and any entity that makes a Modification B Covered Patents mean patent claims licensable by a Contributor that are necessarily infringed by the use or sale of its Modification alone or when combined with the Subject Software C Display means the showing of a copy of the Subject Software either directly or by means of an image or any other device D Distribution means conveyance or transfer of the Subject Software regardless of means to another E Larger Work means computer software that combines Subject Software or portions thereof with software separate from the Subject Software that is not governed by the terms of this Agreement F Modification means any alteration of including addition to or deletion from the substance or structure of either the Original Software or S
84. MS no format UNIX Specifies whether or not the FORMAT attribute is used when writing variable values if the FORMAT attribute exists and an entry exists for the variable NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and 1 CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the f
85. N gt gt lt out cdf gt gt FILE lt filename gt UNIX including Mac OS X Windows o cdfmerge noprefix prefixes lt prefixl gt lt prefix2 gt lt prefixN gt no log no dataonly about lt lt cdf path1 gt lt cdf path2 gt lt cdf pathN gt lt out cdf gt gt file filename Parameter s lt cdf path1 gt lt cdf path2 gt lt cdf pathN gt lt out cdf gt The pathnames of the input CDF files to be merged Two input files must be specified at a minimum The pathname of the output CDF This is the last one in the parameter list Any rVariables in the source files will be converted to zVariables The first source file dictates how the merged file is to be created its majority row column encoding format single multiple and compression are used for the output file FILE lt file name gt VMS file lt file name gt Unix Windows Qualifier s Specifies the name of the file that contains the names of the source CDFs and the output merged file This option provides an alternative way of entering a long list of files at the command line Each input file name and the output file name should be specified on a separate line User defined prefix if desired should be added at the end of the input file name separated by one or more blank spaces Suppose you want to merge five files a cdf b cdf c cdf d cdf and e cdf into a file called merged_ab cdf and provide the name
86. S D IBM RS6000 DEC Alpha OpenVMS DEC Alpha OpenVMS G HP 9000 Mac OS X NeXT Macintosh PPC Network XDR Table 2 5 Equivalent Single Precision Floating Point Encodings Double Precision Floating Point Data Types The double precision floating point encodings on the supported computers are either IEEE 754 floating point Digital s D FLOAT floating point or Digital s G FLOAT floating point There are also two different byte orderings for the computers that use IEEE 754 big Endian and little Endian The double precision floating point encodings for each supported computer are shown in Table 2 6 Network XDR encoding uses IEEE 754 big Endian encoding for double precision floating point data types TEEE 754 Big Endian IEEE 754 Little Endian Sun DECstation SGi Iris PC IBM RS6000 DEC Alpha OSF 1 HP 9000 DEC Alpha OpenVMS I NeXT Mac OS X Macintosh PPC Network XDR Digital s D FLOAT Digital s G FLOAT VAX DEC Alpha OpenVMS G DEC Alpha OpenVMS D Table 2 6 Equivalent Double Precision Floating Point Encodings Performance The best performance when accessing reading or writing a CDF will occur when that CDF is in the host encoding of the computer being used and host decoding is in effect This is because no encoding or decoding has to be performed by the CDF library A CDF that must be portable between two or more different types of computers should normally be network encoded There may be cases h
87. TA gt operations of CDF lib 2 3 17 Hyper Access Hyper access allows more than one value to be read from or written to a variable with a single call to the CDF library In fact the entire variable may be accessed at once if a large enough memory buffer is available to your application Hyper reads cause the CDF library to read from the variable record s in the CDF and place the values into a memory buffer provided by the application Hyper writes cause the CDF library to take values from a memory buffer provided by the application and write them to the variable records in the CDF Six parameters are specified when performing a hyper read write RecordNumber The record number at which to start the access RecordCount The number of records to access RecordInterval The interval between records being accessed An interval of two 2 would indicate that every other record is to be accessed DimensionIndices The indices within each record at which the access should begin DimensionCounts The number of values along each dimension that should be accessed DimensionIntervals For each dimension the interval between values being accessed An interval of three 3 would indicate that every third value is to be accessed For 0 dimensional variables the dimension indices counts and intervals are not applicable A hyper access may or may not read write a contiguous set of values stored for a variable in the CDF However the values in the memory b
88. _ab cdf that contains four variables named pl varl pl var2 p2 varl and p2 var2 cdfmerge prefix pl p2 a cdf b cdf merged_ab cdf If the prefix option is not specified in the above example merged_ab cdf will contain four variables named filel varl filel var2 file2 varl and file2 var2 This option cannot be used with the noprefix option NO DATAONLY VMS no dataonly Unix Windows This option allows to specify to copy the variable data only If this option is selected the merged file will not have a separate variable for each variable in the source CDFs Suppose there are two files to be merged a cdf and b cdf and each file contains two variables varl and var2 The following command will merge these two files into a file called merged_ab cdf that contains two variables named var and var2 cdfmerge a cdf b cdf merged_ab cdf The merged file will have the metadata 1 e global and variable attributes from that of the first source CDF The variable data of the same name are combined in the same sequence as the source CDFs are presented Arrange the source files in a proper sequence if they are sequence sensitive The default is nodataonly NOJLOG VMS noJlog Unix Windows ABOUT VMS about Unix Windows Example s VMS CDFM ERGE Specifies whether or not messages are displayed indicating the progress of CDF merging The default is nolog Shows the library version that was used to create
89. a CDF is closed ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS CDFCOMPARE GISS WETL GISS WETL1 CDFCOMPARE LOG TOLERANCE F DEF D 1 0E 12 NOATTR NUMBER REPORT ERRORS GISS WETL CDF SMPL GISS WETL CDFCOMPARE NOVAR NOETC ZMODES 1 2 NCDS SMPL NCDSSDATA UNIX 85 cdfcompare giss wetl giss wetll cdfcompare log tolerance f def d 1 0e 12 noattr number report errors giss wetl giss wetlx cdfcompare novar noetc zmodes 1 2 user5 CDFs user6 CDFs VMS UNIX Command line help is displayed when CDFcompare is executed without any arguments 3 5 3 Output from the CDFcompare Program The output from CDFcompare consists of messages indicating the differences found If message logging is enabled the progress of each comparison is also displayed 3 6 CDFstats 3 6 1 Introduction The CDFstats program produces a statistical report on a CDF s variable data Both rVariables and zVariables are analyzed For each variable it determines the actual minimum and maximum values in all of the variable records the minimum and maximum values within a valid range of values with illegal fill values being ignored and the variable s monotonicity Monotonicity refers to whether or not a variable s data values increase or decrease from record to record or along a dimension This property is c
90. ables in only the first CDF record and a full 2 dimensional array of values for Temperature in each CDF record The actual physical storage physical view is shown in Table 1 4 The conceptual view of the CDF however is still that of one 2 dimensional array per rVariable in each CDF record as shown in Table 1 2 the physically stored values are shown in boldface type Record rVariables Number Time Longitude Latitude Temperature 40 20 0 19 2 1 0000 165 150 l l 30 21 7 20 7 18 2 22 0 2 0100 l l 19 3 19 2 19 9 19 6 3 0200 l l 19 3 19 0 21 0 18 4 24 2300 l l 19 5 22 0 Table 1 4 Example CDF 2 Dimensional Representation Physical 17 zVariables zVariables are similar to rVariables in all respects except that each zVariable can have a different dimensionality This allows any set of variables to be stored in the same CDF without wasting space or creating confusion in how the variables are logically viewed Consider a data set that consists of some number of images each containing 1024 by 1024 pixels The data set also contains a palette that is used to map pixel values to the actual color shade to be displayed Palettes are also referred to as lookup tables or color lookup tables For this example it assumes that each image pixel is stored in an 8 bit byte and the palette is a 1 dimensional array of 256 colors shades Indexing into the palette array with a pixel value gives the ap
91. ailable and must provide with each copy of the Subject Software information on how to obtain the source code in a reasonable manner on or through a medium customarily used for software exchange Each Recipient must ensure that the following copyright notice appears prominently in the Subject Software Copyright 1996 2013 United States Government as represented by the Administrator of the National Aeronautics and Space Administration All Rights Reserved Each Contributor must characterize its alteration of the Subject Software as a Modification and must identify itself as the originator of its Modification in a manner that reasonably allows subseguent Recipients to identify the originator of the Modification In fulfillment of these requirements Contributor must include a file e g a change log file that describes the alterations made and the date of the alterations identifies Contributor as originator of the alterations and consents to characterization of the alterations as a Modification for example by including a statement that the Modification is derived directly or indirectly from Original Software provided by Government Agency Once consent is granted it may not thereafter be revoked A Contributor may add its own copyright notice to the Subject Software Once a copyright notice has been added to the Subject Software a Recipient may not remove it without the express permission of the Contributor who added the notice A Recipient ma
92. alent 34 host 33 network 34 performance considerations 35 exporting 70 file extension 32 file format 9 31 changing 31 default 31 multi file 32 performance considerations 32 single file 31 filtering 70 leap seconds 61 limits 28 37 listing 70 naming wildcards 64 naming 31 37 trailing blanks 31 opening 30 statistics 86 subsetting 70 TT2000 61 verifying 82 CDFstats 86 executing 87 output 90 CDF validate 106 executing 106 compression 14 algorithms 60 CDF file s 14 37 variable s 14 47 conceptual organization 7 148 data specification 40 attribute entry 57 variable 40 data types 58 character 58 EPOCH 59 EPOCH16 59 equivalent data types 60 floating point 58 integer 58 TT2000 59 decoding CDF 35 definitions file 64 dimensionality variable 39 encoding CDF 33 EPOCH 59 syntax 59 EPOCH 16 59 syntax 59 examples 21 cdfedit 22 conceptual view 16 data set flat 16 physical view 17 skeleton table 22 120 FILLVAL attribute 66 87 FORMAT attribute 66 70 86 format CDF 31 GZIP compression 61 host decoding 36 host encoding 33 Huffman compression 60 hyper access variable 50 IDL CDF s interface 26 IEEE 754 27 34 35 142 indexing variable records 32 initial records 45 interfaces IDL 26 internal 25 standard 25 Internal Interface 25 limits 28 37 attribute name length 37 CDF file name length 37 dimensions
93. allocated to hold a specific number of uncompressed records for that variable The number of records allocated depends on the variable s blocking factor see Section 2 3 12 The staging scratch file is also used when necessary with variables having sparse records If the records being written are not first allocated the staging scratch file will be used to minimize the indexing overhead see Section 2 2 7 by trying to keep consecutive records contiguous in the dotCDF file Compression The compression scratch file is used when writing to a compressed variable in a CDF Because the CDF library does not know how well a block of variable records will compress the compression algorithm first writes the compressed block to the compression scratch file The compressed block is then copied to the dotCDF file Note that when reading a compressed variable a compressed block of records is decompressed directly to the staging scratch file because the CDF library knows the size of the uncompressed block of records Uncompressed dotCDF When overall compression is specified for a CDF the CDF library maintains an uncompressed version of the dotCDF file as a scratch file By default these scratch files are created in the current directory On VMS systems the logical name CDF TMP can be defined with an alternate directory in which to create scratch files On UNIX and MS DOS systems the environment variable CDF TMP would be used An application can also select a
94. alue s The value s could then be sent directly to the client computer by the server computer without a conversion being necessary by either the client or the server The CDF library would perform the necessary conversions 2 If data values were being read from a CDF and written in binary form to a file for use on a different type computer The proper decoding could be selected for the CDF before any of the data values are read No conversions would be necessary by the application program A CDF s decoding may be selected and reselected at any time after the CDF has been opened and as many times as necessary A CDF s decoding is selected via the Internal Interface with the lt SELECT CDF DECODING gt operation Also a CDF s decoding does not affect the values that already exist in a CDF or any values subsequently written A CDF s encoding determines how the values are written to the CDF file s Section 2 2 8 describes a CDF s encoding The supported decodings correspond to the supported encodings They are as follows HOST_DECODING The data representation of the host computer This is the default NETWORK_DECODING The External Data Representation XDR VAX_DECODING VAX and microVAX data representation Double precision floating point values will be in Digital s D FLOAT representation ALPHAVMSd_DECODING DEC Alpha running OpenVMS data representation Double precision floating point values will be in Digital s D FLOAT representation
95. and OpenVMS systems The byte ordering in which the least significant byte LSB is stored in the lowest memory location The order in which the values of a multidimensional array are stored This may be either row major or column major Data about data A CDF stores metadata using attributes and attribute entries The property of a variable that specifies whether or not that variable s values increment or decrement or neither along a dimension or from record to record A CDF format Multi file CDFs consist of one file for control metadata and one file per variable of data 142 multiple variable access network encoding NOVARY NRV variable NSSDC number of elements Object Operation pad value physical record physical value read only record variance reserve percentage eEntry RLE row major RV variable A variable access method in which one full physical record is read written for each of one or more variables The encoding that uses the XDR representation A record dimension variance indicating that the values do not change from record to record or along a dimension Non record variant variable A variable whose values do not change from record to record a record variance of NOVARY National Space Science Data Center For a variable the number of instances of the data type at each value For an attribute entry the number of instances of the data type for that entry When the Int
96. anges the fastest 48 For example an array for an rVariable with VARY VARY dimension variances in a 2 dimensional CDF with dimension sizes 2 4 and row majority would be stored as follows v 1 1 v1 2 v 1 3 vA 2 1 vQ 2 v 2 3 vQ4 where v i j is the value at indices i j If the CDF had column majority the array would be stored as follows v 1 1 v 2 1 v 1 2 v 2 2 v 1 3 v 2 3 v1 4 vQ4 In each case v 1 1 is stored at the low address An application needs to be concerned with the majority of a CDF in the following cases 1 When performing a variable hyper read the values placed in the buffer by the CDF library will be in the variable majority of the CDF The application must process the values according to that majority When performing a variable hyper write the CDF library expects the values in the buffer to be in the variable majority of the CDF The application must place the values into the buffer in that majority 2 When sequential access is used the values are read written in the order imposed by the variable majority of the CDF 3 When single value reads writes are performed the majority could have an effect The CDF library uses a caching scheme to optimize the random access to variable values If all of the values of a record are to be read written there may be an increase in performance if the values are accessed with rather than against the majority For example if the majority is ro
97. ary use a default blocking factor Note that setting the blocking factor too low and not allocating the records being written may result in excessive indexing for a variable Even using the default blocking factor for a variable may result in excessive indexing unless the records to be written are first allocated The indexing scheme used by the CDF library is described in Section 2 2 7 Compressed Variables The blocking factor for compressed variables specifies the number of records that will be compressed together The CDF library stages the records of a compressed variable in a scratch file The number of records in the staging area is also based on the variable s blocking factor When necessary the CDF library compresses the records in the staging area and writes the compressed block of records to the dotCDF file Each block of compressed records has an associated index entry see Section 2 2 7 Setting the blocking factor high will minimize the indexing for a variable but will increase the time needed to access an individual record because the entire block in which it is compressed will have to be decompressed If the blocking factor is too low the decompression of an individual record will not take as long but excessive indexing may result which will increase the access overhead Also most compression algorithms work better as the number of records bytes being compressed is increased Note that if the compressed variable also has sparse records
98. ary will not have been written to the CDF file s An existing CDF that has been opened and only read from should also be closed In a C application CDFs are closed using either the CDFcloseCDF function Extended Standard Interface or the lt CLOSE_ CDF_ gt operation of the CDFlib function Internal Interface In a 2 This also applies to the uncompressed CDF that is maintained as a scratch file 30 Fortran application CDFs are closed using either the CDF close cdf subroutine Extended Standard Interface or the lt CLOSE CDF gt operation of the CDF lib function Internal Interface 2 2 5 Deleting An open CDF may be deleted at any time The dotCDF file is deleted along with any variable files if a multi file CDF Note that if the CDF is corrupted and cannot be opened by the CDF library you will have to delete the CDF file s manually using the capabilities of the operating system being used In a C application CDFs are deleted using either the CDFdeleteCDF function Extended Standard Interface or the lt DELETE CDF gt operation of the CDFlib function Internal Interface In a Fortran application CDFs are deleted using either the CDF delete cdf subroutine Extended Standard Interface or the lt DELETE_ CDF_ gt operation of the CDF lib function Internal Interface 2 2 6 Naming The file name specified when opening or creating a CDF can be any legal file name for the operating system being used This includes logical symbols o
99. as efficient as Internal Interface Standard Interface is recommended if you are not familiar with Internal Interface The Standard and Internal interfaces are callable from both C and Fortran The C and the Fortran interfaces APIs are described in the CDF C Reference manual and the CDF Fortran reference manual respectively The CDF Java APIs are described in the CDF Java Reference Manual The Perl interfaces are described in the Perl to CDF Interfaces document that is included in the CDF Perl distribution package The C interfaces are described in the C to CDF Interfaces document that is included in the C CDF distribution package The C Fortran and Java APIs are part of the standard CDF distribution package but the Perl and C APIs are available as optional packages The Java APIs for the Unix and Linux platforms are also available as an optional package As of this writing the Java APIs are not available for the VMS operating system 1 8 1 Standard Interface 15 SkeletonCDF was previously named CDFskeleton 14 PC running CYGWIN MinGW or Mac OS X can be considered a UNIX box while running the CDF tool programs 20 There are two types of Standard Interfaces Original and Extended The Original Standard Interface was introduced in early 90 s and they only provide a very limited functionality within the CDF library For example it can only handle rVariables not zVariables and has no access to zVariables s attribute entry zEntries
100. as record variant RV NOVARY The values do not change from record to record Only one record is physically written to the variable Each record contains the same values including virtual records beyond the first record Variables of this type are referred to as non record variant NRV Section 2 3 12 describes variable records in more detail A variable s record variance is specified when the variable is created The record variance of an existing variable may be changed only if values have not yet been written to that variable An explicit pad value may have been specified however 2 3 11 Dimension Variance A variable s dimension variances specify whether or not the values change along the corresponding dimension The effects of a dimension variance are defined as follows VARY The values do change along the dimension All of the values for the dimension or all of the subarrays are physically stored NOVARY The values do not change along the dimension Only one value or subarray is physically written for that dimension Each value or subarray along that dimension is the same including virtual values subarrays beyond the first value subarray Figure 2 1 illustrates the effect of dimension variances on a variable with 2 dimensional arrays for a particular record For variable 1 each value in the array is physically stored and therefore unique Because variable does not vary along the second dimension each value along that dim
101. astest The process of encoding a group of bytes into a smaller group of bytes storing the smaller group of bytes and then decoding the smaller group of bytes back to the original group of bytes CDF allows both a CDF and or individual variables to be compressed when stored The way that values along a dimension having a variance of NOVARY are made to appear as if they do actually exist only one value is actually physically stored This also applies to records beyond the last record actually 140 Current data specification data type Decoding DLLCDF DLL dimension variance Dimensionality dotCDF file Encoding Entry error code Format full physical record gAttribute gEntry global scope GZIP host decoding stored The conceptual view of a variable consists of virtual records and values in addition to the physical records and values actually stored When the Internal Interface is used current objects states are those items affected when an operation is performed For example a current CDF is selected and then any operation performed involving a CDF is performed on that CDF until a different current CDF is selected For a variable or attribute entry the data type and number of elements of that data For a variable or attribute entry the type of data being stored e g integer floating point character The integer floating point representation of data values passed to an application
102. ata integrity verification through the checksum failed Error The checksum is not allowed for old versioned files Error An error occurred while compressing a CDF or block of variable records This is an internal error in the CDF library Contact CDF User Support Error This Version 2 CDF is corrupted An error has been detected in the CDF s control information If the CDF file s are known to be valid please contact CDF User Support Error An error occurred while decompressing a CDF or block of variable records The most likely cause is a corrupted dotCDF file Error For a compressed variable a block of records did not compress to smaller than their uncompressed size They have been stored uncompressed This can result if the blocking factor is set too low or if the characteristics of the data are such that the compression algorithm chosen is unsuitable Informational The compressed CDF being opened is empty This will result if a program which was creating modifying the CDF abnormally terminated Error The sequential access current value is at the end of the variable Reading beyond the end of the last physical value for a variable is not allowed when performing sequential access Error A specified parameter was forced to an acceptable value rather than an error being returned Warning An operation involving a buffer greater than 64k bytes in size has been specified for PCs running 16 bit DOS Windows 3
103. be best to think of the CDF as being closed and reopened in that zMode The numbering of variable entries may or may not be as you would expect and the scheme used could change in a future release of CDF Most applications will simply select a zMode immediately after opening a CDF zMode being off is the default if a zMode is not selected NOTE Using zMode does not change the contents of a CDF A CDF containing rVariables will appear to contain only zVariables when in zMode If the same CDF is then opened without using zMode the rVariables will still exist zMode is only applicable to rVariables It is strongly suggested that zVariables do not use any false dimensional variances 0 0 to 0 0 Mode The floating point value 0 0 is legal on those computers which use the IEEE 754 floating point representation e g UNIX based computers the Macintosh and the PC but is illegal on VAXes and DEC Alphas running OpenVMS Attempting to use 0 0 results in a reserved operand fault on a VAX and a high performance arithmetic fault on a DEC Alpha running OpenVMS Because of this the CDF library can be told to convert 0 0 to 0 0 when read from or written to a CDF When reading from a CDF the values physically stored in the CDF are not modified only the values returned to an application are converted When writing to a CDF the values physically stored are modified 0 0 is converted to 0 0 before being written to the CDF This mode is available on all suppo
104. ble For a variable having sparse records only those records written by an application are physically stored Unwritten records are virtual as described in Sparse Records on 48 43 Also when one or more values are written to a new physical record the entire record is physically written with the pad value for the variable being used for the unspecified values if any The remaining values in the record may or may not be subseguently written Variable pad values are described in Section 2 3 20 Numbering The record numbers in a CDF are numbered starting at one 1 for Fortran applications and starting at zero 0 for C applications Sparse Records A variable in a single file CDF can be specified as having sparse records If so then only those records that are explicitly written to the variable will be physically stored If a variable is not specified as having sparse records then all of the records up to the maximum written will be physically stored Sparse records are only allowed in single file CDFs where the indexing scheme used for variable records makes this possible Considerable disk space can be saved in the dotCDF file for a variable that has gaps of missing data if that variable is specified as having sparse records For an uncompressed variable having sparse records it is also beneficial if the blocks of records that are going to be written can first be allocated This will allow the CDF library to optimize the indexing for th
105. ble might appear conceptually along with which records are physically stored Note that only three records are physically stored but that nine records appear in the conceptual view of the variable Sparse records are specified for a variable using the lt PUT_ t zVAR_SPARSERECORDS_ gt operation of the Internal Interface One of the following types of sparse records must be specified NO_SPARSERECORDS The variable does not have sparse records PAD_SPARSERECORDS The variable has pad missing sparse records The notation sRecords PAD is used by the CDF toolkit for pad missing sparse records PREV_SPARSERECORDS The variable has previous missing sparse records The notation sRecords PREV is used by the CDF toolkit for previous missing sparse records 3 Sparse records are not allowed for a variable in a multi file CDF 44 Record Temperature 1 101 4 Physical 2 101 4 Virtual 3 101 5 Physical 4 101 5 Virtual 5 101 5 Virtual 6 101 5 Virtual 7 101 5 Virtual 8 101 6 Physical 9 101 6 Virtual Table 2 7 Previous missing Sparse Records Example Conceptual View vs Physical Storage The lt GET_ t zVAR_SPARSERECORDS_ gt operation can be used to inquire the type of sparse records Allocated Records The Internal Interface may be used to allocate records for an uncompressed variable in a single file CDF Normally the number of records allocated would be the number that are to be written assuming this can be de
106. ble s Maximum field FILLVAL Used as the initial value in a variable s FillValue field MONOTON Used as the initial setting in a variable s Monotonicity field These fields are described in the online help for the appropriate menu The values of these fields can be changed at any time The special attributes are simply used to provide initial values Note also that the usage of these special attributes can be controlled by the options selected with the initial qualifier 3 3 3 Executing the CDFexport Program Usage VMS 70 CDFEXPORT INITIAL lt options gt NO PROMPT ZMODE lt mode gt REPORT lt types gt NO STATISTICS NO NEG2POSFPO CACHE lt sizes gt NO SIMPLE BATCH lt mode gt CDF lt path gt INCLUDE EXCLUDE lt vars gt lt varsfile file gt EPOCHRANGE RECORDRANGE lt ranges gt CONTROLSETTINGS lt cdf spec gt TEXT lt path gt SETTINGS lt path gt ABOUT UNIX including Mac OS X o cdfexport initial lt options gt no prompt zmode lt mode gt report lt types gt no statistics no neg2posfp0 cache lt sizes gt no simple batch lt mode gt cdf lt path gt text lt path gt settings lt path gt about include exclude lt vars gt lt varsfile file gt epochrange recordrange lt ranges gt control
107. by CDF toolkit programs and user applications e g entries of a vAttribute should correspond to variables The CDF library also places some restrictions on the operations that may be performed on an attribute of a particular scope These restrictions consist of the following 1 A gEntry operation may not be performed on a vAttribute 2 A zEntry or rEntry operation may not be performed on a gAttribute 3 While in zMode only zEntry operations may be performed on vAttributes see Section 2 1 2 All other operations involving attributes and their entries remain available Assumed Scopes CDF Version 1 did not allow the scope of an attribute to be explicitly declared This led to ambiguities in the interpretation of attribute entries in the toolkit programs and user applications CDF Version 2 does allow the scope of an attribute to be declared when the attribute is created To ease the transition from Version 1 to Version 2 CDF distributions prior to CDF V2 5 contained the notion of assumed attribute scopes Assumed attribute scopes arose when the CDF library had to guess the scope of an attribute in a Version 1 CDF e g when the CDFconvert program converted a Version 1 CDF to a Version 2 CDF Beginning with CDF V2 5 all assumed attribute scopes are converted to the corresponding definite scope When a CDF is read this conversion occurs only in the CDF library the CDF is not physically altered When an existing CDF is written to each as
108. by the CDF library as they are read from a CDF This is independent of the way the data values are physically stored in the CDF The dynamic CDF library for Windows NT 95 98 systems The property of a variable that specifies whether or not the values along a dimension change or stay the same The number of dimensions and the dimension sizes for the rVariables or a zVariable A file having an extension of cdf or CDF if the operating system being used prefers uppercase For a single file CDF this will be the only file For a multi file CDF this file will exist along with zero or more variable files depending on the number of variables in the CDF The integer floating point representation of the data values physically stored in a CDF A CDF object in which metadata is stored An entry is associated with an attribute A status code indicating that a fatal condition was encountered The operation was aborted In reference to a CDF the way in which files are used to store the CDF s control data metadata This may be single file or multi file A variable record consisting of values exactly as physically stored in the CDF A global scoped attribute An entry for a gAttribute Global scope indicates that an attribute describes some property of the entire CDF The Gnu ZIP compression algorithm The decoding of the computer currently being used 141 host encoding HUFF hyper access IDL Interface IEEE 754 include
109. cache buffers being used In a C application rVariables are closed using either the CDFvarClose function Standard Interface or the lt CLOSE rVAR gt operation of the CDFlib function Internal Interface zVariables are closed using the lt CLOSE_ zVAR_ gt operation of the CDFlib function Internal Interface In a Fortran application rVariables are closed using either the CDF var close subroutine Standard Interface or the lt CLOSE TVAR gt operation of the CDF lib function Internal Interface zVariables are closed using the lt CLOSE ZVAR gt operation of the CDF lib function Internal Interface The closing of variables does not apply to single file CDFs since individual files do not exist for each variable 2 3 5 Naming Each variable in a CDF has a unique name This applies to rVariables and zVariables together i e an rVariable cannot have the same name as a zVariable Variable names are case sensitive regardless of the operating system being used and may consist of up to CDF VAR NAME LEN or CDF VAR NAME LEN256 printable characters including blanks Trailing blanks however are ignored when the CDF library compares variable names LAT and LAT are considered to be the same name so they cannot both exist in the same CDF This was done because Version 1 of CDF padded variable names on the right with blanks out to eight characters When a Version 1 CDF was converted to a Version 2 CDF these trailing blanks remained in the variab
110. ce but is set to the default encoding for your CDF distribution when created using the Internal Interface or the Extended Standard Interface The encoding of an existing CDF may be changed with the Internal Interface or the Extended Standard Interface if no variable values or attribute entries have been written variables and attributes may exist however If the SkeletonCDF toolkit program is used to create a CDF the encoding is specified in the skeleton table The encoding specified when creating modifying a CDF may be any of the native encodings for the computers supported by CDF in addition to network XDR encoding A CDF with any supported encoding is also readable on any computer supported by CDF Host Encodings Host encoding HOST_ENCODING is the default and it specifies that variable and attribute entry data values be written to the CDF in the native encoding of the computer being used In addition the following explicit host encodings are supported VAX_ENCODING VAX and microVAX computers Double precision floating point values are encoded in Digital s D FLOAT representation ALPHAVMSd ENCODING DEC Alpha computers running OpenVMS Double precision floating point values are encoded in Digital s D FLOAT representation ALPHAVMSg ENCODING DEC Alpha computers running OpenVMS Double precision floating point values are encoded in Digital s G FLOAT representation ALPHAVMSi_ENCODING DEC Alpha computers running OpenVMS Double precisi
111. d The compression for a variable is specified with the lt PUT 1 zZ VAR COMPRESSION gt operation of the internal interface A variable s compression may be inquired with the lt GET r zZ VAR COMPRESSION gt operation Section 2 6 describes the available compression algorithms Reserve Percentage If a value in a compressed block of records is changed the amount of compression achieved for that block may also change If it increases the block of compressed records may have to be moved in the dotCDF file This will most likely result in the dotCDF file increasing in size if the block of compressed records is placed at the end leaving a block of unused bytes where the compressed block of records previously existed This is not a desirable situation considering that the variable compression is supposed to make the CDF smaller To alleviate this potential problem a reserve percentage may be selected for a compressed variable When a compressed block of variable records is initially written to the dotCDF file some additional space will be allocated This will allow that block of compressed records to expand in size if necessary The reserve percentage is interpreted as follows 0 No reserve space is allocated This is the default 1 100 Allocates that percentage of the uncompressed size of the block of variable records as a minimum For example if a 1000 byte block of records compressed down to 600 bytes and the reserve percentage is 70 t
112. d provides a very limited functionality and its use is not recommended for users who are creating new CDF files If you have a CDF file that contains only rVariables very old you can either use the Original Standard Interface or Internal Interface 2 2 2 Creating A CDF must be created by the CDF library In a C application CDFs are created using either the CDFcreateCDF function in Extended Standard Interface or the lt CREATE_ CDF_ gt operation of the CDFlib function Internal Interface In a Fortran application CDFs are created using either the CDF_create_cdf subroutine in Extended Standard Interface or the lt CREATE_ CDF gt operation of the CDF_lib function Internal Interface 2 2 3 Opening An application must open an existing CDF before access to that CDF is allowed by the CDF library In a C application CDFs are opened using either the CDFopenCDF function Extended Standard Interface or the lt OPEN_ CDF_ gt operation of the CDFlib function Internal Interface In a Fortran application CDFs are opened using either the CDF_open_cdf subroutine Extended Standard Interface or the lt OPEN_ CDF_ gt operation of the CDF_lib function Internal Interface 2 2 4 Closing It is absolutely essential that a CDF that has been created or modified by an application be closed before the program exits If the CDF is not closed it will in most cases be corrupted and unreadable This is because the cache buffers maintained by the CDF libr
113. dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of NOVARY false are removed NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS NO PAGE VMS no page UNIX Specifies whether or not the output is displayed a page at a time A prompt for the RETURN key will be issued after each page A page is generally 22 lines of output NO LOCATION VMS no location UNIX Specifies whether or not the locations of variable value differences are displayed The locations are displayed in the form lt record number gt lt index 1 gt lt index2 gt lt indexN gt NO VALUE VMS no value UNIX Specifies whether or not the values are displayed when a difference is detected between variable values or attribute entries Note that for variable values to be displayed the display of the locations of the differences must also be enabled REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g
114. e 1 Error Unknown data encoding specified The CDF encodings are defined in cdf h for C applications and in cdf inc for Fortran applications Error Illegal attribute entry number specified Entry numbers must be at least zero 0 for C applications and at least one 1 for Fortran applications Error The specified function or item is illegal Check that the proper number of arguments are specified for each operation being performed Also make sure that NULL is specified as the last operation Error Unknown format specified The CDF formats are defined in cdf h for C applications and in cdf inc for Fortran applications Error An illegal number of records to initially write has been specified The number of initial records must be at least one 1 Error Unknown variable majority specified The CDF variable majorities are defined in cdf h for C applications and in cdf inc for Fortran applications Error Unable to allocate dynamic memory system limit reached Contact CDF User Support if this error occurs Error An illegal 0 0 to 0 0 mode was specified The 0 0 to 0 0 modes are defined in cdf h for C applications and in cdf inc for Fortran applications Error The number of dimensions specified is out of the allowed range Zero 0 through CDF MAX DIMS dimensions are allowed If more are needed contact CDF User Support Error The number of elements of the data type is illegal The number of elements must b
115. e CDF library Standard Interface functions CDFerror for C and CDF_error for Fortran can be used within a program to inquire the explanation text for a given status code The Internal Interface can also be used to inquire explanation text There are three classes of status codes informational warning and error The purpose of each is as follows Informational Indicates success but provides some additional information that may be of interest to an application Warning Indicates that the function completed but possibly not as expected Error Indicates that a fatal error occurred and the function aborted Status codes fall into classes as follows Error codes lt CDF WARN lt Warning codes lt CDF OK lt Informational codes CDF OK indicates an unqualified success it should be the most commonly returned status code CDF WARN is simply used to distinguish between warning and error status codes C 2 Status Codes and Messages The following list contains an explanation for each possible status code Whether a particular status code is considered informational a warning or an error is also indicated ATTR_EXISTS Named attribute already exists cannot create or rename Each attribute in a CDF must have a unique name Note that trailing blanks are ignored by the CDF library when comparing attribute names Error ATTR_NAME_TRUNC Attribute name truncated to CDF ATTR NAME LEN characters The attribute was created but with a truncated na
116. e CDF or variable being compressed will also be necessary The following sections describe each compression algorithm any associated parameters and the types of data for which they are appropriate 2 6 1 Run Length Encoding The run length encoding compression algorithm RLE COMPRESSION takes advantage of repeating bytes in the data Currently only the run length encoding of zeros 0 s is supported RLE_COMPRESSION has one parameter that must be set to RLE_OF_ZEROs The notation RLE 0 is used for this type of RLE compression 2 6 2 Huffman 60 The Huffman compression algorithm HUFF_COMPRESSION takes advantage of the frequency at which certain byte values occur in the data A sequence of bytes that contain a high percentage of a limited number of byte values will compress better than if each byte value occurs with equal probability HUFF_COMPRESSION has one parameter that must be set to OPTIMAL_ENCODING_TREES The notation HUFF 0 is used for this type of HUFF compression 2 6 3 Adaptive Huffman The adaptive Huffman compression algorithm AHUFF_COMPRESSION also takes advantage of the frequency at which certain byte values occur in the data AHUFF_COMPRESSION is very similar to HUFF_COMPRESSION and generally provides slightly better compression AHUFF_COMPRESSION has one parameter that must be set to OPTIMAL ENCODING TREES The notation AHUFF 0 is used for this type of AHUFF compression 2 6 4 GZIP The Gnu ZIP compression algorith
117. e UNIX systems it may be necessary to execute stty tab3 before running CDFedit or CDFexport 11 Some of the toolkit programs have a paging qualifier Paging is not allowed if the output of the program has been directed to a file 12 Most toolkit programs have an about qualifier that can be used to determine the CDF distribution from which the program came On the Macintosh an about selection is available on the apple pull down menu In the following sections the available qualifiers and options for each of the toolkit programs will be presented The default settings for these qualifiers and options will not be shown since they can be configured for a particular CDF distribution Use CDFinquire to determine these defaults On VMS OpenVMS systems you should have executed the command procedure named DEFINITIONS COM before running any of the CDF toolkit programs This will define the necessary logical names and symbols Your system administrator knows the location of DEFINITIONS COM On UNIX systems you should have source d or equivalent the script file named definitions lt shell type gt file located in the lt cdf_install_dir gt bin directory where lt shell type gt is the type of shell you are using C for the C shell csh and tesh K for the Korn ksh BASH and POSIX shells and B for the Bourne shell sh This will define the necessary environment variables and aliases Your system administrator knows the location of definitio
118. e at least one 1 For variables with a non character data type the number of elements must always be one 1 Error Illegal number of variables in a record access operation Error Illegal read only mode specified The CDF read only modes are defined in cdf h for C applications and in cdf inc for Fortran applications Error Illegal record count specified A record count must be at least one 1 Error 129 BAD REC INTERVAL BAD REC NUM BAD SCOPE BAD SCRATCH DIR BAD SPARSEARRAYS PARM BAD VAR NAME BAD VAR NUM BAD zZMODE CANNOT ALLOCATE RECORDS CANNOT CHANGE Illegal record interval specified A record interval must be at least one 1 Error Record number is out of range Record numbers must be at least zero 0 for C applications and at least one 1 for Fortran applications Note that a valid value must be specified regardless of the record variance Error Unknown attribute scope specified The attribute scopes are defined in cdf h for C applications and in cdf inc for Fortran applications Error An illegal scratch directory was specified The scratch directory must be writeable and accessible if a relative path was specified from the directory in which the application has been executed Error An illegal sparse arrays parameter was specified Error Illegal variable name specified Variable names must contain at least one character and each character must be printable Error
119. e g 1 2 3 4 If the elements will not all fit on one line they may be continued on additional lines For example 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 10 0 Note that an individual element value may not be split across lines The format of a value for the CDF_EPOCH data type which is also considered a non character data type is defined in Section 2 5 4 A CDF_EPOCH value may not be split across two lines Several example gAttribute definitions follow GLOBALattributes Attribute Entry Data Name Number Type Value TITLEa 1 CDF_CHAR CDAW 9A SABRE TITLEb 1 CDF_CHAR CDAW 9A SABRE Backscatter Radar 20s y History 1 CDF_CHAR CDF created 02 Jan 1961 2 CDF modified 23 Oct 1964 TIMES 1 CDF_EPOCH 04 Jul 1976 12 00 00 000 31 Oct 1976 00 00 00 000 gt 2 25 Dec 1976 01 10 00 000 01 Jan 1977 01 10 30 000 amp Factors amp 1 CDF_REAL4 12 5 2 174 3 8 5 4 7 5 12 A 4 vAttributes Section The vAttributes section contains the names of the vAttributes in the CDF Any rEntries or zEntries for these vAttributes are defined in the rVariables zVariables sections following the definition of the corresponding variable The format of the vAttributes section is as follows V ARIABLEattributes 113 lt attribute name gt lt attribute name gt lt attribute name gt lt attribute name gt Where lt attribute name gt is a vAttribute name delim
120. e path gt vars lt varl var2 varN gt about noheader recordrange lt startrec endrec gt col2row lt cdf path gt Parameter s lt cdf path gt The pathname of the CDF file to be dumped Qualifier s NO FORMAT VMS no format Unix Windows Specifies whether or not the FORMAT attribute is used when displaying or extracting variable values if the FORMAT attribute exists and an entry exists for the variable The default is to use the format DUMP lt option gt VMS dump lt option gt Unix Windows Specifies how the program should produce a dump Valid options are all data metadata The all option the default output includes detailed information about the CDF not just only the metadata and variable data The data option output includes 103 variable data and minimum information about the variable The metadata option output only includes the global attributes and the variable attributes OUTPUT lt file path gt output lt file path gt Unix Windows By default the contents of file is displayed on the screen This option redirects the output to a designated file indicated by lt file path gt If lt file path gt does not have an extension txt is appended at the end of lt file path gt If this qualifier is entered as source then the source CDF pathname is used as its output name with its extension of cdf being replaced by txt V ARS lt var1 var2 varN g
121. e variable Otherwise the CDF library will use the staging scratch file to minimize the indexing needed Note that records cannot be allocated for compressed variables whether or not they have sparse records Two types of sparse records can be specified for a variable They differ only in how unwritten records are presented in the conceptual view of the variable These missing records are considered virtual records just like the records beyond the last record written Pad missing sparse records specifies that when a virtual record is read the variable s pad value should be returned Previous missing sparse records specifies that when a virtual record is read the previous existing record s values should be returned If a previous record does not exist the variable s pad value will be returned Note that previous missing sparse records can also be used to save disk space for a variable if that variable s values do not change from record to record except occasionally If the only records written were those that changed from the previous record then the virtual records following each record actually written physically stored would all have the same value s This could save considerable disk space if the values do not change often For example consider a 0 dimensional variable having previous missing sparse records that is being used to store temperature data Each record corresponds to a temperature reading at a given time Table 2 7 shows how the varia
122. ecesecesseceececseeeaeeeeececseceececaeeesaeceeeecaeseaeeeeeeeeeeeeneeaes 105 3 14 CDFyalidatei in iba cats 106 IN o RON 106 3 14 2 Executing the CDFvalidate Program 6meeenteeeneonneoeneennnnooneeoeneonnneenaeenneeonneeenaenneneene een enne 106 3 15 EDElgapseGondsinfOs5 4 11 45 v0t v vhve sob eve san skisma eka vokk alone EREE ENEE E EEEE EEEE AE EEE E E 107 JASI NON 107 3 15 2 Executing the CDFleapsecondsinfo Program 0 ecceeccessecencecseeeeeeeeeeececeececeseeesaeceeeecaeseaeeeeeeeeeeeeneenaes 107 List of Figures Figure 1 1 Conceptual View of a CDF 0 Dimensional rVariable eeimeeerenteeoneonnenteonneeneenneenneeeeeeneeeeeeee 10 Figure 1 2 Conceptual View of a CDF 2 Dimensional rVariables ooonconcnnncnnnnnnnnonnnnnconconocononnnonn nono non crac nano rnncnn cra ncnnnos 11 Figure 1 3 Conceptual View of a CDF zVariables eseneorententeenreeteereeteereteeeeeeeree nn cra nono 12 Figure1 4 M lti File Ported an EEE a alan 13 Figure SA E AAA AN 13 Figure 2 1 Physical vs Virtual Dimensions 0 eeeeereereeneeeneenaenteonreoteonteeteeeneetneeneneneneeeneeneeeneneenaeeneet 41 Figure 2 2 Physical vs Virtual Records Standard Variable eeseeeeteneeeeneonneoneenteonnenteonneeteonneenneeneeeneeeeeee 43 Figure 3 1 Window Sections CDFedit weerenreeneeentesenaeeneneeneeveneenrene
123. ecord the only record that actually physically Since an application knows how it will be accessing a variable it knows best how to optimize the caching scheme used See Section 2 1 5 for details on how an application can control the CDF library caching scheme 49 exists If the record variance is VARY the values are written to the actual records The physical records are the same as the virtual records The same applies to any dimension variances that are NOVARY When a set of indices is specified for a single value read write the index for a dimension whose variance is NOVARY is forced to the first index regardless of the actual index specified for that dimension see Section 2 3 11 In a C application single value access for rVariables is performed using either the CDFvarGet and CDFvarPut functions Standard Interface or the lt GET rVAR DATA gt and lt PUT rVAR DATA gt operations of the CDFlib function Internal Interface Single value access for zVariables must be performed using the lt GET ZVAR DATA gt and lt PUT zZVAR DATA gt operations of CDFlib In a Fortran application single value access for rVariables is performed using either the CDF var get and CDF var put subroutines Standard Interface or the lt GET rVAR DATA gt and lt PUT rVAR DATA gt operations of the CDF lib function Internal Interface Single value access for zVariables must be performed using the lt GET ZVAR DATA gt and lt PUT zZVAR DA
124. ed error from file system Make sure that sufficient privilege exists to create the dotCDF file in the disk directory location specified and that an open file quota has not already been reached Error Cannot delete the CDF specified error from file system Insufficient privileges exist the delete the CDF file s Error The CDF named already exists cannot create it The CDF library will not overwrite an existing CDF Error An unexpected condition has occurred in the CDF library Report this error to CDFsupport Error CDF file name truncated to CDF PATHNAME LEN characters The CDF was created but with a truncated name Warning Function completed successfully Cannot open the CDF specified error from file system Check that the dotCDF file is not corrupted and that sufficient privilege exists to open it Also check that an open file quota has not already been reached Error 131 CDF READ ERROR CDF WRITE ERROR CHECKSUM ERROR CHECKSUM NOT ALLOWED COMPRESSION ERROR CORRUPTED V2 CDF DECOMPRESSION ERROR DID NOTCOMPRESS EMPTY COMPRESSED CDF END OF VAR FORCED PARAMETER IBM PC OVERFLOW ILLEGAL EPOCH VALUE ILLEGAL FOR SCOPE ILLEGAL IN zZMODE ILLEGAL ON VI CDF Failed to read the CDF file error from file system Check that the dotCDF file is not corrupted Error Failed to write the CDF file error from file system Check that the dotCDF file is not corrupted Error The d
125. ed for the CDF or any of its variables Sparseness Sparse records or arrays for variables are not allowed Allocation Pre allocation of records or blocks of records is not allowed For each variable the maximum written record is the last allocated record Deletion Deletion of a single variable from a CDF is not allowed Only deleting a whole CDF is possible 5 This file referred to as the dotCDF file These features are covered in the following sections Record rVariable rVariable rVariable Number 1 2 n 1 a a 1 2 LJ LJ 1 3 LJ a 1 m LJ Lj 1 Figure 1 1 Conceptual View of a CDF 0 Dimensional rVariable 10 rVariable rVariable Record rVariable Number 1 Ly Hoo 13003 10101013 131003 10101013 131330 Boy 131322 Ly m m 13000 Oooo 1 2 Ly NOOO 1310103 10001013 131003 Ly OOOO 1333 ol Hoo 0000 131300 13302 See ee 13132 LIL 11 11 1111 13300 Moo MoO 13022 13000 NOOO 13302 Ly 3 Figure 1 2 Conceptual View of a CDF 2 Dimensional rVariables 11 zVariable
126. ed over the ones in CDFfsi exe since they are much more intuitive and easier to use To invoke either program do one of the following Double click the CDFToolsDriver jar icon on the Desktop OR Go to the directory where the CDF library is installed and double click the CDFToolsDriver jar program OR Open a Command Prompt session i e CA and type java CDFToolsDriver at the command prompt Users will be presented with a main menu containing two programs CDFedit and CDFexport from which a desired tool can be selected with a single click 3 1 4 How to Invoke the GUI Toolkit for Unix Java version of the CDF toolkit is available starting with CDF 2 7 A desired CDF tool can be invoked by typing java CDFToolsDriver at the system prompt and selecting the tool of interest from the main menu with a single click 65 There are two environment variables that must be set prior to invoking the toolkit program CDFToolsDriver jar CLASSPATH and LD_LIBRARY_PATH Follow the instructions in the README install file located under the lt cdf installed dir gt cdfjava directory 3 1 5 Special Attributes There is a set of vAttributes that have special meaning to some of the CDF toolkit programs Your CDFs do not have to use these special attributes The CDF toolkit programs will function properly whether or not these special attributes are present in a CDF How the entries of each vAttribute are used for the corresponding variables is as follows
127. ed to avoid confusion with the FILLVAL attribute 54 3 Fora variable having the pad missing style of sparse records if a record is read from a gap of missing records pad values will be returned The previous missing style of sparse records would cause the previous existing record s values to be returned unless there is no previous record in which case pad values would be returned 4 When reading a record beyond the last record written for a variable pad values will be returned except if the variable has the previous missing style of sparse records In that case the last written record s values are returned unless there are no written records in which case pad values are returned The pad value for a variable may be specified with the Internal Interface It should be specified before any values are read from or written to the variable otherwise the default pad value will be used The pad value may be changed at any time and any number of times and will be in effect for all subseguent operations The default pad value for each data type are shown in Table 2 8 Data Type Default Pad Value CDF BYTE 127 CDF_INT1 127 CDF UINTI 254 CDF INT2 32767 CDF UINT2 65534 CDF INT4 2147483647 CDF UINT4 4294967294 CDF_INT8 9223372036854775807 CDF_REAL4 1 0E30 CDF_FLOAT 1 0E30 CDF_REAL8 1 0E30 CDF_DOUBLE 1 0E30 CDF_EPOCH 0 0 as 01 Jan 0000 00 00 00 000 CDF EPOCH16 0 0 and 0 0 as 01 Jan 0000 00 00 00 000
128. eeve A rey dhe bap ve taip ev gs Ui eee GES ee 60 2 0 2 gt A RN 60 2 6 3 gt Adaptive HUFTMAN EEE a EEEa R S 6l 2 64 A E E E E E 6l 2 4 AIESEC AO 6l T olkit 873 91311 7 SD A 20117976 18165 1703 1 EES ESU A NN 63 3 1 1 VMS and UNIX including Mac OS X miieeenneeenaeoneneonaeoeneennenoeneeneneonnneenneeneneenneoenaenneneeneeeene enne 63 3 1 2 How to Invoke the GUI Toolkit for Macintosh OS X iietneeteenreenneeneneenneneneenneneeneenene enne 64 3 1 3 How to Invoke the GUI Toolkit for Windows NT 95 98 2000 XP iernesenemenermem 65 3 1 4 How to Invoke the GUI Toolkit for Unix ieneeonneoeneenneneoneoneneonrneenaeeneneonneeeneenneneeneeeene enne 65 3 19 Speelal AAA NOS 66 FLO Special Qualirer e ies vad riik A chee A i Rh ee aa 66 3 2 MEDEL eal dy hed Ri ad a a ae le a i at 66 31231 MIA OAUCUON E sb ste Sete to av Sots a cae dade Dial be Malate i Des an oe ee 66 3 22 Special Attribute Usage iena eis vies ial tas ek ns cena a yay Reais Meee east 67 3 2 3 Executing the CDFedit POMO ae 67 3 24 Interaction with CDFedit renei ao eis anes hil tes ek ne cena ae yay Raa Mae ee 69 333 CDEEXPOT C ri e diverts ed Ath eae bets E E E a TS Aaa 70 3 31 0 Im OAUCUONE rd MS Sik Sects E cae A cele Nh des Bs ESS et ld De RG a Nd E la 70 3332 Special Attribute Usage societal A E aca 70 3 3 3 Executing the CDFexport Program 41 ois asia tac tees ance wa hams Galas Weare 70 3 3
129. eger CDF UINT2 2 byte unsigned integer CDF INT4 4 byte signed integer CDF UINT4 4 byte unsigned integer CDF INT8 8 byte signed integer NOTE When using C on a 64 bit operating system e g DEC Alpha running OSF 1 and Linux 64 bit on Intel keep in mind that a long is 8 bytes and that an int is 4 bytes Use an int with the data types CDF INT4 and CDF UINTA4 rather than a long 2 5 2 Floating Point Data Types CDF_REAL4 amp CDF_FLOAT 4 byte single precision floating point CDF_REAL8 amp CDF_DOUBLE 8 byte double precision floating point A special case exists with respect to the value 0 0 negative floating point zero This value is legal on those computers that use the IEEE 754 floating point representation e g most UNIX based computers and the PC but is illegal on VAXes and DEC Alphas running OpenVMS Attempting to use 0 0 will result in a reserved operand fault on a VAX and a high performance arithmetic fault on a DEC Alpha running OpenVMS A warning is returned whenever 0 0 is read by an application on a VAX or DEC Alpha running OpenVMS The CDF library can be put into a mode where 0 0 will be converted to 0 0 when detected see Section 2 1 2 If 0 0 is not being converted to 0 0 the CDF toolkit programs are designed to display 0 0 in all cases This includes those computers that normally suppress the negative sign 2 5 3 Character Data Types CDF_CHAR 1 byte character CDF_UCHAR 1 byte unsigned character
130. en executing interactively this file name will initially appear at the output CDF prompt If this qualifier is not specified the default CDF name is default in the current directory TEXT lt path gt VMS 74 text lt path gt UNIX Specifies a file name to be used when exporting to a text file listing in batch mode When executing interactively this file name will initially appear at the text file prompt If this gualifier is not specified the default text file name is default lis in the current directory SETTINGS lt path gt VMS settings lt path gt UNIX Specifies a settings file name to be used when executing in batch mode When executing interactively this file name will initially appear at the settings file prompt when saving restoring the current settings The default settings file is simple set if executing in simple mode and export set otherwise with each being in the current directory INCLUDE EXCLUDE lt vars varsfile lt file gt gt VMS include exclude lt vars varsfile lt file gt gt UNIX Includes or excludes only those specified variables in the CDF for export The variables can be specified in vars a comma separated variable list Alternatively if the list gets too long a text file containing the variables one variable per line can be used The text file option is identified by varsfile This a simple and easier way of exporting variables without settings file nor data
131. ension is the same so only one value for that dimension is physically stored the other values are virtual The same is true for variable 3 that does not vary along the first dimension Variable 4 does not vary along either dimension Only one value is physically stored for the array all of the other values are the same they are virtual A variable s dimension variances are specified when the variable is created The dimension variances of an existing variable may be changed only if values have not yet been written to that variable An explicit pad value may have been specified however rVariable 1 rVariable 2 rVariable 3 rVariable 4 VARY VARY VARY NOVARY NOVARY VARY NOVARY NOVARY E EA physical value virtual value Figure 2 1 Physical vs Virtual Dimensions 41 2 3 12 Records A CDF record is a set of variable arrays one per rVariable and one per zVariable in the CDF The variable arrays in a particular record are generally related to each other in some way often time This does not have to be the case and is not enforced by the CDF library in any way A variable record is simply the corresponding variable array within a CDF record Physical variable records are actually stored in the CDF file s Virtual variable records are not actually stored but do exist in the conceptual view of the variable provided by CDF Virtual records can occur in a CDF because of the following reasons 1 Ifa variable
132. er here However when SkeletonCDF reads a skeleton table this value is ignored but a place holder is necessary The number of gAttributes created is determined by the number of definitions in the gAttributes section 110 lt vAttrs gt The number of vAttributes in the CDF SkeletonTable always places the correct number here However when SkeletonCDF reads a skeleton table this value is ignored but a place holder is necessary The number of vAttributes created is determined by the number of definitions in the vAttributes section lt n recs gt The maximum number of rVariable records in the CDF SkeletonTable always places the correct number here However when SkeletonCDF reads a skeleton table this value is ignored but a place holder is necessary The number of records written to the CDF depends on whether or not any values are specified for variables NRV variables are described in Section 2 3 10 lt n dims gt The number of dimensions for the rVariables in the CDF lt dim sizes gt The dimension sizes for the rVariables in the CDF one value per dimension rVariables have zero 0 dimensions this field would be left blank An example header section for a CDF with 2 dimensional rVariables follows header CDF NAME sample2 DATA ENCODING NETWORK MAJORITY ROW FORMAT SINGLE Variables G Attributes V Attributes Records Dims If the rVariables had zero dimensions the header section would be as follows header
133. ernal Interface is used an item that exists and may be accessed manipulated e g a CDF or variable When the Internal Interface is used a function performed on an item e g creating or writing A value written to a variable by the CDF library in those cases where a physical record must be written but not all of its values have been specified by an application For example when a single value is written to a new record all of the other values are written using the pad value A variable record actually stored in a CDF A variable value actually stored in a CDF A mode of the CDF library in which modifications to a CDF are not allowed The property of a variable that specifies whether or not its values change from record to record For a compressed variable the reserve percentage specifies how much additional space to allocate in the dotCDF file when a compressed block of records is initially written A value of 0 zero causes no reserve space to be allocated Values from 1 to 100 cause at least that percentage of the uncompressed size to be allocated Values greater than 100 cause that percentage of the compressed size to be allocated but not exceeding the uncompressed size An entry for a vAttribute corresponding to an rVariable A run length encoding compression algorithm Currently the only type of RLE compression supported is the run length encoding of bytes containing Zero The variable majority where the last index of
134. eta aE ketta Tan aal Tade V i IAEA Ad 60 Example rVariables CDFstats Monotonicity Checking momtnrnenenneneeneneonneeenaeeneneonneoeneeennne na 86 Preface About This Document This document is intended to serve as both a user s guide and reference manual for the Common Data Format CDF As such it provides a primer for introducing the novice reader to the concepts of CDF as well as a reference manual for the advanced user However it does not serve as a cookbook for the proper methods of designing a CDF The very first guestions usually asked by a reader are What is CDF How is CDF used and How is CDF useful for me Although the reader will find the answers to these guestions in this document we provide here a brief description of the conceptual basis of CDF in order to provide a proper perspective when reading the remainder of this document What is CDF CDF in its most basic terms is a conceptual data abstraction for storing manipulating and accessing multidimensional data sets We refer to CDF as a data abstraction because we never discuss the actual physical format in which data sets are stored Instead we describe the form of the data sets and the means interface by which they may be manipulated This important difference from traditional physical file formats is reflected in the orientation of the document toward defining form and function as opposed to a specification of the bits and bytes in an actual
135. ever way makes sense to the user While CDF s variable is a mechanism for storing representing data CDF s attribute is a mechanism for describing the CDF file and the individual CDF variables in the file There are two types of attributes in CDF global attribute and variable attribute Global attribute is used for describing the CDF file and variable attribute is used for describing individual variables Examples of global attributes would include such things as file creation date file author source of data and data set documentation Examples of variable attributes would include such things as a field name for the variable the valid minimum and maximum the units in which the variable data values are stored the format in which the data values are to be displayed and a fill value for errant or missing data 1 4 Features of the CDF Library The CDF library is a flexible and extensible software package that gives the user many options for creating and accessing a CDF The r stands for regular rVariables are the type of variables that CDF has always supported Perhaps traditional would have been a better term gt The z doesn t stand for anything special We just like the letter z This is generally not recommended In those situations where z variables are necessary it is best to use all zVariables than a mixture of rVariables and zVariables 14 1 File Format Options The CDF library gives t
136. f 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when a CDF is closed ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS CDFSTATS TEST1 CDFSTATS REPORT ERRORS GISS SOIL CDFSTATS NOFILL OUTPUT TEMPLATE3 NORANGE CDFSSMPL TEMPLATE3 UNIX e cdfstats giss soil cd stats range fill report errors SCDF SMPL giss soil cdfstats norange output template3 samples template3 oe oe 89 VMS UNIX Command line help is displayed when CDFstats is executed without any arguments 3 6 4 Output from the CDFstats Program The format of the output from CDFstats is as follows For each variable rVariables and zVariables lt number gt lt name gt lt n dims gt lt dim sizes gt lt rec vary gt lt dim varys gt lt data type gt lt n elems gt min lt min value gt min in range lt min value in range gt valid min lt valid min gt lt low values gt low value s max lt max value gt max in range lt max value in range gt valid max lt valid max gt lt high values gt high value s fill value lt fill value gt lt fill values gt fill value s monotonic lt monotonicity gt If range checking and or
137. fill value filtering is disabled the corresponding fields will not be displayed The fields are defined as follows lt number gt lt name gt lt rec vary gt lt dim varys gt lt data type gt lt n elems gt lt n dims gt lt dim sizes gt lt min value gt lt min value in range gt lt valid min gt lt low values gt lt max value gt The variable number The variable name The record variance of the variable either a T or F The dimension variances of the variable for each dimension either a T or F This field is not present if there are zero 0 dimensions The data type of the variable e g CDF REAL4 The number of elements of the variable s data type The number of dimensions of a zVariable This field is not present for an rVariable The dimension sizes of a zVariable This field is not present for an rVariable or if the zVariable has zero 0 dimensions The minimum value found regardless of any range checking performed The minimum value found within the valid range The minimum valid value VALIDMIN attribute entry value The number of values found that are less than the valid minimum The maximum value found regardless of any range checking performed 90 lt max value in range gt The maximum value found with the valid range lt valid max gt The maximun valid value VALIDMAX attribute entry value lt high values gt The number of values found that are greater than the valid max
138. filtering A warning is shown if a variable s specified is not in the CDF Record numbers and indices will be displayed if applicable for exporting to text include and exclude are mutually exclusive This option and settings option are also mutually exclusive As settings is not allowed the default settings if not overridden will be used for the exports This option works in the batch mode EPOCHRANGE RECORDRANGE lt ranges gt VMS epochrange recordrange lt ranges gt UNIX Selects a range of variable data for exporting from a CDF The range can be based on epoch specified by epochrange or record by recordrange ranges should be a pair of values for starting and ending value separated by a comma Epoch includes data types of CDF EPOCH CDF EPOCH16 and CDF TIME TT2000 For the starting and ending epoch they must be one of the following format pairs dd mm yyyy hh mm ss msc for CDF_EPOCH dd mm yyyy hh mm ss mmm uuu nnn ppp for CDF EPOCH16 and yyyy mm ddThh mm ss mmmuuunnn for CDF TIME TT2000 respectively All date fields should be numeric For the record numbers they start from record one 1 If no data is found from the exported variables nothing is outputted epochrange and recordrange are mutually exclusive This option works in the batch mode CONTROLSETTINGS VMS controlsettings UNIX A batch mode option that uses the settings file to control what variables and their order for export
139. function subroutine call CDF OK indicates unqualified success 144 status handler variable file variable scope variance dimension variance record VARY vAttribute virtual record virtual value warning code XDR zEntry zMode zVariable A function subroutine that acts upon a status code received from the CDF library In a multi file CDF these are the files containing the data values for each variable in one file per variable These files are named using the CDF s base name with extensions of v0 v1 and so on for rVariables and z0 z1 and so on for zVariables Variable scope indicates that an attribute describes some property of each variable The property of a variable that specifies whether or not the values along a dimension change or stay the same The property of a variable that specifies whether or not its values change from record to record A record dimension variance indicating that the values change from record to record or along a dimension A variable scoped attribute A variable record that is not actually stored in a CDF but does appear in the conceptual view of the CDF Virtual records would be those records beyond the first record of an NRV variable and those records beyond the last record actually written to an RV variable A variable value this is not actually stored in a CDF but does appear in the conceptual view of the CDF Virtual values would be those value
140. ge for those who wish to develop CDF applications in Perl The CDF library allows developers of CDF based systems to easily create applications that permit users to slice data across multidimensional subspaces access entire structures of data perform subsampling of data and access one data element independently regardless of its relationship to any other data element CDF data sets are portable on any of the CDF supported platforms These currently include VAX OpenVMS and POSIX shell Sun SunOS amp Solaris DECstation ULTRIX DEC Alpha OSF 1 or Tru64 amp OpenVMS Silicon Graphics Iris and Power Series IRIX IBM RS6000 series AIX HP 9000 series HP UX NeXT Mach PC DOS Windows 3 x Windows NT 95 98 2000 XP Linux Cygwin MinGW and Macintosh Mac OS X or Linux If you need to run the CDF library on an operating system that s not mentioned above please contact the CDF support office at gsfc cdf support Olists nasa gov CDF is supported by commercial and open source data analysis visualization software such as IDL MATLAB and IBM s Data Explorer XP For those who are familiar with a language like IDL or MATLAB can easily create sophisticated plots from CDF files instead of writing a lengthy program in C Fortran or Java Compatibility with Previous CDF Releases One of the CDF 3 0 reguirements was an ability to create files bigger than 2G bytes This reguirement necessitated a change in the internal file structure si
141. has zero dimensions would be specified the brackets are still required The value at the given record indices For character data types CDF_CHAR or CDF UCHAR the string must be delimited with a unique character and enclosed in braces in the same manner as for an attribute entry for a character data type For non character data types the value is not enclosed in braces the braces are not necessary because there can only be one element The format for CDF_EPOCH values is described in Section 2 5 4 The vAttribute zEntries are optional If omitted the terminating period is still required The zVariables values are also optional Several sample zVariable definitions follow Variable Name Instrument Attribute Name FIELDNAM Gonkulator Data Number Record Dimension Type Elements Dims Sizes Variance Variances CDF_CHAR 10 0 F Data Type Value CDF_CHAR Measuring instrument 119 Variable Data Number Record Name Type Elements Dims Sizes Variance Miikse tied a edim Ele eat Ticks CDF_BYTE 1 1 3 T Attribute Data Name Type Value no attribute entries 1 1 1 1 2 2 1 3 3 2 1 3 2 2 2 2 3 1 Variable Data Number Record Name Type Elements Dims Sizes Variance A EA A WIND VELOCITY CDF REAL4 1 3 360 180 10 T Attribute Data Name Type Value A sr FIELDNAM CDF_CHAR Wind velocity VALIDMIN CDF_RE
142. hat zMode should be disabled 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of NOVARY false are removed Note that using zMode 1 or zMode 2 on a source CDF that contains rVariables will produce a destination CDF containing only zVariables The zMode view provided for the source CDF is written to the destination CDF during the conversion NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and i 80 CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for
143. he core library provide the easy way to change the checksum option within a file Use CDFedit to interactively modify the checksum mode Or use CDFconvert a command line tool to convert the source CDF file with the checksum to a non checksum destination file CDF 3 1 or earlier doesn t recognize the checksum bit Hence the checksum bit will be ignored How is CDF Useful to Me Hopefully the answers to the first two questions have provided a basis for answering this question If you still have questions or would like to learn more about CDF please refer to the CDF Frequently Asked Questions FAQ page http cdf gsfc nasa gov html FAO htm1 for more detailed information about CDF It is important to understand that CDF has been designed to solve a number of data management and storage problems and has shown itself to be quite flexible in storing a wide variety of data sets NASA Open Soruce Agreement NASA OPEN SOURCE AGREEMENT VERSION 1 3 THIS OPEN SOURCE AGREEMENT AGREEMENT DEFINES THE RIGHTS OF USE REPRODUCTION DISTRIBUTION MODIFICATION AND REDISTRIBUTION OF CERTAIN COMPUTER SOFTWARE ORIGINALLY RELEASED BY THE UNITED STATES GOVERNMENT AS REPRESENTED BY THE GOVERNMENT AGENCY LISTED BELOW GOVERNMENT AGENCY THE UNITED STATES GOVERNMENT AS REPRESENTED BY GOVERNMENT AGENCY IS AN INTENDED THIRD PARTY BENEFICIARY OF ALL SUBSEQUENT DISTRIBUTIONS OR REDISTRIBUTIONS OF THE SUBJECT SOFTWARE ANYONE WHO USES REPRODUCES DISTRIBUTES
144. he basic components of a message typically the asserted bits and storing the resulting value Later anyone can perform the same operation on the data compare the result to the authentic checksum and assuming that the sums match conclude that the message was probably not corrupted The checksum method used by the CDF is the popular MD5 algorithm If the checksum bit is turned on for a CDF file a 16 byte signature message a k a message digest is computed from the entire file and appended to the end of the file when the file is closed after any create write update activities Every time such file is open other than the normal steps for opening a CDF file this signature serving as the authentic checksum is used for file integrity check by comparing it to the re computed checksum from the current file If the checksums match the file s data integrity is verified Otherwise an error message is issued The checksum operation can be applied to CDF single file format files that were created with V2 7 or later Once the checksum is turned on for a particular file the data integrity check of the file is performed every time it is open and a new checksum is computed and stored when it is closed This overhead performance hit may be noticeable for large files Therefore it is strongly encouraged to turn off the checksum once the file integrity is confirmed or verified A couple of the utilities from the CDF toolkit that is distributed along with t
145. he header section contains general information about the CDF The format of the header section is as follows header CDF NAME lt cdf name gt DATA ENCODING lt data encoding gt Variables MAJORITY lt variable majority gt FORMAT lt cdf format gt G Attributes V Attributes Records Dims Size lt rVars gt lt zVars gt lt gAttrs gt lt vAttrs gt lt n recs gt z lt n dims gt lt dim sizes gt The fields are defined as follows lt cdf name gt lt data encoding gt lt variable majority gt lt cdf format gt lt rVars gt lt zVars gt lt gAttrs gt The name of the CDF When SkeletonTable creates a skeleton table this will be the name of the corresponding CDF not the full file name specified When SkeletonCDF reads a skeleton table this will be the name of the CDF created unless a CDF file name is specified on the command line If the CDF name in the skeleton table is to be used a full file name must be specified if desired or else the CDF will be created in the default current directory The data encoding of the CDF When specifying a data encoding to the SkeletonCDF program the following encodings are valid HOST NETWORK VAX ALPHAVMSd ALPHAVMSg ALPHAVMSi SUN SGi DECSTATION ALPHAOSFI IBMRS HP PC MAC and NeXT When a skeleton table is created by SkeletonTable all of the above encodings with the exception of HOST are possible Data encoding is described in Section 2 2 8 The
146. he skeleton table will be created Do not enter an extension Qualifier s SKELETON lt skeleton path gt VMS skeleton lt skeleton path gt UNIX The file name of the skeleton table to be created Do not enter an extension because skt is appended automatically If this qualifier is not specified the skeleton table will be named lt cdf name gt skt in the default current directory where lt cdf name gt is the name portion of the CDF from which the skeleton table was created V ALUES lt values gt NRVTABLE NONRV VMS values lt values gt nrvtable nonrv UNIX Only one of these qualifiers may be specified The meaning of each is as follows V ALUES lt values gt VMS 92 values lt values gt UNIX No values Selected values radio buttons Macintosh Java UNIX amp Windows NT 95 98 VMS UNIX The lt values gt option specifies which variable values should be put in the skeleton table Select one of the options from the list which follows NONRV VMS nonrv UNIX None VMS No values radio button UNIX Off radio button Macintosh Java UNIX amp Windows NT 95 98 No variable values should be put in the skeleton table Nrv VMS NRV values radio button UNIX NRV radio button Macintosh Java UNIX amp Windows NT 95 98 Only NRV variable values should be put in the skeleton table Rv VMS RV values radio button UNIX RV radio button Macintosh Java UNIX amp Windows
147. he underlying CDF library REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and i ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS 97 SKELETONCDE FGGE3B SKELETONCDF NOLOG CDF TEMP FGGE3B_X REPORT ERRORS FGGE3B UNIX oe skeletoncdf fgge3b skeletoncdf nolog cdf fgge3b_x report errors fgge3b oe VMS UNIX Command line help is displayed when SkeletonCDF is executed without any arguments 3 8 3 Creating the Skeleton Table A skeleton table is a text file having skt as a file extension The normal method of creating and using a skeleton table would be to use SkeletonTable on an existing CDF that is similar to the CDF you want to create Then edit the created skeleton table to meet your needs and use SkeletonCDF to create the new CDF The skeleton table could also be created from scratch with any text editor The format of the skeleton table is described in Appendix A 3 9 CDFinguire 3 9 1 Introduction The CDFinguire program displays the version of the CDF distribution being used most configurable
148. he user the option to choose from one of two file formats in which to store the data and metadata The first option is the traditional CDF multi file format This file format is illustrated in Figure 1 4 assuming a CDF containing four variables The example cdf file contains all of the control information and metadata for the CDF In addition to the CDF file a file exists for each variable in the CDF and contains only the data associated with that variable This is illustrated by the files example v0 through example v3 The second option is the single file format the default format when a CDF file is created As illustrated in Figure 1 5 the whole CDF file consists of only a single example cdf file This file contains the control information metadata and the data values for each of the variables in the CDF Both formats allow direct access The advantage of the single file format is that it minimizes the number of files one has to manage and makes it easier to transport CDFs across a network Use of single file format the default format is recommended over the multi file format albeit it slightly increases the data access time The multi file format on the other hand clearly delimits the data from the metadata and is organized in a consistent fashion within the files Updating appending and accessing data are also done with optimum efficiency However the multi file format has the following restrictions Compression Compression is not allow
149. hecked only if the variable varies along just one dimension considering records to be another dimension For example consider a CDF with the 2 dimensional rVariables shown in Table 3 1 rVariable Record Variance Dimension Variances Check Monotonicity EPOCH VARY NOVARY NOVARY Yes LATITUDE NOVARY VARY NOVARY Yes LONGITUDE NOVARY NOVARY VARY Yes ELEVATION NOVARY VARY VARY No TEMPERATURE VARY VARY VARY No Table 3 1 Example rVariables CDF stats Monotonicity Checking The EPOCH LATITUDE and LONGITUDE rVariables would be checked for monotonicity but the ELEVATION and TEMPERATURE rVariables would not be checked 3 6 2 Special Attribute Usage CDFstats uses the following special attributes FORMAT Used when displaying a variable statistic e g minimum variable value VALIDMIN If range checking is enabled used as the minimum valid value for a variable For a variable with a non character data type only the first element of its VALIDMIN attribute entry is used Also if requested the VALIDMIN attribute entry for a variable will be updated with the actual minimum value found Again if the variable has a non character data type the VALIDMIN attribute entry will be updated to have just one element 86 VALIDMAX If range checking is enabled used as the maximum valid value for a variable For a variable with a non character data type only the first element of its VALIDMAX attribute entry is used Also if requested the VALIDMAX at
150. hen 700 bytes would actually be allocated for the block in the dotCDF file If the reserve percentage is 50 then 600 bytes would of course still have to be allocated 101 Allocates that percentage of the size of the compressed block of variable records but not exceeding the uncompressed size For example if a 1000 byte block of records compressed down to 800 bytes and the reserve percentage is 110 then 880 bytes would be allocated for the block Even specifying a reserve percentage for a compressed variable does not guarantee that the problem with moving blocks of compressed records as the variable s values are changed will be avoided If a CDF does become fragmented in this way remember that the CDFconvert utility can always be used to create a new CDF with each variable s compression being optimized e g no fragmentation The reserve percentage for a compressed variable is selected with the lt SELECT r Z VAR RESERVEPERCENT gt operation A variable s reserve percentage may be confirmed with the lt CONFIRM 1 ZVAR RESERVEPERCENT gt operation 2 3 15 Majority The variable majority of a CDF describes how variable values within each variable array record are stored Each variable in a CDF has the same majority The majority can be either row major or column major The default variable majority is row major ROW_MAJOR Row majority The first dimension changes the slowest COLUMN_MAJOR Column majority The first dimension ch
151. her types of variables in a single file CDF the blocking factor can have a significant impact on I O performance The following sections will describe how a variable s blocking factor is used in each case Standard Variables Space in the dotCDF file for records written to a standard variable is either allocated explicitly by an application or automatically by the CDF library If the records are allocated by the application the exact number needed can be specified This can be used to optimize the indexing for the variable resulting in fewer or even just one index entries that must be searched when accessing the variable If the records are not allocated by the application however they must be automatically allocated by the CDF library Because the CDF library wants to optimize the indexing for a variable it may allocate additional records beyond those needed at the time in an attempt to minimize the number of index entries The variable s blocking factor specifies the minimum number of records to allocate when an application writes to an unallocated record This is based on the assumption that the addition records allocated will eventually be written If that is not the case the allocated but unwritten records will simply waste space in the dotCDF file The best way to prevent that situation is for an application to explicitly allocate the records that are going to be written An application can specify a blocking factor for a variable or let the CDF libr
152. hey are self explanatory The Graphical User Interface GUI version of the CDF toolkit written in Java is available starting with CDF 2 7 and a complete set of the toolkit is available for Unix and Macintosh OS X systems The Windows operating system has its own complete set of GUI based toolkit in CDFfsi exe and CDFso exe programs In addition a Java version of CDFedit and CDFexport programs are also included in the Windows distribution package The Java version of CDFedit and CDFexport is recommended over the ones included in CDFfsi exe since they are much more intuitive and easier to use 3 1 1 VMS and UNIX including Mac OS X Each program is executed at the command line or may be executed from within your applications using the methods provided by the operating system being used The following rules apply to the command line syntax 1 Parameters are required unless noted otherwise Parameters are shown in angle brackets lt gt s in the sections that describe each toolkit program 2 Qualifiers are optional unless noted otherwise 3 Qualifiers can be truncated as long as no ambiguities result 4 Optional parts of a command are shown in brackets s in the sections that describe each toolkit program 5 A vertical line 1 is used to separate two or more options in those cases when only one of the options may be specified 63 6 Wildcard characters are allowed in CDF names to allow more than one CDF to be specified where
153. hich were previously only available through the Internal Interface 4 Retrofit of CDF tools to include the option of the new checksum feature while creating a new CDF 5 Retrofit of CDFedit and CDFexport to create a CDF file that is compatible with CDF 2 7 2 or earlier 6 Miscellaneous bug fixes 7 Increase in cache buffer size from 512 to 10240 bytes 8 When read only mode is set all metadata is read into memory where requests for metadata are then directed This improves metadata access performance in most situations 138 Appendix E Glossary AHUFF allocated records Attribute big endian blocking factor Caching CDF The Adaptive Huffman compression algorithm For uncompressed variables in a single file CDF it is possible for an application to allocate records before they are written This has the advantage of reducing the indexing overhead in the dotCDF file which will improve performance when accessing a variable An application would generally then write to the records that were allocated A CDF object with which entries of metadata are associated The byte ordering in which the most significant byte MSB is stored in the lowest memory location For a standard variable in a single file CDF the blocking factor is the minimum number of records actually allocated when a new record is written More records may be allocated than are actually needed in order to keep the variable s records as contiguous as poss
154. hin the tolerance they are considered to be technically egual Either one or both of these two tolerances one for 4 byte single precision floating point data and the other for 8 byte double precision floating point data respectively can be specified If the given tolerance is positive the following formula is used to check their eguality abs value1 value2 gt tolerance If the given tolerance is negative the following formula is applied abs valuel value2 gt abs tolerance max abs value1 abs value2 tolerancel used for the single precision floating point data may be in one of the two forms default or a value Using default indicates that the default value 1 0E 06 is used for the tolerance check for any single precision floating point data Or the specified value is used for the tolerance check This field applies to data types of CDF_REAL4 and CDF_FLOAT def can be used to substitute for default tolerance2 used for the double precision floating point data may be in one of the two forms default or a value Using default indicates that the default value 1 0E 09 is used for the tolerance check for any double precision floating point data Or the specified value is used for the tolerance check This field applies to data types of CDF_REAL8 CDF_DOUBLE and CDF_EPOCH default can be abbreviated as def NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when
155. ible with the assumption that the records will eventually be written For a compressed variable in a single file CDF the blocking factor is the maximum number of records per compressed block For an uncompressed variable having sparse records in a single file CDF the blocking factor is the number of records allocated in the staging scratch file For this type of variable the staging scratch file is used to optimize the indexing in the dotCDF file by storing sequential records contiguously when possible Blocking factors are not applicable to variables in multi file CDFs The method used by the CDF library to improve performance when accessing a file An attempt is made to keep commonly accessed blocks of the file in memory rather than repeatedly reading them from or writing them to disk This term is used in more than one way 1 The actual files that contain your data metadata For example The CDF library must be used to create a CDF 2 The software distribution containing the CDF library include files and toolkit For example We like using CDF to store our data 139 CDF base name CDF distribution CDF library CDF toolkit CDFedit CDFexport CDFstats CDFcompare CDFconvert CDFinquire CDF_OK cdf h cdf inc edfdf inc cdfdvf2 inc cdfdvf3 inc cdfdvf inc cdfmsf inc column major Compression conceptual view The file name of a CDF minus the extension or extensions if a multi file CDF
156. ibute entry is used as its initial format field NOJFILLVAL VMS no fillval UNIX Specifies whether or not a variable s FILLVAL attribute entry is used as its initial fill value field NO VALIDMIN VMS no validmin UNIX Specifies whether or not a variable s VALIDMIN attribute entry is used as its initial minimum filter value NO VALIDMAX VMS no validmax UNIX Specifies whether or not a variable s VALIDMAX attribute entry is used as its initial maximum filter value NO MONOTON VMS no monoton UNIX Specifies whether or not a variable s MONOTON attribute entry is used as its initial monotonicity NOJRECORD VMS no record UNIX Specifies whether or not the Record item will be present NOJINDICES VMS no indices UNIX Specifies whether or not the Indices item will be present NOJEXCLUSIVE VMS noJexclusive UNIX Specifies whether or not exclusive filters are allowed NOJOUTPUT VMS no output UNIX Specifies whether or not each item variable is initially output NOJDELETE VMS no delete UNIX Specifies the initial setting of whether or not an existing CDF will be deleted when a new CDF is created with the same name NO JPREALLOCATE VMS no preallocate UNIX 72 Specifies the initial setting of whether or not variable records are to be preallocated when creating a new CDF SINGLE or MULTI VMS single or multi UNIX Specifies the initial setting of whethe
157. ile types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number of 512 byte buffers to be used Section 2 1 5 explains the caching scheme used by the CDF library NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when a CDF is closed NO SCREEN VMS no screen UNIX Specifies whether or not the skeleton table is to be displayed on the terminal screen written to the standard output If not the skeleton table is written to a text file 94 NO PAGE VMS no page UNIX Specifies whether or not the output is displayed a page at a time A prompt for the RETURN key will be issued after each page A page is generally 22 lines of output ABOUT VMS about UNIX Shows the library version that was used to create this tool program Example s VMS SKELETONTABLE NOLOG REPORT ERRORS FGGE3B SKELETONTABLE SKELETON FGGE3B NONRV FGGE3B SKELETONTABLE SCREEN VALUES Varl Var2 UNIX oe skeletontable nolog report errors fgge3b skeletontable skeleton fgge3b nonrv cdfs fgge3b skeletontable screen values Varl Var2 oe oe VMS UNIX Command line help is displayed when SkeletonTable is executed without any arguments 3 7 4 Output from the SkeletonTable Program The format of the skele
158. iles that may be open at once by an application and the variables values are accessed variable by variable rather than accessing an entire variable before going to the next variable the multi file format may be much 24 As of CDF 2 6 This notation is used when an operation exists for both rVariables and zVariables In this case the actual operations are lt GET_ zVAR_nINDEXLEVELS gt and lt GET_ VAR_nINDEXLEVELS_ gt 32 slower than the single file format This is because the CDF library will have to close one variable file and then open another as each variable value is accessed by the application since the operating system s open file limit will be reached Ifthe application was to access every value for a variable before going on to the next variable this would not occur but it might create complications for the application Note that the format of a CDF can also be converted using the CDFconvert toolkit program which creates a new CDF with the specified format Section 3 4 describes CDFconvert 2 2 8 Encoding The encoding of a CDF determines how attribute entry data and variable data values are stored on disk in the CDF file s An application program never has to concern itself with the encoding of the CDF being accessed The CDF library automatically performs all of the encoding and decoding of data values for the application A CDF s encoding is specified when the CDF is created when using the Original Standard Interfa
159. imum lt fill value gt The fill value FILLVAL attribute entry value lt fill values gt The number of fill values found lt monotonicity gt The monotonicity of the variable The lt monotonicity gt field may take on one of the following values Steady one value Steady all values the same Increase Decrease noDecrease some values the same noIncrease some values the same False n a 3 7 SkeletonTable 3 7 1 Introduction The variable has only one value in the CDF All values of the variable are the same Values strictly increase with increasing record number dimension index Values strictly decrease with increasing record number dimension index Consecutive values either increase or are the same with increasing record number dimension index Consecutive values either decrease or are the same with increasing record number dimension index Consecutive values both increase and decrease The variable was not checked for monotonicity because it varies along more than one dimension if records are considered another dimension The SkeletonTable program is used to create an ASCII text file called a skeleton table containing information about a given CDF SkeletonTable can also be instructed to output the skeleton table to the terminal screen It reads a CDF and writes to the skeleton table the following information 1 Format single or multi file data encoding variable majority 2
160. ion Elements Variance Variances 121 example2 CDF LONGITUD Attribute Name a ie A An FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT 1 1 50 0 2 1 40 0 3 1 30 0 4 1 20 0 5 1 10 0 6 1 0 0 7 1 10 0 8 1 20 0 9 1 30 0 10 1 40 0 11 1 50 0 Variable Name EE ER LATITUDE Attribute Name PESE FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT 1 1 30 0 1 2 20 0 1 3 10 0 1 4 0 0 1 5 10 0 1 6 20 0 1 7 30 0 Variable Name A AREA TEMPERATURE CDF REAL4 Data Type CDF CHAR CDF REAL4 CDF REAL4 CDF REAL4 CDF REAL4 CDF CHAR CDF CHAR Data Type CDF REAL4 Data Type CDF CHAR CDF REAL4 CDF REAL4 CDF REAL4 CDF REAL4 CDF CHAR CDF CHAR Data Type CDF INT4 AS s n SS AN Number Elements A s A SS AS Number Elements Longitude variable 0 0 3 180 0 50 0 50 0 Degrees F8 3 ae mers Record Variance Latitude variable 0 0 90 0 30 0 30 0 Degrees F8 3 te Record Variance 122 3 Dimension Variances 3 Dimension Variances Attribute Name FIELDNAM VALIDMIN VALIDMAX SCALEMIN SCALEMAX UNITS FORMAT AzVariables Variable Name BIAS Attribute Name FIELDNAM VALIDMIN VALIDMAX UNITS FORMAT 34 28 17 ON oe oe oe
161. isplayed during the conversion of that variable Message logging must also be enabled NO DELETE VMS no delete UNIX Specifies whether or not a destination CDF is deleted if it already exists SINGLE MULTI VMS single multi UNIX 78 The format of the destination CDF s This overrides the format of the skeleton CDF if one was specified If neither this qualifier nor a skeleton CDF is specified then the format of a destination CDF will be the same as that of the source CDF ROW COLUMN VMS row column UNIX The variable majority of the destination CDF s This overrides the variable majority of the skeleton CDF if one was specified If neither this qualifier nor a skeleton CDF is specified then the variable majority of a destination CDF will be the same as that of the source CDF ENCODING lt encoding gt HOST NETWORK VMS encoding lt encoding gt host network UNIX Source Host Network Sun Vax radio buttons Macintosh Java UNIX amp Windows NT 95 98 The data encoding of the destination CDF s This overrides the data encoding of the skeleton CDF if one was specified If neither this qualifier nor a skeleton CDF is specified then the data encoding of a destination CDF will be the same as that of the source CDF The possible values of lt encoding gt are host network sun vax decstation sgi ibmpc ibmrs mac hp next alphaosf1 alphavmsd and alphavmsg and their uppe
162. ited with a character not appearing in the name itself e g VALIDMIN or Units The delimiting characters are not part of the vAttribute name in the CDF There may be zero or more vAttribute names There is no limit on the number of attributes that a CDF may have An example vAttributes section follows V ARIABL Eattributes FIELDNAM VALIDMIN Units A 5 rVariable Section The rVariables section contains the definition of each rVariable in the CDF the values for any vAttribute rEntries associated with each rVariable and optionally data values for those rVariables The format of the rVariables section is as follows variables lt variable definition gt lt variable definition gt lt variable definition gt lt variable definition gt Where lt variable definition gt is an rVariable definition The format of each rVariable definition is as follows Variable Data Number Record Dimension Name Type Elements Variance Variances PAE E EA OT lt var name gt lt var data type gt lt n elems gt lt rec vary gt lt dim varys gt Attribute Data Name Type Value beeen 9 55 ee lt attr name gt lt entry data type gt lt entry value gt lt attr name gt lt entry data type gt lt entry value gt lt attr name gt lt entry data type gt lt entry value gt 114 lt attr name gt lt rec num gt lt indices gt lt rec num gt lt indices gt lt rec num gt lt indices gt lt rec num gt
163. j are physical record numbers and dimension indices If the dimension variances had been VARY NOVARY the values placed in the buffer would have been 5 1 1 5 1 1 5 1 1 50 0 5 2 1 5 2 1 5 2 1 5 2 1 6 1 1 6 1 1 6 1 1 6 1 1 6 2 1 6 2 1 6 2 1 6 2 1 If the record count had been 3 and the record interval 2 the values placed in the buffer would have been 54 5 5 1 2 51 3 30 4 5 2 1 5 2 2 5 2 3 5 2 4 74 1 74 2 74 3 70 4 72 1 7 2 2 7 2 3 7 2 4 9 1 1 94 2 91 3 91 4 92 1 9 2 2 9 2 3 9 2 4 If the dimension counts had been 2 2 and the dimension intervals 1 2 the values placed in the buffer would have been 5 1 1 5 1 3 5 2 1 5 2 3 6 1 1 6 1 3 6 2 1 6 2 3 If the CDF majority had been column major the values placed in the buffer would have been 504 5 5 2 1 50 2 5 2 2 5 1 3 5 2 3 530 4 5 2 4 6 1 1 6 2 1 6 1 2 62 2 6 1 3 6 2 3 6 1 4 6 2 4 Had these examples been for hyper writes the CDF library would have expected to find the values in the application s buffer exactly as they were placed there during the corresponding hyper read In the case where the record interval was 2 the records being skipped would be written using the variable s pad value if they did not already exist If they did already exist they would not be affected In a C application hyper writes for rVariables are performed using the CDFvarHyperPut function Standard Interface or the lt PUT rVAR HYPERDATA gt
164. k against The program can show where what an offending field is when such an anomaly is detected 3 14 2 Executing the CDFvalidate Program Usage VMS CDFVALIDATE VALIDATE NOVALIDATE DEBUG ABOUT lt cdf path1 gt lt cdf path2 gt UNIX including Mac OS X Windows cdfvalidate validate novalidate debug about lt cdf pathl gt lt cdf path2 gt Parameter s lt cdf path1 gt The pathnames of the CDF files to be validated 106 lt cdf path2 gt Qualifier s VALIDATE NOVALIDATE gt VMS validatre novalidate Unix Windows Specifies whether or not the file s is to be validated The default is to validate A CDF file will only go through the normal opening process when the novalidate option is specified DEBUG VMS debug Unix Windows Specifies whether or not the debugging information is to be displayed when the data validation is being performed This option is applicable to the previous validate option ABOUT VMS about Unix Windows Shows the library version that was used to create this tool program Example s VMS CDFVALIDATE my data cdf CDFVALIDATE DEBUG filel file2 UNIX and Windows cdfvalidate my data cdf cdfvalidate debug filel file2 Help is displayed when CDFvalidate is executed without any arguments 3 15 CDFleapsecondsinfo 3 15 1 Introduction The CDFlea
165. le names To allow CDF Version 2 applications to read such a CDF without having to be concerned with the trailing blanks the trailing blanks are ignored by the CDF library when comparing variable names The trailing blanks are returned as part of the name however when a variable is inquired by an application program 2 3 6 Numbering The rVariables in a CDF are numbered consecutively starting at one 1 for Fortran applications and starting at zero 0 for C applications Likewise the zVariables in a CDF are numbered consecutively starting at one 1 for Fortran applications and starting at zero 0 for C applications The CDF library assigns variable numbers as the variables are created 2 3 7 Deleting A variable may be deleted from a single file CDF Deleting a variable also causes the deletion of the corresponding attribute entries for the variable The disk space used by the variable definition the variable s data records and the corresponding attribute entries becomes available for use as needed by the CDF library Also the variables that numerically follow the variable being deleted are renumbered immediately Each is decremented by one 2 3 8 Dimensionality 2 It is required that an application close a CDF before exiting Variables may not currently be deleted from a multi file CDF 39 Variable values are stored in arrays A variable s dimensionality refers to the number of dimensions and the dimension sizes of these arra
166. le to run CDFedit in a browse only mode in order to prevent accidental 45 These special attributes originated as part of the NSSDC standard for CDFs The NSSDC standard is no longer used Note that the FILLVAL attribute is not the same as the pad value for a variable although their values will often be the same The pad value is used by the CDF library The FILLVAL attribute is optionally used by a CDF toolkit program or by your applications 66 modifications CDFedit can also be used to create a new CDF file if a CDF does not exist with the provided file path The newly created CDF file can be of either default version V3 or a backward version i e V2 7 If the environment variable CDF FILEBACKWARD on Unix or Windows or CDF FILEBACKWARD on OpenVMS is set to TRUE the new file is then automatically a V2 7 file If this environment variable is not set or set to anything other than TRUE then there is an option to choose for the file version when the program is executed 3 2 2 Special Attribute Usage The special attribute FORMAT is used by CDFedit depending on the setting of the format qualifier when displaying variable values 3 2 3 Executing the CDFedit Program Usage VMS CDFEDIT NO BROWSE ZMODE lt mode gt NO FORMAT NO PROMPT NO NEG2POSFP0 REPORT lt types gt CACHE lt sizes gt NO STATISTICS NO GWITHENTRIES NO VWITHENTRIES ABOUT lt cdf spec gt
167. lues Virtual values do not apply when performing a multiple variable access see Section 2 3 11 Three parameters are specified when performing a multiple variable read write VariableCount The number of rVariables z Variables that are being accessed VariableList The rVariables z Variables being accessed specified by number RecordNumbers The record numbers at which the reads writes will take place For rVariables the record numbers must all be the same For zVariables the record numbers can vary but for most applications will all be the same Multiple variable access is sensitive to the record variances of the variables being accessed Dimension variances do not apply since full physical records are being read written If a variable has a record variance of NOVARY then a read write to that variable will always occur at the first record regardless of the actual record number specified since at most only one physical record will ever exist If the record variance were VARY the reads writes would take place at the actual record numbers specified For a multiple variable write operation an application must place into a memory buffer each of the full physical records to be written The order of the full physical records must correspond to the order of the list of variables specified and the memory buffer must be contiguous there can be no gaps between the full physical records This memory buffer is then passed to the CDF library that sca
168. m GZIP_COMPRESSION uses the Lempel Ziv coding LZ77 taking advantage of common substrings within the data Significant compression occurs over a wide variety of data sets GZIP_COMPRESSION has one parameter which may be set to a level value in the range from one to 9 nine 1 provides the least amount of compression and executes the fastest 9 provides the most compression but executes the slowest Levels between 1 and 9 allow for a trade off between compression and execution speed The notation GZIP lt level gt is used for GZIP compression where lt level gt is a value from 1 to 9 For example GZIP 7 specifies a level of 7 Tests have shown that GZIP compression always provides a much better compression ratio To balance the execution speed and compression ratio a compression level of six 6 the default level for Unix Linux gzip command is suggested From CDF V3 5 0 the open source ZLIB package by Jean loup Gailly and Mark Adler is used as the sole source code for GZIP compression decompression The original CDF library code which was modified from the zip and unzip code by the same authors is no longer used Only needed source codes from ZLIB papackage are extracted and distributed with CDF Please refer to ACKNOWLEDGMENT txt in src lib zlib directory of the source package for information 2 7 TT2000 and Leap Seconds The CDF_TIME_TT2000 data type is defined as an 8 byte signed integer with a fixed Time_Base J2000 UTC based Julia
169. main requirements driving its development 1 Facilitate ingestion of data sets and data products into CDF 2 Utilize standard common terminology metadata to describe the data sets 3 Development of higher level applications e g NSSDC Graphics System NGS Programming reference manuals for C and Fortran users are provided as separate documents The above reguirements imply two classes of users for CDF One user class performs primarily data acguisition and is mainly involved in designing CDFs and the associated science metadata The other user class builds high level applications interacting with CDF at the programming level CDF has two levels of access one is through the programming interface layer and the other is through a high level toolkit written using the programming interface layer The toolkit provides a suite of utilities for creating browsing and modifying CDF files as well as exporting or importing CDF data to from a regular text file or an eXtensible Markup Language XML file These are very useful for architect ring a CDF and describing the metadata without using the programming level interfaces The browsing tools allow a guick look at CDF data sets and aid in CDF validation The CDF library comes with C Java and Fortran Application programming Interfaces APIs and the APIs provide the essential framework on which graphical and data analysis packages can be created Perl APIs are also available as an optional packa
170. me Warning 127 BAD ALLOCATE RECS BAD ARGUMENT BAD ATTR NAME BAD ATTR NUM BAD BLOCKING FACTOR BAD CACHESIZE BAD CDF EXTENSION BAD CDF ID BAD CDF NAME BAD CDFSTATUS BAD COMPRESSION PARM BAD DATA TYPE BAD DECODING BAD DIM COUNT BAD DIM INDEX An illegal number of records to allocate for a variable was specified For RV variables the number must be one or greater For NRV variables the number must be exactly one Error An illegal undefined argument was passed Check that all arguments are properly declared and initialized Error Illegal attribute name specified Attribute names must contain at least one character and each character must be printable Error Illegal attribute number specified Attribute numbers must be zero 0 or greater for C applications and one 1 or greater for Fortran applications Error An illegal blocking factor was specified Blocking factors must be at least zero 0 Error An illegal number of cache buffers was specified The value must be at least zero 0 Error An illegal file extension was specified for a CDF In general do not specify an extension except possibly for a single file CDF which has been renamed with a different file extension or no file extension Error CDF identifier is unknown or invalid The CDF identifier specified is not for a currently open CDF Error Illegal CDF name specified CDF names must contain at least one cha
171. mes as shown in each 2 dimensional array Table 1 2 Specifying variances circumvents this problem For each rVariable there are variances associated with the array dimensions as well as the records Record variance indicates whether or not an rVariable has unigue values from record to record in the CDF Time changes for each record so the record variance for Time is TRUE One could also say that Time is record variant Latitude and Longitude repeat their values from record to record so the record variance for each is false Latitude and Longitude are non record variant NRV The Temperature values change from record to record so they are record variant The record variances for this example are shown in Table 1 3 Record rVariables Number Time Longitude Latitude Temperature 0000 0000 165 150 40 40 20 0 19 2 1 l l l l 0000 0000 165 150 30 30 21 7 20 7 0100 0000 165 150 40 40 18 2 22 0 2 l l l l 0000 0000 165 150 30 30 19 3 19 2 0200 0000 165 150 40 40 19 9 19 6 3 l l 0000 0000 165 150 30 30 19 3 19 0 2300 0000 165 150 40 40 21 0 18 4 24 l l l l l l l 0000 0000 165 150 30 30 19 5 22 0 Table 1 2 Example CDF 2 Dimensional Representation Conceptual 16 Similarly the term dimension variance indicates whether or not an rVariable changes
172. milar to that of sparse arrays The exact differences in disk space savings and execution overhead between sparse arrays and variable compression will not be known until sparse arrays have been implemented No effort is planned to add this feature to the CDF 2 3 14 Compression A compression may be specified for a variable in a single file CDF that gets performed automatically as values are written The values are transparently decompressed as they are read from the variable The values of a variable are compressed in blocks of one or more variable records The blocking factor for a compressed variable described beginning on page 50 specifies the number of records in each block or the maximum number in the case of a compressed variable with sparse records Properly setting the blocking factor involves a trade off between the compression percentage achieved and execution speed when accessing values in individual variable records The CDF 35 Variable records may be deleted from a multi file CDF Note that variable compression is not allowed in a multi file CDF 47 library also uses a staging area scratch file to minimize access overhead for a compressed variable Note that if a block of variable records actually increases in size when compressed the block of records will be stored uncompressed in the CDF This could happen if the blocking factor is set too low or simply because of the nature of the data and the compression algorithm being use
173. mm nnn ppp where dd is the day of the month 01 31 mmm is the month Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov or Dec yyyy is the year 0000 9999 hh is the hour 00 23 mm is the minute 00 59 ss is the second 00 59 ccc is the millisecond 000 999 mmm is the microsecond 000 999 nnn is the nanosecond 000 999 and ppp is the picosecond 000 999 Functions exist that parse encode compute and decompose CDF_EPOCH and CDF EPOCH16 values These functions are described in the CDF C Reference Manual for C applications and in the CDF Fortran Reference Manual for Fortran applications 2 5 5 TT2000 Data Types CDF_TIME_TT2000 8 byte signed integer The CDF_TIME_TT2000 data type an alternative to CDF EPOCH and CDF EPOCH16 is used to store date and time values referenced from J2000 2000 01 01T12 00 00 000000000 The values also include leap seconds Please read our requirements analysis at http cdf gsfc nasa gov html leapseconds_requirements html and development approach at http cdf gsfc nasa gov html leapseconds html for more details 4 I know what you re thinking The year 0 AD never existed If it makes you feel better think of the epoch year as 1 BC or simply year 0 rather than 0 AD Also year 0 is considered to be a leap year 59 CDF TIME TT2000 values are the number of nanoseconds since 32000 The standard format used to display a CDF TIME TT2000 value is in ISO 8601 format yyyy mm ddThh
174. modifications to the CDF When read only mode is set all metadata is read into memory and references to metadata are then directed there zMode A CDF may be placed into zMode via the Internal Interface using the lt SELECT_ CDF_zMODE_ gt operation When in zMode a CDF s rVariables essentially disappear and are replaced by corresponding zVariables Likewise the rEntries for a vAttribute become zEntries because they are now associated with zVariables While in zMode most operations involving rVariables rEntries will fail Some inquiry operations will be allowed For example inquiring the number of rVariables is allowed but will always be zero When zMode is used the number of variables remains the same rVariables simply change into zVariables Note that the existing contents of the CDF are not changed the CDF simply appears different Each new zVariable has the same exact properties as the corresponding hidden rVariable except for dimensionality and variances The data specification data type and number of elements pad value etc stay the same The dimensionality variances of each zVariable are dependent on which zMode is currently being used zMode l or zMode 2 In zMode l the dimensionality variances stay exactly the same In zMode 2 however those dimensions with a false variance NOVARY are eliminated Consider a CDF with an rVariable dimensionality of 2 180 360 containing the following rVariables S This notation is
175. must be enclosed in double quote marks to preserve case sensitivity For the gzip compression lt level gt must be in the range from 1 fastest compression to 9 best compression For compressions not specified the compression in the source CDF will be used Specifying a variable compression using var overrides a compression specified with vars SPARSENESS lt types gt VMS sparseness lt types gt UNIX Specifies the types of sparseness to be used for the variables The lt types gt option consists of a comma separated list of the following vars lt sT gt Sparseness for all variables var lt name gt lt sT gt Sparseness for one particular variable Where lt sT gt is one of the following srecords no srecords pad or srecords prev and lt name gt is a delimited case sensitive variable name with the following syntax lt delim gt lt char1 gt lt char2 gt lt charN gt lt delim gt In general do not use single or double quote marks as delimiters VMS The entire delimited variable name must be enclosed in double quote marks to preserve case sensitivity For sparsenesses not specified the sparseness in the source CDF will be used Specifying a variable sparseness using var overrides a sparseness specified with vars ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies the zMode that should be used with the source CDF s The zMode may be one of the following 0 Indicates t
176. n Point of Contact Any Recipient contact with Government Agency is to be directed to the designated representative as follows Robert E McGuire nasa gov Chapter 1 1 Primer 1 1 Introduction The CDF Primer is designed for scientists researchers programmers and managers who want to learn about CDF without reading through this entire document or the programming reference guides The primer will address what CDF is and how it can be used for storing and managing different types of data A brief description of the tools and utilities available with CDF in addition to program and toolkit examples will be given More detailed descriptions of the concepts presented herein are provided in the accompanying chapters of this document and the programming reference guides 1 2 Why use CDF When people first hear the term CDF they intuitively think of data formats in the traditional sense of the word i e messy convoluted storage of data on disk or tape CDF is more than just a format CDF is a self describing format for managing data In addition to the actual data being stored CDF also stores user supplied descriptions of the data known as metadata This self describing property allows CDF to be a generic data independent format that can store data from a wide variety of disciplines In addition to being a self describing data format CDF is also a software library The library routines are callable from C Fortran and Java and allow
177. n Redistribution Display mo Boge B Under Patent Rights Subject to the terms and conditions of this Agreement each Contributor with respect to its own contribution to the Subject Software hereby grants to each Recipient under Covered Patents a non exclusive world wide royalty free license to engage in the following activities pertaining to the Subject Software a Use b Distribution c Reproduction d Sale e Offer for Sale C The rights granted under Paragraph B also apply to the combination of a Contributor s Modification and the Subject Software if at the time the Modification is added by the Contributor the addition of such Modification causes the combination to be covered by the Covered Patents It does not apply to any other combinations that include a Modification D The rights granted in Paragraphs A and B allow the Recipient to sublicense those same rights Such sublicense must be under the same terms and conditions of this Agreement 3 OBLIGATIONS OF RECIPIENT A Distribution or Redistribution of the Subject Software must be made B under this Agreement except for additions covered under paragraph 3H 1 Whenever a Recipient distributes or redistributes the Subject software a copy of this Agreement must be included with each copy of the Subject Software and 2 If Recipient distributes or redistributes the Subject Software in any form other than source code Recipient must also make the source code freely av
178. n VMS systems and environment variables on UNIX systems Trailing blanks are also allowed but will be ignored This is so Fortran applications do not have to be concerned with the trailing blanks of a Fortran CHARACTER variable C character strings use terminating NUL characters In almost all cases when a CDF file name is specified the cdf extension should not be appended It will be appended automatically by the CDF library The exception to this is when a user has renamed an existing CDF with a different extension or with no extension for whatever reason When a CDF is opened the CDF library first appends the cdf extension to the file name specified and then checks to see if that file exists If not the CDF library will also check to see if a file exists whose file name is exactly as specified without cdf appended If this is the case the CDF must be single file If the CDF is multi file an error occurs since the CDF library would have no idea as to how the variable files had been renamed Note also that the CDF library always appends cdf to the file name specified when creating a CDF NOTE The CDF toolkit programs will in some cases not recognize a CDF if it does not have an extension of cdf 2 2 7 Format There are two CDF formats multi file and single file The choice of which format to use will depend on how the CDF is to be accessed Note that the CDFconvert toolkit program can be used to change the format of an existing
179. n date 2451545 0 TT or 2000 January 1 12h TT Resolution nanoseconds Time_Scale Terrestrial Time TT Units nanoseconds Reference_Position rotating Earth Geoid with leap seconds included Given a current list of leap seconds conversion between TT and UTC is straightforward TT UTC deltaAT 32 184s where deltaAT is the sum of the leap seconds since 1960 for example for 2009 deltaAT is 34s Use of an 8 byte integer provides time with nanosecond resolution for the next 292 years so data providers will no longer need 16 byte CDF_EPOCH16 variables to carry their highest time resolution and in half the storage space CDF library provides a suite of functions that convert time in TT2000 to UTC based date time components which are the basis of epoch data in one of CDF_EPOCH CDF_EPOCH16 or CDF TIME TT2000 types in CDF The leap seconds are determined by CDF through a leap second table The text formed table a part of the CDF distribution since CDF V3 3 2 is also available from CDF site The table is accessed by the CDF library externally through an environment variable CDF_LEAPSECONDSTABLE on Unix Windows CDF LEAPSECONDSTABLE on VMS If the environment variable is not defined or the table file is not found the 4 OPTIMAL ENCODING TREES causes each buffer of data to be scanned for the best possible compression An alternative method would be to scan the first buffer being compressed and then use the same byte value frequencies for subse
180. n independent variable a dependent variable time and date value or whatever data might be e g image XML file etc In other words the variable doesn t contain any hidden meanings other than the data itself One may describe a variable or one variable s relationship with other variable s through attributes see the last paragraph of this section for more details The dimensionality of a variable depends upon how the data is specified by the user For scalar data as an example the array of values would be O dimensional i e a single value whereas for image data the array would be 2 dimensional Similarly the array for volume data would be 3 dimensional CDF allows users to specify arrays of up to ten dimensions The array for a particular variable is called a variable record A collection of arrays one for each variable is referred to as a CDF record A CDF can and usually does contain multiple CDF records This is useful for data with repeated observations at different times Two types of variables may exist in a CDF rVariables and zVariables Every rVariable in a CDF must have the same number of dimensions and dimension sizes In the scalar data example the CDF s rVariables would be 0 dimensional whereas for the image data example the CDF s rVariables would be 2 dimensional Figures 1 1 and 1 2 illustrate 0 dimensional and 2 dimensional rVariables respectively zVariables may have a different number of dimensions and or dimensi
181. naeenneeonreeeneennnneeneeeeneennneenneeenne enne 69 List of Tables Table 1 1 Table 1 2 Table 1 3 Table 1 4 Table 1 5 Table 2 3 Table 2 4 Table 2 5 Table 2 6 Table 2 7 Table 2 8 Table 2 9 Table 3 1 Example Data Set Flat Representation 0 Dimensional emnmeeeteoneeeteonneeteeneeeneeeneeeeeeeeeeee 16 Example CDF 2 Dimensional Representation Conceptual eeeremeeneeeeere 16 Example CDF Specification for 2 Dimensional Representation eieeeeeeeeeeere eee 17 Example CDF 2 Dimensional Representation Physical ceccessececcecsteeeneeeeeeececeececeseeesaeceaeecaeeeneeeeeeeegs 17 vAttribute rEntries for the Temperature rVariable 0 eee ees cneeseecsecnsecsecescesecesesseceseeeceeeeeeseaeseeseaesaaeenee 19 Cache Size Operations Internal Interface e ee eesecescessseeeceeeeeesseceececeaeeesaeceacecsaeeeneeeeeeecaeceececeeeesaeceaeeesaes 30 Equivalent Byte Orderin gs 4 sesso iiv kiskuda Sv ee 34 Equivalent Single Precision Floating Point Encodings isrteereeteeeneveneeeeeeneneneeeeeneneeenneenaeenee 35 Equivalent Double Precision Floating Point Encoding ceccesseceseeceeecesseceececeaeeeseeeeeeecaeceeeeceeeesaeceaeeesaes 35 Previous missing Sparse Records Example Conceptual View vs Physical Storage ie 45 Default Pad Valles autista ti uksele Seene eve eik ari 55 Equivalent Data Type Sai
182. name only if a single source CDF was specified If the directory paths are the same then a different CDF name must be specified If the source CDF specification is a directory wildcard path then this must be a directory path other than the source directory path This may also be a directory path if only a single CDF is being converted In any case do not specify an extension Qualifier s SKELETON lt skt cdf path gt VMS skeleton lt skt cdf path gt UNIX The file name of a skeleton CDF to be used during the conversions Do not enter an extension The skeleton CDF is used in the following cases 1 Ifa format for the destination CDF was not specified then the format of the skeleton CDF will be used 2 If a variable majority for the destination CDF was not specified then the variable majority of the skeleton CDF will be used 3 If a data encoding for the destination CDF was not specified then the data encoding of the skeleton CDF will be used Specifying a skeleton CDF is optional NOJLOG VMS no log UNIX Specifies whether or not messages about the progress of each conversion are displayed NO PAGE VMS no page UNIX Specifies whether or not the output is displayed a page at a time A prompt for the RETURN key will be issued after each page A page is generally 22 lines of output NO PERCENT VMS no percent UNIX Specifies whether or not the percentage of a variable s values converted is d
183. nce the 32 bit file offset had to be changed to a 64 bit file offset As a result CDF 2 7 2 or earlier won t be able to read CDF files that are created with CDF 3 0 or a later version However CDF 3 can read files that are created with any of the previous CDF releases If one is concerned about using CDF 3 0 or a later version due to the file compatibility problem with previous releases one can create files in the CDF 2 7 format optional with CDF 3 The Backward File Compatibility with CDF 2 7 section of the CDF C Reference Manual or Fortran Reference Manual describes how to create files that can be read by CDF 2 7 2 or earlier or IDL 6 2 or earlier IDL 6 3 can read files created by CDF 3 0 or a later version If a file is created in the CDF 2 7 format using the CDF 3 0 library or a later version the maximum file size is 2G bytes Note To minimize the scope of the coding changes as well as make the code functioning on both 32 64 bit machines when the 2G file size barrier is lift for V3 the maximum record number for each variable in a CDF stays the same as the previous versions It is dictated by the 32 bit integer for the record counter a 2G itself Checksum To ensure the data integrity in a CDF file the checksum option has been added to the CDF Version 3 2 This is a form of redundancy check a very simple measure for protecting the integrity of data by detecting error in data that is sent through space or time It works by adding up t
184. nctionality within the CDF library For example it can only handle rVariables not zVariables and has no access to zVariables s attribute entry zEntries Up until CDF 2 7 2 if you wanted to create or access zVariables and zEntries you had to use the Internal Interface that is harder to use The limitations of the Original Standard Interface were addressed with the introduction of the Extended Standard Interface in CDF 3 1 The Extended Standard Interface allows many new operations that were only previously available through the Internal Interface Both the Original Standard Interface and Extended Standard Interface are callable from both C and Fortran applications and the functions subroutines available in these interfaces are described in the CDF C Reference Manual and the CDF Fortran Reference Manual respectively Internal Interface The Internal Interface may be used to perform all supported CDF operations The Internal Interface is much more efficient than the Standard Interface but it s a bit more difficult to learn It should be used to perform those operations not available with the Original Standard Interface and the Extended Standard Interface The Extended Standard Interface offers almost everything average and sophisticated CDF users need The Internal Interface is callable from both C and Fortran applications and its available operations are described in the CDF C Reference Manual and the CDF Fortran Reference Manual respectively
185. ndard distribution package and it allows users to create a CDF file without programming SkeletonCDF reads a specially formatted text file called a skeleton table and generates a skeleton CDF Everything about a CDF can be specified in a skeleton table except data values for variables that vary from record to record record variant The toolkit program SkeletonTable is also provided in the CDF standard distribution package and it reads an existing CDF file and produces a skeleton table Below is a sample skeleton table file Skeleton table for the example CDF Generated Wed 5 Jan 1994 10 53 58 header CDF NAME examplel DATA ENCODING NETWORK AJORITY ROW FORMAT SINGLE Variables G Attributes V Attributes Records Dims Sizes GLOBALattributes Attribute Name TITLE VARIABLEattributes VALIDMIN 2 1 z 2 282 Entry Data Number Type Value L CDF CHAR An example CDF 22 VALIDMAX variables Variable Data Name Type AAA i Time CDF INT4 Attribute Data Name Type ZA ia a VALIDMIN CDF INT4 VALIDMAX CDF INT4 Variable Data Name Type Le 2 Longitude CDF REAL4 Attribute Data Name Type A et jaa pee VALIDMIN CDF REAL4 VALIDMAX CDF REAL4 NRV values follow 1 1 J 163 0 2 1 150 0 Variable Data Name Type Latitude CDF REAL4 Attribute Data Name Type lt ii VALIDMIN CDF REAL4 VAL
186. ns lt shell type gt 3 1 2 How to Invoke the GUI Toolkit for Macintosh OS X A complete set of the GUI toolkit is available in a file named CDFToolsDriver jar To invoke any of the CDF utilities e g CDFedit CDFexport etc in the toolkit do one of the following Double click the CDFToolsDriver jar icon on the Desktop 64 OR Go to the directory where the CDF library is installed and double click the CDFToolsDriver jar file located under the lt cdf install dir gt bin directory OR Open a Terminal session and type java CDFToolsDriver at the operating system prompt Users will be presented with a main menu containing all the available CDF Java tools from which a desired tool can be selected with a single click 3 1 3 How to Invoke the GUI Toolkit for Windows NT 95 98 2000 XP Two executable programs CDFfsi exe amp CDFso exe are included as part of the standard distribution package and each program contains the following CDF utilities tools CDFfsi exe CDFso exe CDFedit CDFcompare CDFexport CDFconvert CDFinquire CDFdump CDFstats SkeletonCDF SkeletonTable A CDF utility tool can be invoked by running CDFfsi exe or CDFso exe and selecting the tool listed under the File menu For example the SkeletonCDF utility can be invoked by running the CDfso exe program and then selecting the SkeletonCDF option under the File menu A Java version of CDFedit and CDFexport is also available in CDFToolsDriver jar and it is recommend
187. ns through the buffer writing the full physical records to the corresponding variables Likewise for a multiple variable read operation the CDF library places into a memory buffer provided by the application the full physical records read The order of the full physical records will correspond to the order of the list of variables specified and the full physical records will be contiguous The application must then process the buffer as needed Care must be used when generating and processing the memory buffer containing the full physical records If C struct objects or Fortran STRUCTURE variables are being used it may be necessary to order the variables being read written such that there are no gaps between elements of the structures assuming you are defining structures containing one element per full physical record where an element is a scalar variable or an array depending on the corresponding variable definition On some computers the C and Fortran compilers will place gaps between the elements of these structures so that memory alignment errors are not generated when the elements are accessed In general defining the structures so that larger data types are before smaller data types should result in no gaps e g the Fortran REAL 8 data type is larger than a INTEGER 2 which is larger than a BYTE The list of variables would be adjusted accordingly The variable majority must also be considered when performing a multiple variable read
188. nt on Earth at different times That fact is not immediately clear from this representation of the data Looking more closely we note that only two differing values are recorded for Longitude and similarly only two differing values are recorded for Latitude This repetition suggests a 2 dimensional array structure whose dimensions are defined by Longitude and Latitude For each of the two Longitude values there are two Latitude values Time repeats for each Longitude Latitude pair the observations were taken simultaneously at the longitude latitude locations Because of Time s repetition for Longitude Latitude pairs the number of Time values specifies the number of records needed in the CDF Each record conceptually contains a 2 dimensional array per rVariable Table 1 2 The array structure defines the dimensionality of the rVariables in the CDF Although there are four rVariables the array dimensions and the sizes of those dimensions are determined only by Longitude and Latitude Temperature varies across the entire array while Time tells us how many records to expect Therefore the example when reduced as described defines a CDF with 2 dimensional rVariables The number of discrete values for each rVariable that defines a dimension generates the size of that dimension For example Longitude has two unique values so the dimension defined by Longitude has a size of two Record rVariables Number Time Longitude Latitude Temperature 1 0000
189. nto the PromptField the more indicators lt and gt at the left and right ends of the PromptField will display where hidden characters exist EditWindows are used to display edit a text file or group of lines EditWindows are currently used to display online help and to edit gAttribute character string entries as if they were a text file 3 3 CDFexport 3 3 1 Introduction CDFexport allows the contents of a CDF to be exported to the terminal screen a text file or another CDF The variables to be exported can be selected along with a filter range for each variable which allows a subset of the CDF to be generated When exporting to another CDF a new compression and sparseness can be specified for each variable When exporting to the terminal screen or a text file the format of the output can be tailored as necessary When exporting the output to a CDF if the environment variable CDF_FILEBACKWARD on Unix or Windows or CDF FILEBACKWARD on OpenVMS is set to TRUE the output file is then automatically a V2 7 file If this environment variable is not set or set to anything other than TRUE then there is an option to choose for the file version when the program is executed 3 3 2 Special Attribute Usage CDFexport uses the following special attributes FORMAT Used as the initial value in a variable s Format field VALIDMIN Used as the initial filter value in a variable s Minimum field VALIDMAX Used as the initial filter value in a varia
190. nverted file will be set accordingly If the environment variable is not set then the source file s checksum mode is used for the converted file Alternatively to force the checksum mode to be or not to be set this option can be used If it is specified then it will overwrite the environment variable and the mode from the source file Currently the possible values for the checksum option are none md5 or source EPOCH2TT2000 TT20002EPOCH TT20002DEPOCH VMS epoch2tt2000 tt20002epoch tt20002depoch UNIX By default the epoch tt2000 data type is reserved when the file is converted However this option will convert the data in EPOCH EPOCH16 to TT2000 or vice verse As both EPOCH and EPOCH 16 data types do not have leap seconds leap seconds occur in TT2000 will be lost epoch2tt2000 will convert source data in either EPOCH or EPOCH 16 to TT2000 Also as TT2000 and EPOCH EPOCH16 have slightly different time resolution sub millisecond data may get lost or filled after conversion tt20002epoch will convert source data in TT2000 to EPOCH while tt20002depoch convert data in TT2000 to EPOCH 16 ABOUT VMS about UNIX Shows the library version that was used to create this tool program Note For non recond variant NVR variables since they only have a single record if the record size is too small less than 1K the compression will not be turned on even specified so Example s 81 VMS
191. olution offered by the CDF EPOCH data type Although the maximum timestamp resolution of CDF_EPOCH milliseconds 10 3 is adequate for many users there are some users who need a finer timestamp As a result a new data type CDF_EPOCH16 was introduced to accommodate a finer timestamp that address up to picoseconds 10 12 IDL 6 2 or earlier versions understand CDF 2 7 2 or earlier versions but not CDF 3 0 or later versions This means that if you need to take advantage of any of the new CDF 3 0 features e g ability to create a CDF file bigger than 2 GB or need to manipulate CDF files that were created with CDF 3 0 or later in IDL you ll have to wait until RSI manufacturer of IDL incorporates the CDF 3 1 library into IDL 6 3 scheduled for release in late 1Q 2006 In order to address this problem as an interim solution the CDF office obtained a copy of the IDL s built in CDF functions e g CDF_CREATE CDF_OPEN etc from RSI and extended it to support CDF Version 3 x s new file structure and data type If you now need to use any of the CDF 3 0 s new features or manipulate CDF files that were created with CDF 3 0 or a later version in IDL 6 2 or earlier please contact the CDF support office at gsfc cdf support Olists nasa gov for a binary copy of the IDL CDF system routines 125 B 3 Backward File Compatibility with CDF 2 7 By default a CDF file created by IDL 6 3 scheduled for release in late 10 2006 or later cannot be read
192. on floating point values are encoded in IEEE representation ALPHAOSF1_ENCODING DEC Alpha computers running OSF 1 SUN_ENCODING Sun computers SGi_ENCODING Silicon Graphics Iris and Power Series computers DECSTATION_ENCODING DECstation computers IBMRS_ENCODING IBM RS6000 series computers 2 This is a change from previous releases of CDF 33 HP ENCODING HP 9000 series computers PC ENCODING PC personal computers NeXT ENCODING NeXT computers MAC ENCODING Macintosh computers When HOST ENCODING is specified it is translated to the actual host encoding from the above list All host encodings are readable and writeable on any machine supported by CDF Network Encoding Network encoding NETWORK ENCODING specifies that variable and attribute entry data values be written to the CDF in the XDR External Data Representation format As values are written to the CDF the CDF library encodes them into the network encoding Network encoded CDFs are readable and writeable on any machine supported by CDF as are all of the other encodings Equivalent Encodings While an encoding exists for each supported computer not every encoding is different The following sections describe which computers use the same encoding for the various data types Character 1 Byte Integer Data Types Since each supported computer uses the ASCII character set and orders the bits in a byte the same way the character and 1 byte integer data types CDF CH
193. on sizes than that of the rVariables in a CDF Figure 1 3 illustrates several zVariables As you can see since all the rVariables must have the same dimensions and dimension sizes there ll be a lot of disk space wasted if a few variables need big arrays and many variables need small arrays Since zVariable is more efficient in terms of storage and offers more functionality than rVariable use of zVariable is strongly recommended Note that a CDF may contain both rVariables and zVariables The term variable is used when describing a property that applies to both rVariables and zVariables So why would you want to use rVariables over zVariables There s no reason to use rVariables at all since zVariables are much more efficient if you are creating a new CDF file But if you are analyzing data files that were created with early CDF releases or contain rVariables for some reason you ll need to use rVariables One may wonder why there are rVariables and zVariables not just zVariables When CDF was first introduced in early 90 s only rVariables were available The inefficiencies with rVariables were quickly realized and addressed with the introduction of zVariables in later CDF releases It is important to note that there is no single correct way to store data in a CDF The user has complete control over how the data values are stored in the CDF depending on how the user views the data This is the advantage of CDF Data values can be organized in what
194. owever where it would be desirable to create a CDF with host encoding e g on a slow machine and then transfer it to a faster machine for processing or conversion to another encoding Obviously there are trade offs as to which encoding should be used in any one particular case Keep in mind that a CDF can always be converted to the host encoding of the machine being used with the CDFconvert utility included in the CDF standard distribution package before being accessed 2 2 9 Decoding The decoding of a CDF determines how attribute entry and variable data values are passed to a calling application program from the CDF library The default decoding when a CDF is initially opened is host decoding the native encoding of the computer being used When host decoding is in effect all data values read by an application are immediately ready for manipulation and display Almost all of your applications will simply use the default of host 35 decoding and not be concerned with selecting a decoding There are some situations however where selecting a different decoding will be advantageous Some possibilities are as follows 1 A client server model where a number of CDFs are maintained on a server computer in any of the supported encodings Clients on different type computers could request data from a CDF on the server computer The server computer would then select a decoding for the CDF based on the client s computer type and then read the data v
195. parated by commas 10 The notation for variances used here is lt rec vary gt lt dim varys gt where lt rec vary gt is the record variance T TRUE or F false and lt dim varys gt is zero or more dimension variances 18 gAttributes can include any information regarding the CDF and all of its variables collectively Such descriptions could include a title for the CDF data set documentation or a CDF modification history A gAttribute may contain multiple entries called gEntries An example of this would be a modification history kept in the optional gAttribute MODS This attribute could be specified at CDF creation time and a gEntry made regarding creation date Any subsequent changes made to the CDF including additional variables changes in min max values or modifications to variable values could be documented by writing additional gEntries to MODS vAttributes further describe the individual variables and their values Examples of vAttributes would include such things as a field name for the variable the valid minimum and maximum the units in which the variable data values are stored the format in which the data values are to be displayed a fill value for errant or missing data and a description of the expected order of data values increasing or decreasing monotonicity The entries of a vAttribute correspond to the variables in the CDF Each rEntry corresponds to an rVariable and each zEntry corresponds to a zVariable Sample vAt
196. perty that applies to all of the various variable types 2 3 2 Accessing The Original Standard Interface deals exclusively with rVariables while the Extended Standard Interface deals zVariables The Internal Interface may be used to access either rVariables or zVariables 2 3 3 Opening The CDF library automatically opens the variable files in a multi file CDF as the variables are accessed An application never has to concern itself with opening variables The opening of variables does not apply to single file CDFs since individual files do not exist for each variable 2 3 4 Closing 38 The CDF library automatically closes the variable files in a multi file CDF when the CDF itself is closed by an application Variable files are also closed automatically by the CDF library as other variables are accessed if insufficient file pointers exist to keep all of the variables open at once This would be due to an open file guota enforced by the operating system being used A case also exists where it may be beneficial for an application to close a variable in a multi file CDF Since each open variable file uses some number of cache buffers a large amount of system memory could be in use see Section 2 1 5 This may not be a problem on VAX or UNIX machines but could result in a program crashing on an MS DOS machine If memory is limited an application may want to close variables after they have been accessed in order to minimize the total number of
197. plete CDF if the RV variable values also existed in the skeleton table The SkeletonTable toolkit program can be used to create a skeleton table from a CDF A CDF toolkit program which creates a skeleton CDF based on a skeleton table A complete CDF may also be created if the skeleton table contained RV variable values in addition to NRV variable values A CDF toolkit program which creates a skeleton table from a CDF A property assigned to a variable indicating that only those values written to a record should be stored Because the values of a variable record can be written in any order this allows gaps of missing values to occur A property assigned to a variable indicating that only those records written to the variable should be stored Because the records of a variable can be written in any order this allows gaps of missing records to occur A set of routines in the CDF library callable from C and Fortran applications that provide access to a commonly used subset of the capabilities of the Internal Interface This interface was defined with the release of CDF V2 0 and has not changed since New features since that time are available only through the Internal Interface e g zVariables and zMode A variable in a single file CDF that is not compressed nor has sparse records or arrays When the Internal Interface is used a property pertaining to an object e g a CDF s format or variable s data specification The result of a CDF
198. ported encodings may be read from and written to on any supported computer 1 4 3 Compression Compression may be specified for a single file CDF and the CDF library can be instructed to compress a CDFas itis written to disk This compression occurs transparently to the user When a compressed CDF is opened the CDF library automatically decompresses it An application does not have to even know that a CDF is compressed Any type of access is allowed on a compressed CDF When a compressed CDF is closed by an application it is automatically recompressed as it is written back to disk The individual variables of a CDF can also be compressed The CDF library handles the compression and decompression of the variable values transparently The application does not have to know that the variable is compressed as it accesses the variable s values 7 The CDF library supports several different compression algorithms When compression is specified for a CDF or one of its variables the compression algorithm to be used must be selected There will be trade offs between the different compression algorithms regarding execution performance and disk space savings The nature of the data in a CDF or variable will affect the selection of the best compression algorithm to be used 1 4 4 Sparseness Two types of sparseness are allowed for CDF variables sparse records and sparse arrays Sparse records are available now sparse arrays won t be available until a f
199. pressed dotCDF file is compressed and written to the file with the name specified when the CDF was opened created If the application program closing the CDF were to abnormally terminate before the dotCDF file was successfully compressed and written the uncompressed dotCDF scratch file would remain in the scratch directory The scratch directory used by the CDF library is described in Section 2 1 4 Overall compression for a CDF is specified with the lt PUT_ CDF_COMPRESSION_ gt operation of the Internal Interface It may be respecified as often as desired A CDF s overall compression may be inquired using the lt GET CDF COMPRESSION gt operation for an open CDF and the lt GET_ CDF_INFO_ gt operation for a CDF that has not been opened which saves the overhead of actually decompressing the CDF The available compression algorithms are described in Section 2 6 2 2 11 Limits Limits within a CDF are defined in the appropriate include files cdf h for C applications and cdf inc for Fortran applications The following limits exist CDF_MAX_DIMS The maximum number of dimensions that rVariables z Variables may have CDF_VAR_NAME_LEN256 The maximum number of characters in a variable name This limit was extended in CDF 3 0 to allow to create a longer variable name CDF_ATTR_NAME_LEN256 The maximum number of characters in an attribute name This limit was extended in CDF 3 0 to allow to create a longer attribute name CDF_PATHNAME_LEN The ma
200. propriate color shade to use Attempting to store the images and the palette using only rVariables would result in one of two undesirable situations If the CDF s rVariables had a dimensionality of 2 1024 1024 to store the images the palette would have to be stored in a 1024 by 1024 array that does not make sense logically and would waste disk space regardless of how the dimension variances are set If the CDF s rVariables had a dimensionality of 3 1024 1024 256 the images could be stored in an rVariable having dimension variances T TTF and the palette could be stored in an rVariable having dimension variances F FFT This would not waste any disk space but is not the intuitive way to store the data nothing in the data set is 3 dimensional Using zVariables to store the images and palette would solve both problems The images would be stored in a zVariable with dimensionality 2 1024 1024 and variances of T TT and the palette would be stored in a zVariable with a dimensionality of 1 256 and variances of F T This would waste no disk space and logically makes sense The use of zVariables is recommended because of this added flexibility Note that zVariables can always be used instead of rVariables In the rVariable example where temperature values were being stored zVariables could also have been used Each zVariable would have the same dimensionality and their dimension variances would be used in the same way as they were used for the
201. psecondsinfo program displays the information of the leap seconds table that the CDF uses for processing the epoch data in CDF_TIME_TT2000 type 3 15 2 Executing the CDFleapsecondsinfo Program Usage VMS CDFLEAPSEONDSINFO DUMP NODUMP ABOUT UNIX including Mac OS X 107 Windows cdfleapsecondinfo dump nodump about Parameter s Qualifier s DUMP NODUMP gt VMS dump nodump Unix Windows Specifies whether or not the table contents of the leap seconds is to be displayed ABOUT VMS about Unix Windows Shows the library version that was used to create this tool program Example s VMS CDFLEAPSECONDSINFO NODUMP CDFLEAPSECONDSINFO DUMP UNIX and Windows cdfvalidate nodump cdfvalidate dump Help is displayed when CDFleapsecondsinfo is executed without any arguments 108 Appendix A Skeleton Table Format A l Introduction Skeleton tables are both created by and read by CDF utility programs SkeletonTable creates a skeleton table by reading a CDF SkeletonCDF creates a CDF by reading a skeleton table In almost all cases the format of the skeleton tables read and written will be the same Any differences are minor and will be described where appropriate The skeleton table has a free format except where noted you need not be concerned with any column alignments spaces between fields or spaces between successi
202. quent buffers 61 hard coded internal table within the CDF library is used If the CDF version is released after the last leap second was added the internal table should be identical to external table The tool program CDFleapsecondsinfo at Section 3 15 will show how the table is used On Unix based system a shell script checkleapseconds sh is distributed which can be used to check if the local leap seconds table exists in either internal or external form if exists it also checks whether it s up to date A similar batch file checkleapseconds bat is also available for Windows When a new leap second is added the table will be updated and a new CDF library version will be released Using the updated external table with existing CDF version will continue to produce correct TT2000 data Using a un updated external internal table will cause a problem if the epoch data is after the date the new leap second is added 62 Chapter 3 3 Toolkit Reference 3 1 Introduction The CDF toolkit is a set of utility programs that allow the creation analysis and modification of CDFs The following sections describe how to use the CDF tools in the toolkit Two versions of the toolkit command line version and GUI version are included as part of the standard CDF distribution package and the CDF tools described in this chapter are the command line version The Graphical User Interface GUI version of the CDF tools are not described here since t
203. r single file or multi file CDFs are created HOST or NETWORK VMS host or network UNIX Specifies the initial setting of whether host encoded or network encoded CDFs are created ROW or COLUMN VMS row or column UNIX Specifies the initial setting of whether row major column major or input major CDFs listings are generated Input majority is the majority of the input CDF Input majority is selected by specifying neither row majority nor column majority EPOCH EPOCH1 EPOCH2 EPOCH3 ISO8601 EPOCHf or EPOCHx VMS epoch epoch1 epoch2 epoch3 iso8601 epochf or epochx UNIX Specifies the initial EPOCH encoding style HORIZONTAL or VERTICAL VMS horizontal or vertical UNIX Specifies the initial setting of whether horizontal or vertical listings are generated Note that these options can be changed at any time after the CDF has been opened If this qualifier is not specified each of these options has a default setting These default settings are also used for options not specified with this qualifier ZMODE lt mode gt VMS zmode lt mode gt UNIX Specifies which zMode should be used The zMode may be one of the following O Indicates that zMode should be disabled 1 Indicates that zMode 1 should be used The dimension variances of rVariables will be preserved 2 Indicates that zMode 2 should be used The dimensions of rVariables having a variance of NOVARY false are removed NO NEG2POSFP0 VMS no
204. racter and each character must be printable Trailing blanks are allowed but will be ignored Error Unknown CDF status code received The status code specified is not used by the CDF library Error An illegal compression parameter was specified Error An unknown data type was specified or encountered The CDF data types are defined in cdf h for C applications and in cdf inc for Fortran applications Error An unknown decoding was specified The CDF decodings are defined in cdf h for C applications and in cdf inc for Fortran applications Error Illegal dimension count specified A dimension count must be at least one 1 and not greater than the size of the dimension Error One or more dimension index is out of range A valid value must be specified regardless of the dimension variance Note also that the combination of dimension index count and interval must not specify an element beyond the end of the dimension Error The status code BAD BLOCKING FACTOR was previously named BAD EXTEND RECS 128 BAD DIM INTERVAL BAD DIM SIZE BAD ENCODING BAD ENTRY NUM BAD FNC OR ITEM BAD FORMAT BAD INITIAL RECS BAD MAJORITY BAD MALLOC BAD NEGtoPOSfp0 MODE BAD NUM DIMS BAD NUM ELEMS BAD NUM VARS BAD READONLY MODE BAD REC COUNT Illegal dimension interval specified Dimension intervals must be at least one 1 Error Illegal dimension size specified A dimension size must be at least on
205. racters including blanks Trailing blanks however are ignored when the CDF library compares attribute names UNITS and UNITS are considered to be the same name so they cannot both exist in the same CDF This was done because Version 1 of CDF padded attribute names on the right with blanks out to eight characters When a Version 1 CDF was converted to a Version 2 CDF these trailing blanks remained in the attributes names To allow CDF Version 2 applications to read such a CDF without having to be concerned with the trailing blanks the trailing blanks are ignored by the CDF when comparing attributes names The trailing blanks are returned as part of the name however when an attribute is inquired by an application program 2 4 2 Numbering The attributes in a CDF are numbered consecutively starting at one 1 for Fortran applications and starting at zero 0 for C applications The CDF library assigns attribute numbers as the attributes are created Note that there are not separate lists of global and variable scoped attributes Only one list of attributes exists in a CDF containing both global and variable scoped attributes 2 4 3 Attribute Scopes Attribute scopes declare the intended purpose of an attribute Global scope attributes gAttributes describe some aspect of the entire CDF Variable scope attributes vAttributes describe some property of each variable An attribute s scope exists to assist in the interpretation of its entries
206. rcase equivalents Note that the host and network qualifiers are no longer necessary but are supported for compatibility with previous CDF distributions COMPRESSION lt types gt VMS compression lt types gt UNIX Specifies the types of compression to be used for the CDF and or variables The lt types gt option consists of a comma separated list of the following cdf lt cT gt CDF s compression vars lt cT gt Compression for all variables vars lt cT gt lt bF gt Compression for all variables with a blocking factor specified vars lt cT gt lt bF gt lt r gt Compression for all variables with a blocking factor and reserve percentage specified var lt name gt lt cT gt Compression for one particular variable var lt name gt lt cT gt lt bF gt Compression for one particular variable with a blocking factor specified var lt name gt lt cT gt lt bF gt lt r gt Compression for one particular variable with a blocking factor and reserve percentage specified Where lt cT gt is one of the following compressions none rle 0 huff 0 ahuff 0 or gzip lt level gt lt bF gt is a blocking factor lt r gt is a reserve percentage and lt name gt is a delimited case sensitive variable name with the following syntax lt delim gt lt char1 gt lt char2 gt lt charN gt lt delim gt 79 In general do not use single or double quote marks as delimiters VMS The entire delimited variable name
207. report lt types gt cache lt sizes gt no statistics about lt cdf path gt Parameter s lt cdf path gt The file name of the CDF to analyze Do not specify an extension Qualifier s NO RANGE VMS no range UNIX 87 Specifies whether or not range checking will be performed To perform range checking the CDF must contain VALIDMIN and VALIDMAX attributes A variable must also have an entry for each of these attributes in order for range checking to be performed on that variable Note that for variables having a non character data type only the first element of the VALIDMIN and VALIDMAX attribute entries are used NO FILL VMS no fill UNIX Specifies whether or not fill values are ignored when collecting statistics The FILLVAL attribute entry for a variable if it exists is used as the fill value OUTPUT lt file path gt VMS output lt file path gt UNIX If this gualifier is specified the statistical output is written to the named file If the named file does not have an extension sts UNIX amp Macintosh or STS VMS amp MS DOS is appended automatically If this qualifier is not specified the output is displayed on the screen NOJFORMAT VMS no format UNIX Specifies whether or not the FORMAT attribute is used when displaying variable values if the FORMAT attribute exists and an entry exists for the variable NO PAGE VMS no page UNIX Specifies whether or not
208. rmination This Agreement and the rights granted hereunder will terminate automatically if a Recipient fails to comply with these terms and conditions and fails to cure such noncompliance within thirty 30 days of becoming aware of such noncompliance Upon termination a Recipient agrees to immediately cease use and distribution of the Subject Software All sublicenses to the Subject Software properly granted by the breaching Recipient shall survive any such termination of this Agreement Severability If any provision of this Agreement is invalid or unenforceable under applicable law it shall not affect the validity or enforceability of the remainder of the terms of this Agreement Applicable Law This Agreement shall be subject to United States federal law only for all purposes including but not limited to determining the validity of this Agreement the meaning of its provisions and the rights obligations and remedies of the parties Entire Understanding This Agreement constitutes the entire understanding and agreement of the parties relating to release of the Subject Software and may not be superseded modified or amended except by further written agreement duly executed by the parties Binding Authority By accepting and using the Subject Software under this Agreement a Recipient affirms its authority to bind the Recipient to all terms and conditions of this Agreement and that that Recipient hereby agrees to all terms and conditions herei
209. rs in the string For example if a gEntry value for a gAttribute named TITLE were Example CDF Title not including the double quotes the data type would be CDF_CHAR and the number of elements would be 18 a character string of size 18 For non character data types the number of elements is the size of an array of the data type For example if a zEntry value of a vAttribute named RANGE were 100 0 900 0 the data type would be CDF_REAL4 and the number of elements would be two an array of two values 57 Deleting An entry may be deleted from an attribute The disk space used by the entry becomes available for use as needed by the CDF library There is no renumbering of entries as with deleting a variable or attribute Entries are deleted using the lt DELETE_ gENTRY_ gt lt DELETE rENTRY gt and lt DELETE_ zENTRY_ gt operations of the Internal Interface 2 5 Data Types CDF supports a variety of data types consistent with the types available with C and Fortran compilers on most computers All data types are based on an 8 bit byte The size of an element of a data type is the same regardless of the computer operating system being used The lt GET DATATYPE SIZE gt operation of the Internal Interface may be used to inguire the size in bytes of a particular data type 2 5 1 Integer Data Types CDF_BYTE 1 byte signed integer CDF INTI 1 byte signed integer CDF UINTI 1 byte unsigned integer CDF INT2 2 byte signed int
210. rted computers but is only really necessary on VAXes and DEC Alphas running OpenVMS The CDF library is told to convert 0 0 to 0 0 for a CDF via the Internal Interface using the lt SELECT_ CDF_NEGtoPOSfp0_MODE_ gt operation When this mode is disabled a warning NEGATIVE FP ZERO is returned when 0 0 is read from a CDF and the decoding is that of a VAX or This notation is also used throughout this document The record variance is before the slash and the dimension variances 27 DEC Alpha running OpenVMS or written to a CDF and the encoding is that of a VAX or DEC Alpha running OpenVMS 2 1 3 Limits Open CDFs The only limit on the number of CDFs that may be open simultaneously is the operating system s limit on the number of open files that an application may have Each open CDF will always have at least one associated open file the dotCDF file The CDF library will open and close the variable files of a multi file CDF as needed see Sections 2 3 3 and 2 3 4 2 1 4 Seratch Files The CDF library will make use of scratch files when necessary These scratch files are associated with an open CDF Scratch files are used instead of core memory in an effort to prevent memory limitation problems especially on the Macintosh and PC The following types of scratch files are used Staging The staging scratch file is used when a CDF contains compressed variables As each variable is accessed a portion of the staging scratch file is
211. s beyond the first value of a dimension whose variance is NOVARY A status code indicating that the operation did complete but probably not as expected External Data Representation An integer floating point representation using big endian byte ordering and the IEEE 754 floating point representation An entry for a vAttribute corresponding to a zVariable A mode of the CDF library in which rVariables are made to appear as zVariables and rEntries appear as zEntries Z variable A CDF object in which data values are stored zVariables can have dimensionalities that are different than those of the rVariables and each other 145 Index 0 0 to 0 0 Mode 27 Adaptive Huffman compression 61 allocated records 45 assumed scope 56 attributes 18 55 creating 55 deleting 57 entries 19 57 accessing 57 data specification 57 data type 57 number of elements 57 deleting 58 gEntry 19 56 numbering 57 rEntry 19 56 FILLVAL 66 87 naming 56 case sensitivity 56 trailing blanks 56 numbering 56 83 assigning 56 SCALEMAX 66 87 SCALEMIN 66 87 scopes 56 assumed 56 converting 56 correcting 56 global 56 purpose 56 restrictions 56 variable 56 special 66 usage 67 70 86 92 VALIDMAX 66 87 VALIDMIN 66 86 vAttributes 19 56 big endian 34 blocking factor 46 caching scheme files 28 CDF 7 definition 7 deleting 72 78 96 CDF Java Interface 21 CDF library 8 25 caching scheme
212. s errors instructions as necessary Cycling through the four parts of the SelectionWindow allows the selection of the output to be generated The online help explains the purpose of each field in the four parts of the SelectionWindow The OptionMenu allows additional selections affecting the output The ActionMenu is then used to generate the desired type of output as well as some other miscellaneous operations The easiest way to learn how to use CDFexport is to read through the online help while generating the various types of output using a CDF with which you are familiar 3 4 CDFconvert 3 4 1 Introduction 76 The CDFconvert program is used to convert various properties of a CDF In all cases new CDFs are created Existing CDFs are not modified Any combination of the following properties may be changed when converting a CDF 1 The format of the CDF may be changed see Section 2 2 7 2 The data encoding of the CDF may be changed see Section 2 2 8 3 The variable majority of the CDF may be changed see Section 2 3 15 4 The compression of the CDF see Section 2 2 10 or the CDF s variables see Section 2 3 14 may be changed 5 The sparseness of the CDF s variables may be changed see Sections 2 3 12 and 2 3 13 6 The file version may be changed to be backward compatible 7 The checksum method may be changed 3 4 2 Executing the CDF convert Program
213. s field would be left blank The name of the vAttribute for which to specify an rEntry for this rVariable The vAttribute must have been specified in the vAttributes section The name must be delimited with a character not appearing in the name itself e g SCALEMAX or range The delimiting characters are not part of the vAttribute name in the CDF lt entry data type gt The data type for the vAttribute rEntry The data type must be one of the following CDF BYTE CDF INT1 CDF_UINT1 CDF INT2 CDF UINT2 CDF INT4 CDF_UINTA CDF INT8 CDF REAL4 CDF FLOAT CDF REAL8 CDF DOUBLE CDF EPOCH CDF EPOCH16 CDF TIME TT2000 CDF CHAR or CDF UCHAR The value s for the vAttribute rEntry The format of attribute entry values is described in Section A 3 NOTE The last rEntry MUST be followed by a period If no rEntries are specified for an rVariable the period must still be present The record number of an rVariable value This will be present only for record variant RV rVariables 115 lt indices gt lt value gt The indices of an rVariable value The indices are enclosed in brackets and separated by commas e g 23 1 or 1 80 If the rVariables have zero dimensions would be specified the brackets are still reguired The value at the given record indices For character data types CDF CHAR or CDF UCHAR the string must be delimited with a unigue character and enclosed in braces 1 in the same manner as for an a
214. s of the input and output CDFs in a text file called my merge txt Then the contents of my_merge txt should contain the following a cdf b cdf c cdf d cdf e cdf merged_ab cdf NOPREFIX VMS noprefix Unix Windows This option specifies not to add the system default prefix to the beginning of each variable name in the merged file If this option is specified it s assumed that all the variable names in the source CDFs are unique If a duplicate variable name is encountered the cdfmerge program will abort This option cannot be used with the prefixes option 101 PREFIXES lt prefix 1 gt lt prefix2 gt lt prefixN gt VMS prefixes lt prefix1 gt lt prefix2 gt lt prefixN gt Unix Windows This option allows to specify the user provided prefixes optional to be used when naming the variables in the merged CDF file Prefixes should be separated by a comma and the number of prefixes must match the number of source CDFs The first prefix corresponds to the first input file the second prefix corresponds to the second input file and so on Prefix can be any text combination of letters and numbers that describes the file User defined prefix followed by a period is added at the beginning of each variable name in the merged file Suppose there are two files to be merged a cdf and b cdf and each file contains two variables varl and var2 The following command will merge these two files into a file called merged
215. s this variable see Section 2 3 16 only one cache buffer would be necessary for the variable file if the second dimension were incremented the fastest i e 1 1 1 2 10 63 10 64 This is because the values of a record would be accessed sequentially from the first block to the last block If however the first dimension were incremented the fastest 1 e 1 1 2 1 9 64 10 64 10 cache buffers would improve performance The values of a record are not being accessed sequentially but rather each read would be from a different block Since the reads would be spread access 10 blocks having at least 10 cache buffers would be optimal A similar situation arises when accessing standard variables in a single file CDF If values are accessed for each variable at a particular record number then performance will be improved by setting the number of cache buffers for the dotCDF file to be equal to or greater than the number of variables This is because the variable values will most likely be located in that many different file blocks for a particular record number The Internal Interface is used to select and confirm the cache sizes being used for various files by the CDF library Confirming a cache size if it has not been explicitly selected will determine the default being used The operations used for each type of file are shown in Table 2 3 NOTE If the performance of your application is critical it is very important to
216. scheme used by the CDF library NO STATISTICS VMS no statistics UNIX Specifies whether or not caching statistics are displayed when a CDF is closed NO SIMPLE VMS no simple UNIX Specifies if a simplified version of CDFexport should be executed The following conditions apply to simple mode Only text listings can be generated to the screen or a file No filtering is available When listing to a text file FORMAT attribute entries are ignored and standard formats are used instead Only a limited set of the options for the initial qualifier may be specified zMode 2 is used by default Horizontal listings are created by default BATCH lt mode gt VMS batch lt mode gt UNIX Specifies if CDFexport should execute in a non interactive batch mode The mode option may be either text to generate a text file listing or cdf to output to a new CDF A settings file will be used if one exists with the default name in the current directory or is explicitly specified with the settings qualifier The settings file contains the parameters necessary to specify how the output CDF or text file should be generated If a settings file is not available default parameters will be used CDFexport must be used interactively to create a settings file CDF lt cdf gt VMS cdf lt cdf gt UNIX Specifies an output CDF file name to be used when exporting to a CDF in batch mode Do not include an extension Wh
217. settings lt cdf spec gt Parameter s lt cdf spec gt The specification of the CDF s from which to export Do not specify an extension This may be either a single CDF file name or a directory wildcard path Wildcards are allowed in the CDF name but not in the directory path Qualifier s NO PROMPT VMS no prompt UNIX Specifies whether or not a prompt is issued for the CDF s specification If this qualifier is not specified the CDF s specification must be entered on the command line and is automatically opened If a CDF s specification was entered on the command line that CDF s specification will initially appear at the prompt Otherwise the current directory will appear at the prompt JINITIAL lt defaults gt VMS initial lt defaults gt UNIX The default settings that are initially in affect when a CDF is opened These setting are only the settings initially in effect The user may change any of them at any time More detailed descriptions of each option may be found in the appropriate sections that follow lt defaults gt is a comma separated list of settings consisting of one or more of the options in the list that follows NOJFILTER VMS no filter UNIX Whether or not each item variable is initially filtered NOJFILLS VMS no fills UNIX Whether or not the use of fill values is enabled 71 NOJFORMAT VMS no format UNIX Specifies whether or not a variable s FORMAT attr
218. sumed attribute scope detected will be physically converted to the corresponding definite scope Note that if this automatic conversion is incorrect the scope of an attribute can be corrected using the Internal Interface in an application program or by editing the CDF with the CDFedit program This was not necessarily the case in previous releases of CDF These new restrictions should not however cause any conflicts with existing applications 56 2 4 4 Deleting An attribute may be deleted from a CDF Deleting an attribute also deletes the corresponding entries The disk space used by the attribute definition and the corresponding entries becomes available for use as needed by the CDF library Also the attributes that numerically follow the attribute being deleted are renumbered immediately Each is decremented by one Attributes are deleted using the lt DELETE_ ATTR_ gt operation of the Internal Interface 2 4 5 Attribute Entries Attribute entries are used to actually store metadata Each attribute in a CDF may have zero or more associated entries For vAttributes two types of entries are supported rEntries and zEntries rEntries describe some property of the corresponding rVariable and zEntries describe some property of the corresponding zVariable Note that an entry does not have to exist for each variable in the CDF For gAttributes only one type of entry is supported and is referred to as a gEntry The gEntries are independent
219. t VMS vars lt varl var2 varN gt Unix Windows Specifies which variables in the CDF should be dumped By default all the variables In the CDF are dumped Variable names must be separated by a comma RECORDRANGE lt startrec endrec gt VMS recordrange lt startrec endrec gt Unix Windows Specifies the record range for variables in the CDF to be dumped By default all the variables records in the CDF are dumped If only one record number is provided then all records after that number are dumped ABOUT VMS about Unix Windows Shows the library version that was used to create this tool program NOHEADER VMS noheader Unix Windows Whether to show the 1 line header Dumping cdf from from the dump output No display if noheader is specified COL2ROW VMS col2row Unix Windows Whether to show the column major multi dimensional variable data in a row major form By default the data is showed as the order as it s stored in the file For a row major file the sequence of the displayed data matches up to the C based dimensional indices For column major file the displayed data will not match up to the ivairable s indices With this option the column major data is transferred to row major form so they can be matched to the indices For example a 2 d 3 by 2 column major variable record the displayed data with this option will represent values from indices of 0 0 0 1 1 0 1 1
220. t can be organized into arrays Two types of variables are supported rVariable and zVariable and they can coexist in the same CDF file Use of zVariable over rVariable is strongly recommended since it is much more efficient and offers more functionality than the rVariable So why would you want to use rVariables over zVariables There s no reason to use rVariables at all if you are creating a new CDF file But if you are analyzing data files that were created with early CDF releases or contain rVariables for some reason you ll need to use rVariables One may wonder why there are rVariables and zVariables not just zVariables When CDF was first introduced in early 90 s only rVariables were available The inefficiencies with rVariables were quickly realized and addressed with the introduction of zVariables in later CDF releases rVariables rVariables all have the same dimensionality number of dimensions and dimension sizes An example of the type of data set that may be stored in a CDF s rVariables is shown in Table 1 1 Each record holds one value for each of the four variables Time Longitude Latitude and Temperature CDF can store scalar data in a at 0 dimensional representation such as this but storage in this manner may hide fundamental relationships among the data values Consistent repetitions found in the data for this example suggest another way to organize the data set Note that every fourth record is an observation at the same poi
221. te entries 1 1 1 1013 1 2 1 1 1015 0 3 1 1 1012 3 A sample variable definition for a CDF with 0 dimensional rVariables follows Variable Data Number Record Dimension Name Type Elements Variance Variances siit OOOO Ne Latitude CDF_REAL4 1 F Attribute Data Name Type Value Drees O ea G VALIDMIN CDF_REAL4 90 0 gt VALIDMAX CDF_REAL4 900 12 3 A 6 zVariable Section The optional zVariables section contains the definition of each zVariable in the CDF the values for any vAttribute zEntries associated with each zVariable and optionally data values for those zVariables The format of the zVariables section is as follows zVariables 117 lt variable definition gt lt variable definition gt lt variable definition gt lt variable definition gt Where lt variable definition gt is a zVariable definition The format of each zVariable definition is as follows Variable Data Number Name Type Elements Dims lt var name gt Attribute Data Name Type Value lt var data type gt lt n elems gt lt dims gt lt attr name gt lt attr name gt lt attr name gt lt attr name gt lt entry data type gt lt entry data type gt lt entry data type gt lt entry data type gt lt entry value gt lt entry value gt lt entry value gt lt entry value gt lt rec num gt lt indices gt lt value gt lt rec num gt lt indices gt
222. termined This can greatly improve performance when writing and reading values for the variable because of reduced overhead when searching the index entries as described in Section 2 2 7 The application is normally expected to write to all of the allocated records For NRV variables only one record may be allocated because only one record will ever physically exist If the variable has sparse records only those blocks of records that are going to be written would be allocated Records cannot be allocated by an application for compressed variables because they are allocated automatically by the CDF library when their compressed size is known Performance is improved when using this method because the allocated records will be as contiguous as possible requiring the fewest number of index entries This will greatly improve the time needed to locate a particular record when the variable is accessed In addition the CDF will be slightly smaller because of the reduced number of index records Note that records do not have to be allocated by an application before they are written to a variable The CDF library will automatically allocate any needed records based on the variable s blocking factor Also records may be allocated at any time not only before records have been written as in previous CDF releases Records are allocated using the lt PUT_ r zVAR_ALLOCATERECS gt and lt PUT_ t zVAR_ALLOCATEBLOCK_ gt operations of the Internal Interface
223. the ordering of full physical records in the memory buffer Even if a void memory buffer is used with type casting to access individual values the alignment of the values in the memory buffer is important on some computers Multiple variable writes are performed using the lt PUT 1 ZVARs RECDATA gt operation of the Internal Interface Multiple variable reads are performed using the lt GET 1 ZVARs RECDATA gt operation The selection of record numbers is performed using the lt SELECT r zZVARs RECNUMBER gt operation 2 3 20 Variable Pad Values Variable pad values are used in several situations 1 When the first value is written to a new record for records containing multiple values the other values in that record will contain the pad value This also applies to hyper writes if less than the entire record is written The unwritten values will contain the pad value 2 Fora variable not having sparse records when a new record is written that is more than one record beyond the last record already written the intervening records will also be written and will contain pad values This does not apply to NRV variables because only one physical record actually exists 38 This notation is used throughout this document The data type is before the slash and the number of elements is after the slash In this case the data type is CDF INT2 and the number of elements is one 1 These were previously known as fill values but were renam
224. the user to randomly access and manage data and metadata without regard to their physical storage This completely relieves the user of low level I O operations allowing more time for data analysis The actual format used to store the data and metadata is completely transparent to the user If an application is written in Java it can be executed without any modifications on any of the Java supported platforms The term CDF is also used to refer to the physical files that the CDF library generates A data set stored using the CDF library is called a CDF CDF files created on one operating system can be read without any modifications on any of the following CDF supported platforms VAX OpenVMS and POSIX shell Sun SunOS amp Solaris DECstation ULTRIX DEC Alpha OSF 1 or Tru64 amp OpenVMS Silicon Graphics Iris and Power Series IRIX IBM RS6000 series AIX HP 9000 series HP UX NeXT Mach PC DOS Windows 3 x Windows NT 95 98 2000 XP Linux Cygwin amp MinGW and Macintosh Mac OS X or Linux 1 3 Conceptual Organization An important feature of CDF is that it can handle data sets that are inherently multidimensional in addition to data sets that are scalar To do this CDF groups data by variables whose values are conceptually organized into arrays CDF s variable is a generic name or an object that represents data and it does not have any scientific context associated it For example a variable can be data representing a
225. tivity 39 trailing blanks 39 non record variant NRV 41 numbering 39 assigning 39 opening 38 pad values 54 default 55 usage 54 records 42 allocated 45 blocking factor 46 compression 47 reserve percentage 48 deleting 47 indexing 32 initial 45 maximum 42 numbering 44 physical 42 sparse 44 virtual 42 record variant RV 41 reserve percentage 48 rVariables 15 sparse arrays 47 sparse records 44 zVariables 18 variance dimension 41 record 40 XDR 34 zMode 26
226. to record or F the values do not vary from record to record The dimension variances of the zVariable For each dimension there must be either a T the values vary along that dimension or F the values do not vary along that dimension Each dimension variance must be separated by at least one space If the zVariable has zero dimensions this field would be left blank The name of the vAttribute for which to specify a zEntry for this zVariable The vAttribute must have been specified in the vAttributes section The name must be delimited with a character not appearing in the name itself e g SCALEMAX or range The delimiting characters are not part of the vAttribute name in the CDF The data type for the vAttribute zEntry The data type must be one of the following CDF_BYTE CDF INT1 CDF UINT1 CDF_INT2 CDF_UINT2 CDF_INT4 CDF UINT4 CDF INT8 CDF REAL4 CDF FLOAT CDF REAL8 CDF DOUBLE CDF EPOCH CDF EPOCH16 CDF TIME TT2000 CDF CHAR or CDF UCHAR The value s for the vAttribute zEntry The format of attribute entry values is described in Section A 3 NOTE The last zEntry MUST be followed by a period If no zEntries are specified for a zVariable the period must still be present The record number of an zVariable value This will be present only for record variant RV zVariables The indices of an zVariable value The indices are enclosed in brackets and separated by commas e g 23 1 or 1 80 If the zVariable
227. ton table is described in Appendix A 3 8 SkeletonCDF 3 8 1 Introduction The SkeletonCDF program is used to make a fully structured CDF called a skeleton CDF by reading a text file called a skeleton table The SkeletonCDF program allows a CDF to be created with the following 1 The necessary header information the number of dimensions and dimension sizes for the rVariables format data encoding and variable majority 2 The gAttribute definitions and any number of gEntries for each 3 The rVariable and zVariable definitions 4 The vAttribute definitions and the entries corresponding to each variable This program was originally named CDFskeleton It has been renamed to ease the confusion caused some users Now SkeletonCDF is used to create skeleton CDFs and SkeletonTable is used to create skeleton tables 95 5 The data values for any or all of the variables The created CDF is referred to as a skeleton CDF 3 8 2 Executing the SkeletonCDF Program Usage VMS SKELETONCDE CDF lt cdf path gt NO LOG NO DELETE NO FILLVAL REPORT lt types gt NO NEG2POSFP0 CACHE lt sizes gt ZMODE lt mode gt BACKWARD ABOUT lt skeleton path gt H UNIX including Mac OS X skeletoncdf cdf lt cdf path gt no log no delete no fillval report lt types gt no neg2posfp0 cache lt sizes gt zmode lt mode gt
228. tribute entry for a variable will be updated with the actual maximum value found Again if the variable has a non character data type the VALIDMAX attribute entry will be updated to have just one element FILLVAL If fill value usage is enabled used as the value that is ignored while collecting statistics for a variable MONOTON If requested the MONOTON attribute entry for a variable will be updated with the actual monotonicity found The possible values for the MONOTON attribute entry are described in Section 3 1 5 SCALEMIN If requested the SCALEMIN attribute entry for a variable will be updated with the actual minimum value found SCALEMAX If requested the SCALEMAX attribute entry for a variable will be updated with the actual maximum value found The usage of these special attributes can be controlled with command line qualifiers 3 6 3 Executing the CDFstats Program Usage VMS CDFSTATS OJRANGE NO FILL OUTPUT lt file path gt NO FORMAT O PAGE NO UPDATE VALIDS NO UPDATE SCALES OJUPDATE MONOTONIC ZMODE lt mode gt NO NEG2POSFPO REPORT lt types gt CACHE lt sizes gt NO STATISTICS ABOUT lt cdf path gt UNIX including Mac OS X cd stats no range no fi11 output lt file name gt no format no page no update valids nolupdate scales no update monotonic zmode lt mode gt no neg2posfp0
229. tribute rEntries for the Temperature rVariable from the example above are shown in Table 1 5 The term entry is used when describing a property that applies to gEntries rEntries and zEntries vAttribute rEntry value FIELDNAM Recorded temperature VALIDMIN 40 0 VALIDMAX 50 0 SCALEMIN 17 0 SCALEMAX 24 0 UNITS deg C FORMATS F4 1 MONOTON Increasing FILLVAL 999 9 Table 1 5 vAttribute rEntries for the Temperature rVariable 1 7 CDF Toolkit A set of utility programs are provided with the CDF distribution which allow a user to perform a variety of operations on CDFs without having to write an application program Each toolkit program is described in detail in Chapter 3 The available toolkit programs are as follows CDFedit Allows the display creation and modification of attribute and variable data ina CDF CDFexport Allows the contents of a CDF to be exported to the terminal screen a text file or another CDF The CDF may be filtered in order to export a subset of its contents CDF convert Allows the format encoding majority compression and sparseness of a CDF to be changed It also can reorganize a fragmented CDF file to make the file access more efficiently In all cases a new CDF is created The original CDF is not modified CDFedit has replaced CDFbrowse The alias symbol CDFbrowse still exists in the definitions file on UNIX VMS systems but now executes CDFedit in
230. ttribute entry for a character data type For non character data types the value is not enclosed in braces the braces are not necessary because there can only be one element The format for CDF EPOCH values is described in Section 2 5 4 The vAttribute rEntries are optional If omitted the terminating period is still required The rVariable values are also optional Several sample rVariable definitions for a CDF with 2 dimensional rVariables follow Variable Name Latitude Attribute Name VALIDMIN VALIDMAX scale 1 1 60 0 1 2 30 0 13 00 1 4 30 0 1 5 60 0 HU A A E man Variable Name EPOCH Attribute Name Variable Name Data Number Record Dimension Type Elements Variance Variances CDF_REAL4 1 F FT Data Type Value CDF_REAL4 90 0 CDF_REAL4 90 0 CDF_REAL4 60 0 60 0 Data Number Record Dimension Type Elements Variance Variances CDF_EPOCH 1 F F F Data Type Value CDF_REAL4 10 Oct 1991 00 00 00 000 20 Oct 1991 23 59 59 999 Data Number Record Dimension Type Elements Variance Variances 116 Tmp CDF_INT2 1 T TT Attribute Data Name Type Value paent aaan RE E Fieldname CDF_CHAR Temperature C Variable Data Number Record Dimension Name Type Elements Variance Variances PNA AA a ELA SESE pres_lv1 CDF_REAL4 1 T F F Attribute Data Name Type Value no attribu
231. ubject Software and includes derivative works as that term is defined in the Copyright Statute 17 USC 101 However the act of including Subject Software as part of a Larger Work does not in and of itself constitute a Modification G Original Software means the computer software first released under this Agreement by Government Agency with Government Agency designation GSC 14272 and entitled Coordinated Data Analysis workshop Web CDA Web including source code object code and accompanying documentation if any Recipient means anyone who acquires the Subject Software under this Agreement including all Contributors Redistribution means Distribution of the Subject Software after a Modification has been made Reproduction means the making of a counterpart image or copy of the Subject Software Sale means the exchange of the Subject Software for money or equivalent value Subject Software means the Original Software Modifications or any respective parts thereof Use means the application or employment of the Subject Software for any purpose T AS 2 GRANT OF RIGHTS A Under Non Patent Rights Subject to the terms and conditions of this Agreement each Contributor with respect to its own contribution to the Subject Software hereby grants to each Recipient a non exclusive world wide royalty free license to engage in the following activities pertaining to the Subject Software Use Distribution Reproduction Modificatio
232. uffer received provided by the application are contiguous Hyper access is sensitive to the record and dimension variances of a variable For instance if a variable has a record variance of NOVARY with one record written and a hyper read of the first five records for that variable is requested the CDF library will read the single record that is physically stored and place it five times contiguously into the memory buffer provided by the application The same applies to any dimension variances that are NOVARY For example if the count for a dimension is three and the dimension variance is NOVARY the one value or subarray physically stored will be read by the CDF library and placed into the application s memory buffer three times contiguously Example Fortran application 50 Assume a 2 dimensional variable array with sizes 2 4 row majority a record variance of VARY dimension variances of VARY VARY and hyper read parameters as follows record number 5 record count 2 record interval 1 dimension indices 1 1 dimension counts 2 4 dimension intervals 1 1 The values placed in the application s buffer would be as follows with the first value being in low memory 54 5 5 1 2 5 1 3 30 4 30 1 5 2 2 5 2 3 5 2 4 6 1 1 6 1 2 6 1 3 6 1 4 62 1 6 2 2 6 2 3 6 2 4 where r i j is a physically stored value with r being the record number 1 being the first dimension index and j being the second dimension index r i and
233. un Solaris DECstation ULTRIX Silicon Graphics Iris amp Power Series IRIX IBM RS6000 series AIX HP 9000 series HP UX PC MS DOS Windows NT 95 98 2000 XP Linux amp QNX NeXT Mach DEC Alpha OSF 1 amp OpenVMS 10 Macintosh Mac OS X D 2 CDF V3 1 is backward compatible with the previous versions of CDF and it can read CDF files that were created with CDF 3 0 or CDF 2 7 2 or earlier If a file was created with CDF 2 7 and read and modified by CDF 3 1 the resultant file will be saved in the CDF 2 7 format not CDF 3 1 The same principle applies to files that were created with CDF 2 5 and 2 6 CDF files that are created from scratch with CDF V3 1 are compatible with CDF 3 0 but not compatible due to a 64 bit file offsets used in CDF 3 0 or later versions with CDF 2 7 2 or earlier and an attempt to read CDF 3 0 Compatibility with CDF 2 7 2 and Earlier Versions or 3 1 files from CDF 2 7 2 or earlier will produce an error 22 Due to lack of user s interest and hardware this operating system is not tested If you need to run the CDF V3 1 library on either HP UX or IBM s AIX operating system please contact the CDF support office at gsfc cdf support lists nasa gov 137 Users of CDF 3 0 or later versions will be able to create CDF files that can be read by CDF 2 7 2 or earlier by using the CDFsetFileBackward function in C or CDF set FileBackward subroutine in Fortran or using the C
234. uture CDF release When a variable is specified as having sparse records only those records actually written to that variable will be stored in the CDF This differs from variables without sparse records in that for those variables every record preceding the maximum record written is stored in the CDF For example if only the 1000th record were written to a variable without sparse records the 999 preceding records would also be written using a pad value If sparse records had been specified for the variable only the 1000th record would be stored in the CDF saving a considerable amount of disk space Sparse records are ideal for variables containing gaps of missing data 1 4 5 Variable Data Access Options A program can access variable data one value at a time or it can access an entire multidimensional array structure or substructure spanning contiguous or non contiguous record boundaries The latter feature allows the user to perform aggregate access or uniform subsampling of the data at greatly increased rates over traditional value by value access 7 If compressed data turns out to be larger than uncompressed data it makes no sense to use compression thus variable data will be stored in a CDF uncompressed even if a compression is requested 14 1 5 Organizing Your Data in a CDF 1 5 1 Variables The first component of a CDF is the actual data organized into arrays for the individual variables CDF can accommodate any type of data tha
235. variance of NOVARY false are removed NO NEG2POSFP0 VMS no neg2posfp0 UNIX Specifies whether or not 0 0 is converted to 0 0 by the CDF library when encountered in a CDF 0 0 is an illegal floating point value on VAXes and DEC Alphas running OpenVMS REPORT lt types gt VMS report lt types gt UNIX Specifies the types of return status codes from the CDF library that should be reported displayed The lt types gt option is a comma separated list of zero or more of the following symbols errors warnings or informationals Note that these symbols can be truncated e g e w and i CACHE lt sizes gt VMS cache lt sizes gt UNIX Specifies the cache sizes to be used by the CDF library for the dotCDF file and the various scratch files The lt sizes gt option is a comma separated list of lt size gt lt type gt pairs where lt size gt is a cache size and lt type gt is the type of file as follows d for the dotCDF file s for the staging scratch file and c for the compression scratch file For example 200d 100s specifies a cache size of 200 for the dotCDF file and a cache size of 100 for the staging scratch file The dotCDF file cache size can also be specified without the d file type for compatibility with older CDF releases e g 200 100s Note that not all of the file types must be specified Those not specified will receive a default cache size chosen by the CDF library A cache size is the number o
236. ve lines However certain syntax rules do apply to skeleton tables 1 Lines are limited to 132 characters 2 Keywords for the header section gAttributes section vAttributes section rVariables section and end section must always be specified in that order The zVariables section is optional its keyword may be omitted 3 An exclamation point at any point signifies a comment until the end of the line Any characters encountered after the exclamation point will be ignored An exclamation point may begin a line making the entire line a comment Exclamation points inside delimited character strings are part of the string and do not cause the start of a comment 4 Attribute and variable names must be delimited Any character not in the name may be used as the delimiter with the following exceptions a Do not use an exclamation point to delimit an attribute or variable name b Do not use a period to delimit an attribute name in the variables section c Do not use a left square bracket or a numeral to delimit a variable name 5 When specifying a character string attribute entry value do not use a hyphen to delimit the string or strings if the string is split across one or more lines 6 All items are referenced from one 1 These include gAttribute gEntry numbers and NRV variable index values In the descriptions that follow optional fields are shown in brackets A 2 Header Section 109 T
237. w major increment the last index the fastest 4 When performing a multiple variable read write the full physical records in the buffer will must be in the variable majority of the CDF A CDF s variable majority is specified when the CDF is created when using the Standard Interface but is set to the default variable majority for your CDF distribution when created using the Internal Interface The majority of an existing CDF may be changed using the Internal Interface only if variable values have not yet been written Variables may exist and explicit pad values may have been specified however 2 3 16 Single Value Access Single value access allows only one value to be read from or written to a variable with a single call to the CDF library Two parameters are specified when performing a single value read write RecordNumber The record number at which to perform the access DimensionIndices The indices within the record at which to perform the access For 0 dimensional variables the dimension indices are not applicable Single value access is sensitive to the record and dimension variances of a variable For instance if a variable has a record variance of NOVARY with one record written and a value is read from the fourth record the CDF library will actually read the value from the first record the record that is physically stored If a value were written to the fourth record the CDF library would actually write the value to the first r
238. with respect to the CDF dimensions In the example above with 2 dimensional rVariables the Longitude rVariable defines the first dimension of the CDF with its values repeating along the second dimension so its dimension variances would be TRUE false The Latitude rVariable defines the second dimension of the CDF with its values repeating along the first dimension so its dimension variances would be false TRUE Because the Temperature values change for each latitude longitude location its dimension variances are TRUE TRUE Time does not change from one latitude longitude location to another so its values are the same along both dimensions The dimension variances for Time would be false false The dimension variances for the above example are shown in Table 1 3 rVariables Time Longitude Latitude Temperature Record Variance TRUE false false TRUE First Dimension Variance false TRUE false TRUE Second Dimension Variance false false TRUE TRUE Table 1 3 Example CDF Specification for 2 Dimensional Representation When the record and dimension variances have been defined correctly the amount of physical storage needed for the CDF is drastically reduced In the above example 2 dimensional arrays are not physically stored for each rVariable in a CDF record Instead the physical storage for each rVariable consists of just one value for Time in each CDF record a single 1 dimensional array of values for the Longitude and Latitude rVari
239. write to a scratch file error from file system Error The specified operation is not applicable to CDFs with the single file format For example it does not make sense to close a variable in a single file CDF Informational Some of the records being allocated were already allocated Informational A type of sparse arrays or compression was encountered having too many parameters This could be causes by a corrupted CDF or if the CDF was created modified by a CDF distribution more recent than the one being used Error A multi file CDF on a PC may contain only a limited number of variables because of the 8 3 file naming convention of MS DOS This consists of 100 rVariables and 100 zVariables Error An unknown type of compression was specified or encountered Error An unknown type of sparseness was specified or encountered Error 134 UNSUPPORTED_OPERATION VAR_ALREADY_CLOSED VAR_CLOSE_ERROR VAR_CREATE_ERROR VAR_DELETE_ERROR VAR_EXISTS VAR_NAME_TRUNC VAR_OPEN_ERROR VAR_READ_ERROR VAR_WRITE_ERROR VIRTUAL_RECORD_DATA The attempted operation is not supported at this time Error The specified variable is already closed Informational Error detected while trying to close variable file Check that sufficient disk space exists for the variable file and that it has not been corrupted Error An error occurred while creating a variable file in a multi file CDF Check that a file quota has not
240. ximum number of characters in the name of a file used to specify a CDF Most of these limits can be raised Contact CDF User Support if that becomes necessary 2 3 Variables CDF s variable is a generic name for an object that represents data where data can be 0 dimensional scalar data or multi dimensional up to 10 dimension and it does not have any scientific context associated with it For example a 27 Compression is not allowed with multi file CDFs 2 Previous releases of CDF limited the number of variables a CDF could contain That limit has been eliminated except for multi file CDFs on an PC because of the 8 3 naming convention 37 variable can be data representing an independent variable a dependent variable time and date value or whatever data might be e g image XML file etc In other words a variable doesn t contain any hidden meanings other than the data itself One may describe one variable s relationship with other variable s through attributes There are two types of variables rVariable and z Variable in CDF and they can happily coexist in a CDF Every rVariable in a CDF must have the same number of dimensions and dimension sizes while each zVariable can have its own dimensionality Since all the rVariables in a CDF must have the same dimensions and dimension sizes there ll be a lot of disk space wasted if a few variables need big arrays and many variables need small arrays Since zVariable is more efficient
241. y be specified for a gAttribute and there are no restrictions on the gEntry numbers that may be used except that they must be greater than zero The data type for the gEntry The data type must be one of the following CDF_BYTE CDF INT1 CDF_UINT1 CDF INT2 CDF_UINT2 CDF_INT4 CDF_UINT4 CDF_INT8 CDF_REAL4 CDF_FLOAT CDF_REAL8 CDF_DOUBLE CDF_EPOCH CDF_EPPCH16 CDF_TIME_TT2000 CDF_CHAR or CDF_UCHAR The lt data type gt field is optional for all but the first gEntry specified If omitted the data type of the previous gEntry is assumed The value s for the gEntry A period follows the value s of the last gEntry for a gAttribute Attribute Entry Values An attribute entry can have more than one element of the specified data type For character data types CDF_CHAR and CDF_UCHAR each character is the element of a string The character string must be delimited with a character not appearing in the string itself and the entire delimited string must be enclosed in braces e g The CDF title If the string will not fit on one line it may be continued on additional lines The substrings are each delimited with a unique character and a dash is placed at the end after the terminating delimiter of each line except the last one For example This is a longer 112 CDF title that will not fit on one line For non character data types the elements are enclosed in braces and separated by commas
242. y not make any representation in the Subject Software or in any promotional advertising or other material that may be construed as an endorsement by Government Agency or by any prior Recipient of any product or service provided by Recipient or that may seek to obtain commercial advantage by the fact of Government Agency s or a prior Recipient s participation in this Agreement In an effort to track usage and maintain accurate records of the Subject Software each Recipient upon receipt of the Subject Software is reguested to provide Government Agency by e mail to the Government Agency Point of Contact listed in clause 5 F the following information Name and Affiliation Recipient s name and personal information shall be used for statistical purposes only Once a Recipient makes a Modification available it is reguested that the Recipient inform Government Agency by e mail to the Government Agency Point of Contact listed in clause 5 F how to access the Modification Each Contributor represents that that its Modification is believed to be Contributor s original creation and does not violate any existing agreements regulations statutes or rules and further that Contributor has sufficient rights to grant the rights conveyed by this Agreement A Recipient may choose to offer and to charge a fee for warranty support indemnity and or liability obligations to one or more other Recipients of the Subject Software A Recipient may do so however onl
243. y on its own behalf and not on behalf of Government Agency or any other Recipient Such a Recipient must make it absolutely clear that any such warranty support indemnity and or liability obligation is offered by that Recipient alone Further such Recipient agrees to indemnify Government Agency and every other Recipient for any liability incurred by them as a result of warranty support indemnity and or liability offered by such Recipient A Recipient may create a Larger Work by combining Subject Software with separate software not governed by the terms of this agreement and distribute the Larger Work as a single product In such case the Recipient must make sure Subject Software or portions thereof included in the Larger Work is subject to this Agreement Notwithstanding any provisions contained herein Recipient is hereby put on notice that export of any goods or technical data from the United States may require some form of export license from the U S Government Failure to obtain necessary export licenses may result in criminal liability under U S laws Government Agency neither represents that a license shall not be required nor that if required it shall be issued Nothing granted herein provides any such export license 4 DISCLAIMER OF WARRANTIES AND LIABILITIES WAIVER AND INDEMNIFICATION A No Warranty THE SUBJECT SOFTWARE IS PROVIDED AS IS WITHOUT ANY WARRANTY OF ANY KIND EITHER EXPRESSED IMPLIED OR STATUTORY INCLUDING
244. yed on the screen This option redirects the output to a designated file indicated by lt file path gt If lt file path gt does not have an extension dmp is appended at the end of lt file path gt NO PAGE VMS no page Unix Windows Specifies whether or not a page breaker is on when displaying the data on the screen The default is no page breaker BRIEF FULL gt VMS brief full Unix Windows Specifies how the program should produce a dump whether it will display the full hex dump of all internal records or just the summary The default is full dump NO SUMMARY VMS no summary Unix Windows 105 Specifies whether or not the summary is to be displayed The default is to display the summary ABOUT VMS about Unix Windows Shows the library version that was used to create this tool program Example s VMS CDFirsDUMP my data cdf UNIX and Windows o cdfirsdump my_data cdf Help is displayed when CDFirsdump is executed without any arguments 3 14 CDFvalidate 3 14 1 Introduction The CDFvalidate program optionally performs sanity checks on certain data in the CDF files The program goes through the Internal Records IRs which construct a CDF file and tries to detect if the file is compromised The relevant data fields in IRs are checked against its range and predefined values The variable data values however are not checked as there are no criteria to chec
245. ys Each rVariable in a CDF has the same dimensionality An array of values exists for each rVariable at each record in a CDF The values may not be physically stored but may be virtual A zVariable may have a dimensionality that is different from that of the rVariables and the other zVariables An array of values exists for each zVariable at each record in a CDF As with rVariables the values may not be physically stored but may be virtual zVariables are intended for use in those situations where using an rVariable would waste disk space or not logically make sense A variable array having two or more dimensions also contains subarrays For instance in a 3 dimensional array with dimension sizes 10 20 30 each array consists of ten 2 dimensional subarrays of size 20 30 and each of those 2 dimensional subarrays consists of twenty 1 dimensional subarrays of size 30 Subarrays will be referred to when discussing other properties of CDF variables 2 3 9 Data Specification Each variable in a CDF has a defined data specification A variable s data specification consists of a data type and a number of elements of that data type A variable s data specification is specified when the variable is created The data specification of an existing variable may also be changed if either of the following conditions is true 1 Values have not yet been written to the variable including an explicitly written pad value see Section 2 3 20 2 The old
246. zVariable Record zVariable Number 1 00007 JO 13300 MOO 111111 13302 Moo 111 11 1 00000 ol LoL 13300 2 oo ooo LIL 11 11 11 13300 Moo 3 oo 000 Figure 1 3 Conceptual View of a CDF zVariables 12 example cdf gt 3 gt U gt 0m2Z example v0 rors example v1 example v2 D D A A T T A A example v3 gt gt Uu Figure 1 4 Multi File Format example cdf M E T A D A T A amp ARU Figure 1 5 Single File Formats 13 1 4 2 Data Encoding Options When creating a CDF a user has the option of using any of the supported encodings VAX Sun SGi Personal Iris and Power Series DECstation DEC Alpha OSF1 DEC Alpha OpenVMS D FLOAT G FLOAT or IEEE FLOAT double precision floating point IBM RS6000 series HP 9000 series NeXT PC Macintosh or network XDR eXternal Data Representation The created CDF may then be copied to any of the supported computers and read by the CDF library When a value is read from the CDF the CDF library may be requested to decode the value into the encoding of the computer being used or any of the other encodings which may be desirable for various reasons A CDF with any ofthe sup
247. zVariable is available via the Internal Interface Figure 2 2 illustrates the relationships between physical and virtual records for a standard variable Variable has five records that were physically written Only two records were physically written to variable 2 so the following records are virtual containing the pad value for that variable Only one record can be physically written to variable 3 because its record variance is NOVARY The other records are virtual and contain the same values as the first record Because a record has not been physically written to variable 4 all of its records are virtual containing the pad value for that variable Likewise since no records have been written to variable 5 all of its records are also virtual and contain the pad value for that variable 42 rVariable 1 rVariable 2 rVariable 3 rVariable 4 rVariable 5 VARY VARY NOVARY NOVARY VARY E EI LD 7 E 0 LL AK A KK AKI Jt JI JE IL physical record virtual record Figure 2 2 Physical vs Virtual Records Standard Variable Note that a variable s records do not have to be written sequentially starting at the first record The records may be written in any order For a variable not having sparse records with a VARY record variance if a new record more than one record beyond the current maximum record for the variable is written the intervening records will be physically written and contain the pad value for that varia

CDF User's Guide (V3.5.0) - CDAWeb

Contents

Download Pdf Manuals

Related Search

Related Contents