Home

world climate programme - water development, use and

1. te CHAPTER 5 Saving and sharing your work 1 Hydrospect documents To save the current state of the analysis performed with Hydrospect you can choose Save or Save as from the File menu or click on the diskette button on the toolbar or press Ctrl 5 Then you can start working from scratch again by choosing New from the File menu or clicking on the leftmost button on the toolbar or pressing Ctrl N To restore your saved file choose Open from the File menu or click on the open folder button on the toolbar or press Ctrl O Saved Hydrospect files are typically very small so it is easy to share them using diskettes or e mail Of course if such a file is transferred to another computer and opened with Hydrospect there all the data files used in the analysis will have to be present on the other computer as well They should be exactly the same to make sure that all the options are interpreted correctly If Hydrospect cannot find a data file in fact it is not likely to find it exactly in the same directory on a different computer the missing time series is displayed in boldface in the left pane Clicking such an entry with the right mouse button and choosing Options from the context menu or selecting it and pressing Enter allows you to pick the new data file location as explained in Chapter 2 2 Reporting the results The Report menu entry allows you to create a text file containing the results of all the tests performed T
2. WORLD CLIMATE PROGRAMME WATER DEVELOPMENT USE AND APPLICATION OF THE HYDROSPECT DATA ANALYSIS SYSTEM FOR THE DETECTION OF CHANGES IN HYDROLOGICAL TIME SERIES FOR USE IN WCP WATER AND NATIONAL HYDROLOGICAL SERVICES Report by Dr Maciej Radzeijewski and Professor Zbigniew W Kundzewicz WCASP 65 WMO TD No 1240 Poznan June 2004 UNITED NATIONS EDUCATIONAL WORLD METEOROLOGICAL SCIENTIFIC AND CULTURAL ORGANIZATION ORGANIZATION The HYDROSPECT software and the manual can be downloaded from the following ftp site ftp www wmo int documents hwr hydrospect zip Comments with regard to the HYDROSPECT software should be directed to Dr Maciej Radziejewski Research Centre for Agricultural and Forest Environment Polish Academy of Sciences ul Bukowska 19 60 809 Poznan Poland E mail maciejr amu edu pl NOTE The designations employed and the presentation of material in this document do not imply the expression of any opinion whatsoever on the part of the Secretariat of the World Meteorological Organization concerning the legal status of any country territory city or area or of its authorities or concerning the delimitation of its frontiers or boundaries Editorial note This report has been produced without editorial revision by the WMO Secretariat It is not an official WMO publication and its distribution in this form does not imply endorsement by the Organization of the ideas expressed Development use and appli
3. dent or are seasonal lend themselves well to application in hydrological context The se lection of a test from the available set of methods e g present in Hydrospect for a particular situation properties of data can follow the guidelines encapsulated in the following table from Kundzewicz amp Robson 2004 The present Report contains the User s Manual to the Hydrospect software package version 2 0 as Appendix It is expected that the Hydrospect package will be of considerable use in the World Climate Programme Water and in National Hydrolo gical Services of WMO Member Countries References Kundzewicz Z W 2004 Searching for change in hydrological data Hydrol Sci J 49 1 3 6 Kundzewicz Z W amp Robson A J 2004 Change detection in river flow records review of methodology Hydrol Sci J 49 1 7 19 Radziejewski M amp Kundzewicz Z W 2000 Hydrospect So ftware for detecting changes in hydrological data Appendix 2 in Kundzewicz Z W amp Robson A J eds Detecting Trend and Other Changes in Hydrological Data World Climate Programme Water World Climate Programme Data and Monitoring WCDMP 45 WMO TD no 1013 World Meteorological Organization Geneva Switzerland Appendix User s Manual Hydrospect Version 2 0 Maciej Radziejewski 8 Zbigniew W Kundzewicz June 2004 Contents Chapter 1 Overview of Hydrospect 1 New features in version 2 0 2 System requirements and install
4. The box Data preview contains initial data lines in the file You can select a column with the time series you wish to study and as many date time columns as you like by clicking on the headings in this box Possible selections are time data and skip Note that for some columns Hydrospect cannot determine if they contain time series values or date time info so both choices time and data are available It is best to select all the relevant date time entries but you will only be able select one column for time series values data Below the box Hydrospect displays a summary of unused comment lines if any Presence of such lines may indicate that valid information could not be read or that the file is corrupted particularly if they are present in the middle of the file You can view the unused comment lines at the beginning and within the file by clicking on the underlined parts of the summary If you find that a valid data line is to be ignored it means that formatting of data is not uniform so you may need to reformat your data or that the file contains an error in that case you may try to correct it Hydrospect displays each ignored line with a line number at the beginning The last option is to specify a special code denoting a missing value It must be a valid floating point or integer number Hydrospect will check for this value using literal i e textual not numerical compari sons so there is no danger of accidental misinterpr
5. then the test statistic equals n 2 W Vilqzya Wa yee a 1 lt k lt n 1 g i n i m is the mean of time series values and c is the standard deviation If there are missing values in the series the Worsley s likelihood ratio test is applied to the non missing values as if they were conse cutive in the series There is also an option to compute the equivalent V statistic The change point estimate is also displayed as an option A table of percentage points for W for series of 50 observations or shorter under the null hypothesis of independent normally distribu ted observations is given in 16 This test is particularly useful for detection of abrupt step change in time series where k Y a m 1 i 1 8 Kruskal Wallis test This is a non parametric test for equ ality of subperiod means It was introduced in 6 and described also in 2 and 14 In Hydrospect this test can only be applied to a series of ranks or ranked deviations from the median so the raw data need to be pre processed and ranks calculated Please refer to Section 2 of Chapter 4 page 31 for details on computing ranks with Hydrospect This test is based on partitioning the time series to subperiods of possibly differing lengths For each subperiod the mean of ranks of values within the subperiod is calculated If n is the series length m the global mean of ranks N the number of subperiods n the number of values in the i th subperiod a
6. tion 1 of Chapter 4 page 26 e compute a POT series see Section 4 of Chapter 4 page 32 e compute ranks and normal scores remove the annual cycle select a sub sequence etc e study changes in various characteristics of a time series using statistical tests and resampling You may also like to use Hydrospect in conjunction with other tools using only a subset of its features for example you may extract data from a text file in an unusual format using Hydrospect s robust data im port mechanism or compute annual indices of extremes or POT series and save the resulting time series in a simple text file for further study with another application Hopefully it will make a useful addition to your toolbox Hydrospect will perform computations efficiently and without con suming more system resources than necessary It includes numerous extra features that do not make the user interface more complicated and turn out very useful in practice For example 1 You can enter fractional parameters in many places where you would normally expect only integers would be allowed For instance when you compute annual means out of a daily time series not tagged by any dates you may just enter 365 25 as period length and Hydrospect will use periods of length 365 and 366 in turns so that the mean period length will indeed be 365 25 1 NEW FEATURES IN VERSION 2 0 7 2 Hydrospect recognizes and properly handles missing values and ti
7. to decide which columns in the file contain the time series you wish to study The details are given below 1 Data file format Hydrospect can read data from text files The import feature has been designed for flexibility so that it will most probably be able to read your existing data files without special preparation The requ irement is that the lines containing time series data dates and or times and values follow a uniform pattern That is each of them has the same number of values and dates in the same order and corresponding dates are in the same format i e consist of the same number of fields and use the same separators In fact a date does not have to look like 1972 05 21 or 1972 21 05 It can also be 1972 21 05 09 30 54 or in genera any sequence of natural num bers separated by any of the following characters ff with one exception two numbers separated by a dot like 1972 05 do not form a legitimate date since they might be confused with a decimal fraction You can also use multiple date time entries so 1972 21 05 09 30 54 is ok too Finally an integer number can also 9 1 DATA FILE FORMAT 10 be treated as denoting date time information This allows using da tes like 1987 11 30 An index denoting e g the number of days since January the 1 1900 is also a valid date field Time series values must be specified as integer or floating point numbers Exponential notation using e o
8. 2 an l where m is the mean of time series values and is the standard de viation Significance level is computed according to the limit distribution valid for long time series given in 1 For series of 100 observations or shorter Buishand 1 gives a table of percentage points for the statistic divided by the square root of the series length This measure can also be computed here as an option so the significance can be easily looked up in the table The same table is reproduced in 14 although it appears under a different heading Please note that you do not need to use such a table if you compute the significance using resampling cf Section 4 of Chapter 3 page 24 1 TESTS FOR CHANGES 20 If there are missing values in the series the Cumulative deviations test is applied to the non missing values as if they were consecutive in the series Both the limit distribution significance estimate and the tables of percentage points are valid under the null hypothesis of independent normally distributed observations Another option is whether or not to display the estimated change point i e the point k where the accumulated deviation from the mean was largest 1 7 Worsley s likelihood ratio This is a parametric test defi ned in 16 and described also in 14 2 and 1 The test statistic is denoted as W in each of the above sources If n is the length of the time series and Q2 denote the time series values
9. 609 out of print REPORT OF THE TECHNICAL CONFERENCE ON TROPICAL URBAN CLIMATES TeCTUC Dhaka Bangladesh 28 March 2 April 1993 WMO TD No 647 REPORT OF THE FIRST SESSION OF THE CCI WORKING GROUP ON OPERATIONAL USE OF CLIMATOLOGICAL KNOWLEDGE Vacoas Mauritius 22 26 November 1994 WMO TD No 663 out of print REPORT FROM THE MEETING OF EXPERTS ON CLIMATE INFORMATION amp PREDICTION SERVICES CLIPS Melbourne Australia 28 to 31 March 1995 WMO TD No 680 out of print REPORT FROM THE MEETING OF EXPERTS ON CLIMATE TOURISM AND HUMAN HEALTH Topes de Collantes Cuba 22 29 January 1995 WMO TD No 682 out of print REPORT OF THE TENTH SESSION OF THE ADVISORY WORKING GROUP OF THE COMMISSION FOR CLIMATOLOGY Geneva 20 22 September 1995 also appears as WCDMP 24 WMO TD No 711 out of print REPORT OF THE FIFTH SESSION OF THE ADVISORY COMMITTEE ON CLIMATE APPLICATIONS AND DATA ACCAD Geneva 26 September 1995 also appears as WCDMP 25 WMO TD No 712 out of print BIBLIOGRAPHY OF URBAN CLIMATOLOGY FOR THE PERIOD 1992 1995 Prepared by Professor E Jau regui CCl Rapporteur on Urban Climatology May 1996 WMO TD No 759 REPORT OF THE SECOND SESSION OF THE CCI WORKING GROUP ON OPERATIONAL USE OF CLIMATOLOGICAL KNOWLEDGE Geneva 28 31 May 1996 and REPORT OF THE MEETING OF EXPERTS ON CLIPS Geneva 22 24 May 1996 WMO TD No 774 ECONOMIC AND SOCIAL BENEFITS OF CLIMATOLOGICAL INFORMATION AND SERVICES A REVIEW
10. 8 56 1999 14 02 41 9 7 8 1999 11 03 8 0 5 0 Hydrospect will recognize the date time columns and suggest the first likely time series values column Tmax in this case You can choose to import either Tmax or Tmin Example 3 Data from a regional climate model is stored in large files Each file contains daily Tmax temperatures for 360 days in 11766 ground level cells of the model the cells form a rectangular grid 106 x 111 in the following order first all the data for the first day then for the second day and so on The data for each day were arranged to 10 columns wrapped for easy handling with Fortran programs therefore each day s data consists of 1176 lines with 10 values each and one more line with 6 values Each day s data is preceded by a header Year 2070 Months 1 Days 1 Daily max temperatures K 11766 Format is 10F8 2 missing value is 9999 99 290 00 291 00 292 00 280 00 293 00 295 00 294 00 292 00 292 00 291 00 291 00 292 00 280 00 293 00 295 00 294 00 292 00 292 00 291 00 290 00 292 00 291 00 290 00 291 00 292 00 280 00 293 00 295 00 294 00 292 00 292 00 280 00 293 00 295 00 294 00 292 00 Year 2070 Months 1 Days 2 Daily max temperatures K 11766 Format is 10F8 2 missing value is 9999 99 280 00 293 00 295 00 294 00 292 00 292 00 291 00 290 00 291 00 292 00 Suppose you were asked to extract data for just one cell say cell No 5378 from such a data file In other words you wish to extract the va
11. ON USER REQUIREMENTS AND NEED FOR DEVELOPMENT Reports of the CCl rapporteurs on Users Requirements and Publicity F Singleton and New Approaches in Applications D W Philips to the tenth session of the Commission for Climatology Lisbon April 1989 WMO TD No 281 WCAP 7 DROUGHT AND DESERTIFICATION Report of the CCl Rapporteur on Drought and Desertification in Warm Climates to the tenth session of the Commission for Climatology Lisbon April 1989 L J Ogallo and lectures presented at the training seminar in Mufia Philippines 14 24 November 1988 by N Gbeckor Kove WMO TD No 286 out of print WCAP 8 REPORT OF THE FIRST SESSION OF THE CCI WORKING GROUP ON CLIMATE AND URBAN AREAS INCLUDING BUILDING AND OTHER ASPECTS AND SOME RELATED PAPERS by Professors E Jauregui and Shen Jianzhu Members of the Working Group WMO TD No 287 WCAP 9 REPORT OF THE EXPERT MEETING ON CLICOM CLIMATE APPLICATIONS INCLUDING CARS Geneva 6 10 November 1989 WMO TD No 336 WCAP 10 URBAN DESIGN IN DIFFERENT CLIMATES by B Givoni University of California U S A WMO TD No 346 out of print WCAP 11 FIFTH PLANNING MEETING ON WORLD CLIMATE PROGRAMME WATER Laxenburg Austria 30 April 4 May 1990 WMO TD No 374 out of print WCAP 12 IMPACT POSSIBLE DES CHANGEMENTS CLIMATIQUES A VENIR SUR LES RESSOURCES EN EAU DES REGIONS ARIDES ET SEMI ARIDES par Jacques Sircoulon ORSTOM Paris France June 1990 WMO TD No 380 out of print W
12. a value which is the largest in the neighbourhood based on the time distance criteria you supply The threshold is computed to achive a required number of independent peaks per year or another time period as necessary Hydrospect computes the threshold based on your input and displays it in the POT dialog box Time period can be defined in the same way as by aggregation The resulting POT series will include date time info from the origi nal if periods were defined based on this information If you chose to use a specific period length Hydrospect will display the period num ber and the number of the value within the period The values for the peaks will also be displayed This way you obtain information when the peaks occurred and what their magnitude was Other series values in the POT series are treated as missing You can apply a statistical test directly to the POT series or aggre gate it for example count the peaks per year Aggregation applied to the POT series behaves differently the missing values data quality criteria is verified against the original series not the POT series 5 Selecting a subseries Menu entry Select a subseries A subseries of the time series of concern can be created by restricting to a certain time range You can enter the position of the first and of the last observation to be included in the subseries in the From and To fields at the top of the dialogue box If you leave any of those blank the
13. choice to the list To add a test to the list e select a test from the Test menu or e click on the list of tests with the right mouse button and choose a test from the context menu If a menu item is greyed the particular test is not available for the time series presently selected Some tests can only be applied to a series of ranks or ranked deviations from the median Please look at the test s description in one of the following sub sections before using it Most tests will be added to the list with default options so you will not be bothered with a dialogue box every time you add a test To change the options you can click on the appropriate list entry with the right mouse button and choose Options from the context menu that appears Alternatively you can double click the list entry or after the list entry is selected you can press Enter to access its options You can also press the Delete key or choose Delete from the context menu to remove the test from the list If you wish to remove all the tests performed for this time series follow the same procedure for the heading of the list Brief descriptions of the tests available in Hydrospect are given in the following sub sections For more detailed information about testing for changes including the theoretical foundations and detailed discussion of the properties and applicability of various tests please refer to 7 and to the original sources cited in the text A short revi
14. compute the length of the longest spell of ice days with day temperature below 0 C in a year using the Longest consec below carry over index If such a spell happens to last from Dec 19 1987 to Jan 14 1988 the recorded length for the year 1987 would be 13 days Dec 19 Dec 31 unless there was a longer spell earlier the same year On Jan 14 1988 the recorded length would already be 13 14 27 days Dec 19 1987 Jan 14 1988 so the index value for the year 1989 would be 27 unless a longer spell of ice days happened later that year If you do not like the effect that some of the ice days would be taken into account twice you can use the Longest consec below index without the carry over suffix The non carry over Maz consec indices take into account only the periods of consecutive days placed entirely within the boundaries of a given year Neither option is recommended over the other it is your decision which one to choose In fact in the situation described here when the ice periods studied are likely to happen mostly in winter it may be more sensible to use yet another solution consider July to June one year long aggregation periods instead of calendar years This can be achieved by deriving a subseries of the original time series see the section on subseries later on starting at the beginning of July on the first year of the time series and then aggregating using periods of length 365 25 days 3 COMP
15. observation If the time series consists of n distinct values the ranks will be the numbers from 1 to n with n corresponding to the highest value in the series n 1 to the second highest and so on If there are ties equal values in the series each value in the tie group is assigned the same mean rank For example if the lowest value occurs four times in the series it is assigned the rank 2 3 4 2 5 and this rank will appear in the four places where the lowest value appeared in the original series The series of ranks has the same length as the original series Ranks are used by some non parametric tests For instance to perform the Mann Kendall s test on a time series you must compute the time series of ranks from it and then apply the test to the ranks series 3 Computing normal scores Menu entry Normal scores The series is transformed in such a way that the marginal distri bution becomes normal with zero mean and unit standard deviation while the relative ranks of the values are preserved More precisely for each value in the series its rank r is computed In the derived series the 5 SELECTING A SUBSERIES 32 original value is replaced by the value whose accumulated probability under the normal distribution is equal to p r 1 n 1 4 Peaks over threshold Hydrospect facilitates Peaks Over Threshold POT analysis by letting you create series of independent peaks An independent peak is defined as
16. options cf the previous section There fore Hydrospect provides a compatible mode in which data are read in exactly the same way as before In any case it is a good idea to review the import options for every time series in a document created by Hydrospect 1 0 and examine the ignored lines to make sure that your data file was correctly read The Standard option is recommended to use unless absolutely ne cessary Using both options interchangeably might result in some data being ignored without warning 4 Viewing the data When the time series is successfully read its name appears in the left pane of the program window and the series itself is presented in the lower pane on the right If the name is not entirely visible you can position the mouse pointer over it and a tool tip will appear showing all of the text When you click on the lower right pane to activate it one of the rows will be selected and the row number will appear on the status bar at the bottom of the window You can move back and forth through the data using the scroll bar and the keyboard Up Down arrow keys PageUp PageDown Home End to make sure the data in the file have been interpreted correctly If there are missing values in the time series they will normally appear as blank cells Presence of missing values activates the toolbar button with the letter M that acts as a switch it lets you hide all missing values in the time series It only
17. run it for a certain time series derive normal scores cf Section 3 of Chapter 4 page 31 and 10 from the series and apply the linear regression test to the derived series The significance level is valid only under the null hypothesis of independent identically distributed observations in the series from which normal scores are derived This test is particularly useful for detection of gradual change in time series 1 TESTS FOR CHANGES 18 1 3 Spearman s rank correlation This non parametric test involves applying linear regression to the series of ranks 8 To com pute it in Hydrospect derive a series of ranks from your time series cf Section 2 of Chapter 4 page 31 and perform the linear regression test The significance estimate is based on the Student s t distribution as in 8 It is valid under the null hypothesis of independent identically distributed series values This test is particularly useful for detection of gradual change in time series 1 4 Mann Kendall s test This is a robust non parametric test based on the tau statistic introduced by Kendall 4 Kendall s tau was adapted by Mann to time series analysis cf 11 and 14 In Hydrospect this test can only be applied to a series of ranks or ranked deviations from the median so the raw data need to be pre processed and ranks calculated Please refer to Section 2 of Chapter 4 page 31 for details on computing ranks with Hydrospect The test
18. test Meacham test This is a non parametric test It involves computing differences between the ranks of subsequent values in the series Let U denote the sum of absolute va lues of such rank differences The test statistic displayed in Hydrospect 1S 3U n 1 V10 z n 23 n 1 4n 7 Under the null hypothesis of independent equally distributed values this statistic is normally distributed with mean 0 and variance 1 see 13 In Hydrospect this test can only be applied to a series of ranks or ranked deviations from the median so the raw data need to be pre processed and ranks calculated Please refer to Section 2 of Chapter 4 page 31 for details on computing ranks with Hydrospect 3 How to interpret results Hydrospect returns the values of the test statistic and of the signi ficance level actually achieved High value of the actual significance level means that the hypothesis of a lack of change is rejected in the light of evidence If the significance level returned by Hydrospect is 99 tantamount to 1 in another convention there is a high chance that a change exists in the hydrological time series subject to analysis A value of significance equal to 99 3 means that a change is detected at the 95 or 5 significance level and at the 99 significance level but not at the 99 9 or 0 1 level Statistical significance does not 4 RESAMPLING 24 always imply that there is a true change or trend in the se
19. 975 13 R Srikanthan T A McMahon and J L Irish Time series analysis of annual flows of Australian rivers J Hydrol 66 213 226 1983 14 WCAP 3 World Climate Application Programme Analysing long time series of hydrological data with respect to climate variability Project Description WMO TD No 224 WMO Geneva 1988 15 W A Woodward and H L Gray Global warming and the problem of testing for trend in time series data J Climate 6 953 962 1993 16 K J Worsley On the likelihood ratio test for a shift in location of normal populations J Am Stat Assoc 74 365 367 1979 37 REPORTS PUBLISHED IN THE WORLD CLIMATE APPLICATIONS PROGRAMME WCAP WORLD CLIMATE APPLICATIONS AND SERVICES PROGRAMME WCASP SERIES WCAP 1 CLIMATE AND HUMAN HEALTH Proceedings of the Symposium in Leningrad 22 26 September 1986 Volume I WCAP 2 CLIMATE AND HUMAN HEALTH Proceedings of the Symposium in Leningrad 22 26 September 1986 Volume II WCAP 3 ANALYZING LONG TIME SERIES OF HYDROLOGICAL DATA WITH RESPECT TO CLIMATE VARIABILITY Project Description WMO TD No 224 out of print WCAP 4 WATER RESOURCES AND CLIMATIC CHANGE SENSITIVITY OF WATER RESOURCE SYSTEMS TO CLIMATE CHANGE AND VARIABILITY Norwich U K November 1987 WMO TD No 247 out of print WCAP 5 FOURTH PLANNING MEETING ON WORLD CLIMATE PROGRAMME WATER Paris 12 16 September 1988 WMO TD No 271 out of print WCAP 6 CLIMATE APPLICATIONS
20. CAP 13 INFORMATION ON METEOROLOGICAL EXTREMES FOR THE DESIGN AND OPERATION OF ENERGY SYSTEMS by G A McKay Consulting climatologist Canada September 1990 WMO TD No 385 out of print WCAP 14 EXTREMES AND DESIGN VALUES IN CLIMATOLOGY by Tibor Farag Hungarian Meteorological Service Budapest Hungary and Richard W Katz National Center for Atmospheric Research Boulder U S A WMO TD No 386 2 WCAP 15 WCAP 16 Note WCASP 17 WCASP 18 WCASP 19 WCASP 20 WCASP 21 WCASP 22 WCASP 23 WCASP 24 WCASP 25 BIBLIOGRAPHY OF URBAN CLIMATE 1981 1988 Prepared by Prof T R Oke Atmospheric Science Programme Department of Geography University of British Columbia Vancouver B C Canada WMO TD No 397 REPORT OF THE WORKSHOP ON A CLICOM HOMS INTERFACE University of Reading U K 6 15 March 1990 WMO TD No 409 Following the change of the name of the World Climate Applications Programme WCAP to World Climate Applications and Services Programme WCASP by the Eleventh WMO Congress May 1991 the subsequent reports in this series will be published as WCASP reports the numbering being continued from No 16 the last WCAP report A NONPARAMETRIC FRAMEWORK FOR LONG RANGE STREAMFLOW FORECASTING by J A Smith G N Day and M D Kane Hydrologic Research Laboratory National Weather Service U S A WMO TD No 428 REPORT OF THE FIRST SESSION OF THE ADVISORY COMMITTEE ON CLIMATE APPLICATIO
21. ICES CLIPS WORKSHOP FOR REGIONAL ASSOCIATION VI Erfurt Germany 12 18 June 2003 WMO TD No 1164 PROCEEDINGS OF THE MEETING ON ORGANIZATION AND IMPLEMENTATION OF REGIONAL CLIMATE CENTRES Geneva Switzerland 27 28 November 2003 WMO TD No 1198 PROCEEDINGS OF THE MEETING OF EXPERTS TO DEVELOP GUIDELINES ON HEAT HEALTH WARNING SYSTEMS Freiburg Germany 14 16 April 2004 WMO TD No 1212 DETECTION OF CHANGE IN WORLD WIDE HYDROLOGICAL TIME SERIES OF MAXIMUM ANNUAL FLOW June 2004 WMO TD No 1239 DEVELOPMENT USE AND APPLICATION OF THE HYDROSPECT DATA ANALYSIS SYSTEM FOR THE DETECTION OF CHANGES IN HYDROLOGICAL TIME SERIES FOR USE IN WCP WATER AND NATIONAL HYDROLOGICAL SERVICES Poznan June 2004 WMO TD No 1240 TRENDS IN FLOOD AND LOW FLOW HYDROLOGICAL TIME SERIES July 2004 WMO TD No 1241 EXPERT MEETING ON HYDROLOGICAL SENSITIVITY TO CLIMATE CONDITIONS CENTRE FOR ECOLOGY AND HYDROLOGY CEH Wallingford UK 2 4 December 2003 WMO TD No 1242
22. ING OF HYDROLOGICAL TIME SERIES WITH RESPECT TO CLIMATE VARIABILITY AND CHANGE Prepared for the World Climate Programme Water Project A2 by George S Cavadias November 1992 WMO TD No 534 out of print TECHNICAL CONFERENCE ON TROPICAL URBAN CLIMATES EXTENDED ABSTRACTS Dhaka Bangladesh 28 March 2 April 1993 WMO TD No 538 out of print BIBLIOGRAPHY OF URBAN CLIMATE IN TROPICAL SUBTROPICAL AREAS 1981 1991 Prepared by Dr E Jauregui CCl Rapporteur on Urban Climatology May 1993 WMO TD No 552 out of print WCASP 26 WCASP 27 WCASP 28 WCASP 29 WCASP 30 WCASP 31 WCASP 32 WCASP 33 WCASP 34 WCASP 35 WCASP 36 WCASP 37 WCASP 38 WCASP 39 WCASP 40 HYDROLOGICAL DESIGN DATA ESTIMATION TECHNIQUES Prepared by Oldcich Novicky Ladislav Ka p rek SvOtlana Kolac ova Czech Hydrometeorological Institute Report of the WCP Water Project C 5 Re analysis of Hydrological Observations in Czechoslovakia May 1993 WMO TD No 554 out of print REPORT OF THE WORKSHOP ON USER NEEDS AND REQUIREMENTS Norrk ping Sweden 4 8 October 1993 WMO TD No 586 out of print DROUGHT AND DESERTIFICATION Reports to the Eleventh session of the Commission for Climatology Havana February 1993 by Kerang Li and A Makarau CCl Rapporteurs on Drought WMO TD No 605 out of print SIXTH PLANNING MEETING ON WORLD CLIMATE PROGRAMME WATER Wallingford 1 5 March 1993 WMO TD No
23. NS AND DATA ACCAD Geneva 19 20 November 1991 also appears as WCDMP 17 WMO TD No 475 out of print URBAN CLIMATOLOGY IN AFRICA Special issue of the journal African Urban Quarterly Yinka R Adebayo guest editor August 1992 WMO TD No 509 out of print OPERATIONAL CLIMATOLOGY CLIMATE APPLICATIONS ON OPERATIONAL CLIMATE SERVICES AND MARKETING INFORMATION AND PUBLICITY Reports to the eleventh session of the Commission for Climatology Havana February 1993 by the CCl rapporteurs on Operational Climatological Services J M Nicholls and Marketing Information and Publicity D W Phillips WMO TD No 525 out of print CLIMATE APPLICATIONS ON USER REQUIREMENTS AND CLICOM APPLICATIONS Reports to the eleventh session of the Commission for Climatology Havana February 1993 by the CCl rapporteurs on User Requirements O Moch and CLICOM Applications P David and S Roy WMO TD No 536 Disponible en francais APPLICATIONS CLIMATOLOGIQUES LES BESOINS DES USAGERS LE CLICOM APPLICATIONS Rapports a la onzi me session de la Commission de climatologie La Havane f vrier 1993 par les rapporteurs de la CCI pour les besoins des usagers O Moch et le CLICOM Applications P David et S Roy WMO TD No 536 REPORT OF THE SECOND SESSION OF THE ADVISORY COMMITTEE ON CLIMATE APPLICATIONS AND DATA ACCAD Geneva 16 17 November 1992 also appears as WCDMP 22 WMO TD No 529 out of print A SURVEY OF CURRENT APPROACHES TO MODELL
24. OD 1988 1995 in English and Russian Prepared by Emil Moralijski CCl Rapporteur on Building Climatology June 1997 WMO TD No 825 SEVENTH PLANNING MEETING ON WORLD CLIMATE PROGRAMME WATER Koblenz Germany 13 16 May 1997 WMO TD No 854 GRID ESTIMATION OF RUNOFF DATA Prepared by Lars Gottschalk and Irina Krasovskaia Report of the WCP Water Project B 3 Development of Grid related Estimates of Hydrological Variables February 1998 WMO TD No 870 REPORT OF THE ELEVENTH SESSION OF THE ADVISORY WORKING GROUP OF THE COMMISSION FOR CLIMATOLOGY Mauritius 9 14 February 1998 also appears as WCDMP 35 WMO TD No 895 PREVISION CLIMATIQUE POUR L HYDROLOGIE EN AFRIQUE CLIMATE FORECASTING FOR HYDROLOGY IN AFRICA ACMAD Niamey Niger 26 April 2 June 1999 WMO TD No 982 REPORT OF THE PLANNING MEETING FOR THE SHANGHAI CLIPS SHOWCASE PROJECT HEAT HEALTH WARNING SYSTEM Shanghai 6 8 October 1999 WMO TDNo 984 BIOMETEOROLOGY AND URBAN CLIMATOLOGY AT THE TURN OF THE MILLENNIUM Selected Papers from the Conference ICB ICUC 99 Sydney 8 12 November 1999 Edited by R J de Dear J D Kalma T R Oke and A Auliciems WMO TD No 1026 REPORT OF THE FIRST STEERING COMMITTEE MEETING ON WORLD CLIMATE PROGRAMME WATER Geneva 23 25 October 2000 WMO TD No 1048 GENERAL SUMMARY OF THE SESSION OF THE INTERCOMMISSION TASK TEAM ON REGIONAL CLIMATE CENTRES Geneva 30 April 3 May 2001 WMO TD No 1070 GENERAL SUMMARY OF THE CLIPS WO
25. OF EXISTING ASSESSMENTS Prepared by Mr J M Nicholls U K November 1996 WMO TD No 780 CLIMATE INFORMATION AND PREDICTION SERVICES FOR FISHERIES Prepared by Jean Luc Le Blanc January 1997 WMO TD No 788 REPORT OF THE MEETING OF THE TASKFORCE ON TRUCE Geneva 14 16 October 1996 WMO TD No 789 WCASP 41 WCASP 42 WCASP 43 WCASP 44 WCASP 45 WCASP 46 WCASP 47 WCASP 48 WCASP 49 WCASP 50 WCASP 51 WCASP 52 WCASP 53 WCASP 54 REGULATORY APPLICATIONS OF THE RELATIONSHIPS BETWEEN NATURAL GAS USAGE AND WEATHER prepared by J A Gray D L Patterson M S Proctor and H E Warren USA and CLIMATE INFORMATION FOR THE APPLICATION OF SOLAR ENERGY INFORMACI N CLIMATOLGICA PARA EL USO DE LA ENERGI A SOLAR prepared by preparado por Sandra Robles Gil Mexico May 1997 WMO TD No 816 REPORTS TO THE TWELFTH SESSION OF THE COMMISSION FOR CLIMATOLOGY GENEVA AUGUST 1997 by the CCl rapporteurs on Financial Insurance and Legal Sectors J Hopkins Agriculture and Food H Bhalme Tourism and Recreation L Lecha Estela and Report of Meeting of Experts on Climate and Human Health Freiburg Germany 28 29 January 1997 June 1997 WMO TD No 822 METEOROLOGICAL ASPECTS AND RECOMMENDATIONS FOR ASSESSING AND USING THE WIND AS AN ENERGY SOURCE IN THE TROPICS Prepared by A Daniels and T Schroeder Hawaii June 1997 WMO TD No 826 BIBLIOGRAPHY OF BUILDING CLIMATOLOGY FOR THE PERI
26. RKING GROUP MEETING Toulouse France 26 29 March 2001 WMO TD No 1087 REPORT OF THE SECOND SESSION OF THE INTER COMMISSION TASK TEAM ON REGIONAL CLIMATE CENTRES Geneva 25 28 March 2002 WMO TD No 1107 ro WCASP 55 WCASP 56 WCASP 57 WCASP 58 WCASP 59 WCASP 60 WCASP 61 WCASP 62 WCASP 63 WCASP 64 WCASP 65 WCASP 66 WCASP 67 5 REPORT OF THE FIRST SESSION OF THE MANAGEMENT GROUP OF THE COMMISSION FOR CLIMATOLOGY Berlin Germany 5 8 March 2002 also appears as WCDMP 48 WMO TD No 1110 REPORT OF THE SECOND STEERING COMMITTEE MEETING ON WORLD CLIMATE PROGRAMME WATER Geneva 23 25 January 2002 WMO TD No 1144 not yet published REPORT OF THE THIRD STEERING COMMITTEE MEETING ON WORLD CLIMATE PROGRAMME WATER Wallingford United Kingdom 21 23 October 2002 WMO TD No 1145 REPORT OF THE CLIMATE INFORMATION AND PREDICTION SERVICES CLIPS TRAINING WORKSHOP FOR EASTERN AND SOUTHERN AFRICA Nairobi Kenya 29 July 9 August 2002 WMO TD No 1152 REPORT OF THE CAPACITY BUILDING TRAINING WORKSHOP ON REDUCING THE IMPACTS OF CLIMATE EXTREMES ON HEALTH Nairobi Kenya 11 15 February 2002 WMO TD No 1162 PROCEEDINGS OF THE RA VI TASK TEAM ON THE PROVISION OF SEASONAL TO INTER ANNUAL FORECASTS AND REGIONAL CLIMATE CENTRE SERVICES RA VI TT SIRCC Reading United Kingdom 14 16 April 2003 WMO TD No 1163 REPORT OF THE CLIMATE INFORMATION AND PREDICTION SERV
27. UTING NORMAL SCORES 31 1 1 Computing a moving average time series and other uses of indices Hydrospect has no built in function to directly com pute a moving average of time series values but it does include indices based on moving averages It might not be immediately obvious that you can use these indices to compute the moving average times series out of your time series Suppose you have a time series of daily temperatures and you would like to compute moving 5 day temperature means When you aggregate this time series please select the Maz period average or Min period average index and enter 5 as the only index parameter to specify 5 day averages Select aggregation periods defined by period length with length equal to 1 This way each day will be a separate aggregation period and the aggregated series length will be the same as that of the original series The first four values in the aggregated series will be missing because the 5 day average was not yet known on those days Subsequent values will represent the averages in appropriate 5 day windows Setting the aggregation period length to 1 may also be meaningful for other indices for instance the Count all index used in this way will result in a binary series of zeros and ones of the same length as the original series with 0s in place of missing values and 1s in place of non missing ones 2 Computing ranks Menu entry Rank observations A rank will be assigned to each
28. affects the display not the computations or underlying data The menu entry View Show missing values has the same function You can import as many series as your computer s memory permits Selecting one of them e g with a mouse click will bring it into focus and its data will appear in the lower right pane 5 Exporting a time series If you wish to study a time series with another program e g to create a graph you can save it in a text file To save a time series in a text file e select the series and choose Time series Export or e select the series and choose File Export the time series or e click the series name with the right mouse button and choose Export Then you can enter the name of the file to hold the data and click the Save button to complete the process There is little merit in exporting a raw file you have just imported but as will be described later on in the manual Hydrospect allows you 5 EXPORTING A TIME SERIES 15 to perform a number of operations on a time series i e derive new series from existing ones and those derived series might be of some use elsewhere too CHAPTER 3 Statistical tests To perform a test on a time series select its name in the left pane of the program window The time series should appear in the lower right pane data view The upper right pane will contain the list of tests performed on this series You can choose an entry from the Test menu to add a test of your
29. ation Chapter 2 Working with data files Data file format Import options Simulating Hydrospect 1 0 data file reading mode Viewing the data Exporting a time series ee PS Chapter 3 Statistical tests 1 Tests for changes 2 Tests for serial dependence 3 How to interpret results 4 Resampling Chapter 4 Working with time series Aggregation and indices of extremes Computing ranks Computing normal scores Peaks over threshold Selecting a subseries Analysis of changes in variance De seasonalisation D gt Pn P 65 bo ja Chapter 5 Saving and sharing your work 1 Hydrospect documents 2 Reporting the results Chapter 6 Reference 1 Keyboard shortcuts Bibliography CHAPTER 1 Overview of Hydrospect Hydrospect is a software package for detecting changes in long time series of hydrological data It makes use of a set of eight different tests for change detection and lets the user create customized derived series to analyse changes in different aspects of time series behaviour The significance of changes may be computed with standard test formulae or using several resampling techniques including block permutation and block bootstrapping Hydrospect is easy to learn and use It is a tool specifically designed for time series analysis It allows you to e read time series data from text files in a variety of formats see Section 1 of Chapter 2 page 9 e aggregate the time series using a variety of indices see Sec
30. cation of the HYDROSPECT data analysis system for the detection of changes in hydrological time series for use in WCP Water and National Hydrological Services Final Report by Dr Maciej Radziejewski amp Professor Zbigniew W Kundzewicz Pozna June 2004 Hydrospect is a software package for detecting changes in long time series of hydrological records The first version of the software was pro duced by Dr Maciej Radziejewski under the supervision of Professor Zbigniew W Kundzewicz in 1999 and reported in Radziejewski amp Kun dzewicz 2000 It was a contribution to the Project A 2 Analysing Long Time Series of Hydrological Data and Indices with Respect to Cli mate Variability and Change of the World Climate Programme Water prepared for the World Meteorological Organization in 1999 Based on remarks of the users of this software package and articulated needs for its extension the present version of the package has been produced by the same authors The most essential new elements of the brand new version of Hydrospect as compared to its first version of 1999 are resampling incl block bootstrapping and block permutation compu ting indices of extremes tests for serial dependence and support for extreme value analysis using POT Peaks Over Threshold Besides the software was subject to a substantial general overhaul The Hydro spect software and the manual can be downloaded from the following ftp site ftp www wmo int docume
31. ct does not use Internet Explorer in any way it makes use of some basic components of Windows that are updated along with Internet Explorer Most probably you will be able to use Hydrospect right away without any preparatory steps but if you experience dif ficulties getting it to start please try to install the newest available version of Internet Explorer on your system Hydrospect does not require any installation The program file can be run from any location You may also place it in the standard system folder e g C Program Files and create a shortcut to it on the desktop or in the Start menu CHAPTER 2 Working with data files Before you can do anything useful with Hydrospect you need to load i e import a data file When you start the program a message displayed in the program window prompts you to do so Creating or directly editing the data set is not supported within Hydrospect However a number of built in functions let you transform your data in a variety of ways If you need to make direct changes for example correct errors you can do it with another tool e g a spreadsheet and then use the altered file rather than the original one with Hydrospect To load a data file e choose Time series Import new data file from the menu or e choose File Import new data file from the menu or e click the button with a green triangle on the toolbar You will be given the opportunity to pick the file of your choice and
32. ding date field must have a value greater than or equal to the From value or less than or equal to the To value to be selected If the date time information is not present you can enter the period length into the field next to Set a range in the period of length the option will then be selected automatically In our example the period length will be 365 25 Section 1 of Chapter 4 page 26 describes how the series is split to periods of given length Now if the series starts on the 1 of November then the first of December will be the 315 day of the annual period in this series You can enter 31 and 92 in the From and To fields at the bottom to define the desired subseries Selecting a subseries in this way is inhernetly inprecise without the dates Hydrospect can only approximate true calendar years and as a result the selected period may by shifted by up to one day e g Hydrospect may select data for the period from the 274 of December to the 1 of February on some years If you are willing to accept such minor imprecision you can define seasonal subseries in a very flexible way even without accompanying dates 6 Analysis of changes in variance Menu entry Study variance As suggested in 14 after 12 some tests for changes in the mean can be applied to detect changes in variance This involves computing the distance of each value in the series from the overall mean and ap plying the test to the series of distances Please
33. es in the data ties influence the values of some stati stical tests some resampling techniques ranks etc 3 When importing data from a text file Hydrospect analyses the file and determines where the data may be located what separators were used etc You only need to choose the data column you wish to use This version of the program is still limited to time series analysis Several variables may be analysed but each one is analysed separately Multivariate analysis may be possible in future versions For now if you require such facilities you may still be able to benefit from using Hydrospect in combination with other programs to achieve your goal Hydrospect may be useful to pre process your data which may then be exported i e saved in a text file and to evaluate partial or final results The source code of Hydrospect Version 2 0 has been written in Visual C The package delivered to users consists of the executa ble programme file hydrospect ere and the present User s Manual in PDF format the two side version is formatted for two sided prin ting Hydrospect is a contribution to the Project Change Detection in Hydrological Data of the World Climate Programme Water pre pared for the World Meteorological Organization This software was developed by Dr Maciej Radziejewski under supervision of Professor Zbigniew W Kundzewicz 1 New features in version 2 0 Major new features in Hydrospect 2 0 are the
34. es without replacement or by sampling its values with repla cement bootstrapping It is a good idea to always verify test results using both permutation and bootstrapping You will notice that the estimates of significance levels are chan ged when you activate resampling but other results do not change If the results obtained from resampling differ much from the ones obta ined without resampling computed using standard formulae the ones given by resampling are probably more valid You may also notice that computations take much longer when resampling is on because tests have to be performed for numerous time series instead of just one Computation time depends on the number of times to resample a series You may select a lower value first e g 100 to see how much time the computations may take and then change it to a higher value A common practice is to use at least 1000 random series so that the result is credible Block resampling methods offer a way to deal with the problem of serial or seasonal dependence in the data Block permutation or bootstrapping works by dividing the data to blocks of fixed size and sampling the entire blocks instead of single values For instance selec ting the block size of 12 or a multiple of 12 should account for annual seasonality effects in monthly data Using longer multi year blocks should be adequate to treat longer term serial dependence If the se ries length is not a multiple of block size t
35. etation because of floating point rounding errors After you have set the options you can press the OK button The time series is loaded and its name appears in the left pane You can click on the name with the right mouse button and choose Options if you wish to change the import options If the import has failed a replacement name is displayed in bold Changing the options may then allow you to read the series successfully or pick a different data file 3 SIMULATING HYDROSPECT 1 0 DATA FILE READING MODE 13 The contents of your data file are not stored in the Hydrospect document and wil have to be read every time the document is opened Only the options you selected are stored In fact if you replace the original file with another file having the same structure and then open the document the original analysis will be repeated for the new file If you click the series name with the right mouse button and choose Delete or if you press the Delete key after selecting the series you are given the option to remove it from the Hydrospect window This only means that Hydrospect will stop using it but it will not be deleted from the disk 3 Simulating Hydrospect 1 0 data file reading mode There is a very minor difference in the way Hydrospect 2 0 and the previous version Hydrospect 1 0 treat data files If you used Hydro spect 1 0 before and open documents created with it Hydrospect 1 0 in Hydrospect 2 0 you may wish to turn on the Sim
36. ew of tests can also be found in 14 1 Tests for changes 1 1 Linear regression The test statistic is the correlation coeffi cient also called Pearson s r of the time variable and the observations Optionally the slope of the fitted line and the significance level are also 16 1 TESTS FOR CHANGES 17 computed The significance level cf 15 3 11 is valid under the null hypothesis of independent and normally distributed observations The test statistic is the correlation coefficient between the time variable and the values of the time series If n is the length of the time series and a a2 an denote the time series values then the test statistic is defined as pa LSM Jf UUt where 1 n m at n t 1 1 n Ty t 1 y Fa ad m n and If there are missing values in the series the above sums are computed only over the values of such that a is not missing and 1 is replaced by 4 where n is the number of non missing values in the series In presence of missing values regression can still be performed and the correlation coefficient of time and observations computed based on the data available However the significance level computed using the standard formula may not be valid in this case so it has to be computed using resampling cf the corresponding section later on in this chapter 1 2 Normal scores linear regression This is a non parametric test based on linear regression To
37. f a given period then the index value for the period is the value number e g day number between 1 and n when the first admissible value occurred or z 1 if no value in the period satisfied the given criterion First below The first occurrence of a value below a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the first admissible value occurred or n 1 if no value in the period satisfied the given criterion First geq The first occurrence of a value greater than or equal to a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the first admissible value occurred or n 1 if no value in the period satisfied the given criterion First leq The first occurrence of a value less than or equal to a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the first admissible value occurred or n 1 if no value in the period satisfied the given criterion Last above The last occurrence of a value above a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the last admissible value occurred or 0 if no value in the period satisfied the given criteri
38. following e tests for serial dependence see Section 2 of Chapter 3 page 21 e resampling see Section 4 of Chapter 3 page 24 e indices of extremes see Section 1 of Chapter 4 page 26 and e POT Peaks Over Threshold see Section 4 of Chapter 4 page 32 Some previously existing features have been improved e missing values are more visible displayed as blank cells in time series preview pane e a toolbar button with the letter M becomes available when missing values are present to quickly inform you about that and let you hide show them e more convenient data file import interface with a summary of not interpreted lines e a possibility to treat some numeric constants as missing values in data files e g 9999 00 e when data files loaded by a document cannot be found Hydro spect searches for them in the document s directory useful 2 SYSTEM REQUIREMENTS AND INSTALLATION 8 when you moved your document and data files to another ma chine e the percentage of missing values allowed in an aggregation period is now user definable e Hydrospect remembers more of the parameters you entered or choices you made and makes them defaults for later A short description of the previous version of Hydrospect can be found in 9 2 System requirements and installation Hydrospect should work well on every PC with Microsoft Windows 98 or later and Microsoft Internet Explorer 5 0 or later installed Al though Hydrospe
39. he report will only include those series for which some tests were actually performed After you click on the Report menu item you can give the name to the text file to be created Click on Save to complete the procedure You should be able to open the report with your text editor 35 CHAPTER 6 Reference 1 Keyboard shortcuts navigation between panes F6 Shifi F6 series tree move the selection Up Down arrow keys PageUp PageDown Home End expand a branch right arrow numeric keypad expand a branch entirely Shift FI0 test list move the selection Up Down Left Right arrow keys PageUp PageDown Home End fest options remove the test Delete remove all tests Delete when the Tests performed heading is selected Shift F10 data view move the selection Up Down arrow keys PageUp PageDown Home End 36 Bibliography 1 T A Buishand Some methods for testing the homogeneity of rainfall records J Hydrol 58 11 27 1982 2 F H S Chiew and T A McMahon Detection of trend or change in annual flow of Australian rivers Intern J of Climatology 13 643 653 1993 3 R M Hirsch D R Helsel T A Cohn and E J Gilroy Statistical treat ment of hydrologic data In D R Maidment editor Handbook of Hydrology chapter 17 Mc Graw Hill New York 1992 4 M G Kendall A new measure of rank correlation Biometrika 30 81 93 1938 5 M G Kendall Rank Correlation Meth
40. he two methods behave slightly differently Block permutation divides all the data to blocks allowing the last block to be shorter than the rest Block bootstrapping samples only full blocks of specified size disregarding a non fitting part of the data at the end and constructs a series of appropriate length from these full blocks the excessive portion of generated data is cut off if necessary You can also specify a fractional block size e g if you need to han dle daily values with annual seasonal dependence you can use blocks of 365 25 days Hydrospect will then use blocks of 365 or 366 days so that mean block size is 365 25 Resampling options are set individually for each time series in the document The menu command Test Resampling shows the options for the current series The corresponding toolbar button with a Monte Carlo icon acts as a switch turning resampling on and off for the selected series Switching off is immediate Switching on brings up a dialogue box with the last selected resampling options for this series or your last entered options if this series was not resampled yet The selected resampling method and parameters are displayed at the top of the list of tests and written in the report cf Section 2 of Chapter 5 page 35 CHAPTER 4 Working with time series Standard procedures like ranking observations in a sample or remo ving annual regime from a series of flows can be seen as producing a new derived t
41. ils If the data exhibit serial or seasonal dependence one option is to aggregate Annual indices are often independent but experience shows that it is not always the case Block resampling methods offer a way to deal with that problem as explained in Section 4 The three tests for independence included in this version of Hydro spect are 1 Median crossing test Fisz test 2 Turning points test Kendall s test 3 Rank difference test Meacham test They are all non parametric A high significance level means depen dence Negative sign of the test statistic indicates positive correlation 2 1 Median crossing test Fisz test This is a non parametric test based on splitting the values in the series to two equal more or less categories above and below the median Let n be the length of the series and N be the number of times when a value below the median is followed by one above the median or conversely Under the null hypothesis of independent equally distributed values N is normally distributed with mean gt and variance gt see 13 and the references cited there When there are values euqal to the median most notably in case of numerous ties it may be impossible to divide the data into two equal bins and the above formulae will not be valid anymore In such a case Hydrospect splits the values as evenly as possible and computes the test statistic in the following way Let p and pz denote the fraction of values i
42. ime series from an existing one Hydrospect implements a number of such procedures They are available from the Time se ries menu and from the context menu that appears when you click the series name in the left pane with the right mouse button To construct a derived series e select the series and select the operation of your choice from the Time series menu or e click the series name with the right mouse button and select the operation of your choice from the context menu In some cases you will be given some additional options to choose from To change those options later click on the derived series name with the right mouse button and choose Options from the context menu or select the name of the derived time series and press Enter You can produce a number of derived series from any series For example you can compute annual means and monthly means for a series of daily data as well as select a subseries consisting only of observations recorded in June Then you can rank the annual means etc The relationships between derived time series are presented in the left pane in the form of a tree 1 Aggregation and indices of extremes Menu entry Aggregate The series will be divided to subperiods and the values in each subperiod will be replaced by one value for example the mean or the maximum of the values in the subperiod The following indices are implemented Mean The mean Sum The sum Minimum The minimum Maximum The
43. isfies the formula KS max n1 lt k lt n k 2 sen a M where M denotes the median of time series values and the sign of KS matches the sign of the corresponding inner sum for which the maximum absolute value was reached This test is based on the assumption that the number of values in the series below the median and the number of values above the median are the same In case of ties specifically if values equal to the median are present in the series this might not be true In this case distinct weights are used for the values above and below the median to make sure that the total cumulative sum is unbiased and the assessment of significance by means of the Kolmogorov Smirnov test is valid The weights for each group of values are equal to the reciprocal of their quantity analogously to 2 If there are missing values in the series the Distribution free CU SUM test is applied to the non missing values as if they were con secutive in the series Significance level computed is valid under the null hypothesis of independent identically distributed values As an option the point k at which the cumulative sum was greatest is also displayed 1 6 Cumulative deviations This parametric test is described in 1 2 and 14 The test statistic is the value denoted by Q in those papers If n is the length of the time series and a a2 a denote the time series values then the test statistic equals 1 k max
44. lue in row 538 column 8 from each day s data Hydrospect will analyse the file you need an apropriate amount of memory slightly more than the size of the file and ignore the headers at the beginning and in the middle of the file The non fitting lines with 6 values at the end of each day s data will have to be ignored as well and there is no way to read those with Hydrospect without reformatting the file You will need to select column 8 for your time series Hydrospect will load 2 IMPORT OPTIONS 12 all the values from column 8 Then to select only the values in row 538 out of 1176 rows for each day you can create a subseries of your imported series cf Section 5 of Chapter 4 page 32 2 Import options After you choose the Import new data file command from the Time series menu Hydrospect lets you select and open the data file analyzes it to find where the data is located in the file and how it is formatted and then displays a dialogue box with some options You can type the name of your dataset in the appropriate field or accept the default one that Hydrospect suggests either the first line of the file unless it is recognized as data or the file name If Hydrospect recognizes column headings cf the previous section you may choose to use them too In that case the column heading corresponding to your selected time series will be appended to the dataset name to form the name of the time series displayed in the program window
45. maximum Median The median 26 1 AGGREGATION AND INDICES OF EXTREMES 27 Tukey s trimean The weighted mean of the median with weight 0 5 and the 25 and 75 percentiles each with we ight 0 25 Percentile E g the 25 quartile is computed by specifying 25 as the parameter Percentile above E g the median 50 percentile of values greater than 0 Percentile below E g the median 50 percentile of values less than 0 Percentile geq E g the median 50 percentile of values gre ater than or equal to 1 Percentile leq E g the median 50 percentile of values less than or equal to 1 Standard deviation Standard deviation within a period Count all The number of all non missing values in a period Count above The number of values above a threshold Count below The number of values below a threshold Count geq The number of values greater than or equal to a threshold Count leq The number of values less than or equal to a thre shold Proportion above E g the ratio of precipitation received du ring heavy rain days to the annual total Proportion below E g the ratio of precipitation received du ring low precipitation days to the annual total Proportion geq E g the ratio of precipitation received during heavy rain days to the annual total Proportion leq E g the ratio of precipitation received during low precipitation days to the annual total Sum exceedances The sum of exceedances above a
46. n the lower and upper category respectively p p2 1 Then the expected value of N under the null hypothesis is m 2pypa n 1 and the variance equals v 4p p2 1 3p1p2 n pipe 6 20p1P2 Hydrospect makes use of these formulae to compute the significance level for the Fisz test The test statistic displayed in Hydrospect equals N m S A v 3 HOW TO INTERPRET RESULTS 23 In Hydrospect this test can only be applied to a series of ranks or ranked deviations from the median so the raw data need to be pre processed and ranks calculated Please refer to Section 2 of Chapter 4 page 31 for details on computing ranks with Hydrospect 2 2 Turning points test Kendall s test This non parametric test is based on counting turning points in the series i e thri ples of subsequent values Ti 1 Zi Zi 1 such that z lt Z gt Tin OT Zj 1 gt Ti lt X41 If N is the number of turning points then the test statistic displayed in Hydrospect is g SN 2n 4 V10 16n 29 Under the null hypothesis of independent equally distributed values this statistic is normally distributed with mean 0 and variance 1 see 13 In Hydrospect this test can only be applied to a series of ranks or ranked deviations from the median so the raw data need to be pre processed and ranks calculated Please refer to Section 2 of Chapter 4 page 31 for details on computing ranks with Hydrospect 2 3 Rank difference
47. nd m the mean rank for the i th subperiod then the test statistic equals 12 z CED rrwma gt 2 tapy PA mj 2 TESTS FOR SERIAL DEPENDENCE 21 where T is the tie correction 1 3 t denoting the number of ties in subsequent tie groups If missing values are present in the series subperiod means are computed based on non missing values The partitioning of the series to subperiods is based on the pe riod starting points entered in the dialogue box separated by spaces For example if you would like to test for equality of the means of subsequent 30 year periods in a 90 years long time series of annual mean temperatures you need to compute the series of ranks add the Kruskall Wallis test from the Test menu and specify 1 31 61 to define three subperiods years Nos 1 30 the first 30 years years Nos 31 60 and years Nos 61 90 in the time series Equivalently you can specify 31 61 because the first subperiod always starts from the first value so the leading 1 is redundant If you would like the period to start at a given date and do not know the corresponding number to enter in the dialog box you can highlight the desired time series value in the data view The value number position in the time series will be displayed on the status bar next to Selected position This is the number you should enter in the dialogue box to make this observation start a new subperiod The Auto fill button simplifies defining s
48. new series will extend to the beginning and or the end of the original one The observation s position corresponds to the number displayed on the status bar next to Selected position after you highlight the observation in the data view The Seasonal range part of the dialogue box allows you to select only those observations which were measured for instance in a specific time of the year Suppose you have a time series of daily data from say several decades and you wish to look only at the data from December and January If the date time information is present in the series and there is a date time field with the month number you can select this date time field using the small arrows spin buttons next to the time field number below the Set a range for the following date time field option The option will then be automatically selected The space to the right contains the date time entry associated with the first value in the time 7 DE SEASONALISATION 33 series Use the spin buttons to change the time field number until the appropriate field is highlighted on the right Now enter 12 and 1 in the From and To fields at the bottom to select the seasonal range from December to January inclusive When the value in the From field is less than or equal to the value in the 70 field Hydrospect selects the values whose corresponding date field value falls in the appropriate range When the From value is greater than the To value the correspon
49. note that this approach is based on the assumption that the overall mean or another measure of central tendency exists and is constant throughout the series The Hydrospect function described here allows you to compute such a series of distances so the tests for changes in the mean can be applied to it In general case you are given the option to compute distances from the mean the median or Tukey s trimean In the special case when this procedure is applied to a series of ranks distances from the median multiplied by two are computed and no options are displayed Multi plication by the factor of two ensures that the derived series can for most purposes be treated as a series of ranks It has no effect on the significance of test s results 7 De seasonalisation Menu entry De seasonalise 7 DE SEASONALISATION 34 The seasonal means regime are subtracted from each value and the remainder is divided by the seasonal standard deviation Any of the two steps can be disabled at your option The means and deviations are smoothed using harmonic functions By default harmonics of degree up to five are used both for the means and the deviations but this can be changed by entering the desired values in the appropriate fields in the dialogue box The first entry in the dialogue box defines the period length for which seasonality should be altered For example to subtract the regime from the series of monthly flows use 12 as the period length
50. nt that comment lines can be clearly distinguished from data lines either by that they contain some text or at least have a differrent number or format of entries Some comment lines may have special meaning the first line may contain a descriptive name of the dataset in the file the line preceding the first data line may constist of column headings as in a table i e one word names for each column Dataset name and or column headings can form a default name for your time series displayed in the program window Dataset name can be quite long to identify the data unambiguously It is practical to put the most essential part of the name at the beginning for example Warta monthly mean flows 1822 1990 rather than Monthly mean flows of the river Warta 1822 1990 Example 1 A simple time series of daily river flows accompanied by dates including calendar years and hydrological years can be easily imported Alph river daily flows in cubic meters per second 1980 10 29 1980 210 4 1 DATA FILE FORMAT 11 1980 10 30 1980 181 0 1980 10 31 1980 123 9 1980 11 01 1981 299 3 1980 11 02 1981 312 0 Hydrospect will analyse the file and suggest correct defaults for the interpretation of date time and flow data columns Example 2 A file containing several variables should preferably contain column headings describing the variables Ground temperature in the town of Xanadu Year Month Day Tmax Tmin 1999 11 O03 12 0
51. nts hwr hydrospect zip Comments with regard to the Hydrospect software should be direc ted to Dr Maciej Radziejewski Research Centre for Agricultural and Forest Environment Polish Academy of Sciences ul Bukowska 19 60 809 Pozna Poland E mail maciejrQamu edu pl The new version of the Hydrospect software package has undergone testing by several users who made constructive comments which helped the author to upgrade the product Detection of changes in long time series of hydrological records is indeed an essential activity both within the so called scientific hydro logy and operational hydrology Abrupt or gradual changes in river discharge are fundamental for planning of future water resources and 2 flood protection Traditionally design rules are based on the assump tion of stationary hydrology resulting in the principle that the Past is the key to the Future which has a limited validity in the era of global change Kundzewicz 2004 If the stationarity assumption is not correct then the existing procedures for designing water related infrastructures dams dikes etc will have to be revised Otherwise systems would be over or under designed and might either not serve their purpose adequately or be overly costly The hypothesis that climate change leads to the acceleration of the hydrological cycle and may cause increases in the frequency and severity of extreme hydrological events has resulted in growing recent inte
52. ods Charles Griffin London 1975 6 W H Kruskal and W A Wallis Use of ranks in one criterion variance analysis J Am Stat Assoc 47 583 621 and 907 911 1952 7 Z W Kundzewicz and A Robson editors Detecting Trend and Other Changes in Hydrological Data World Climate Programme Water World Climate Data and Monitoring Programme WCDMP 45 WMO TD No 1013 Geneva 2000 8 W H Press S A Teukolsky W T Vetterling and B P Flannery Numerical Recipes in C The Art of Scientific Computing Cambridge University Press Cambridge New York Port Chester Melbourne Sydney second edition 1988 1992 9 M Radziejewski and Z W Kundzewicz Hydrospect software for detecting changes in hydrological data In Z W Kundzewicz and A Robson editors Detecting Trend and Other Changes in Hydrological Data pages 151 152 ap pendix 2 World Climate Programme Water World Climate Data and Mo nitoring Programme WCDMP 45 WMO TD No 1013 Geneva 2000 10 A J Robson and D W Reed Non stationarity in UK flood records Flood Es timation Handbook Note No 24 Report to MAFF Project FD 0409 Institute of Hydrology Wallingford Oxfordshire U K 1996 11 J D Salas Analysis and modelling of hydrologic time series In D R Ma idment editor Handbook of Hydrology chapter 19 Mc Graw Hill New York 1992 12 R Sneyers Sur l analyse statistique des s ries d observations Technical Note No 143 WMO 1
53. of those arrows contains the date time information associated with the first value in the time series Use the spin buttons to change the time field number until the appropriate field is highligh ted on the right In principle the index value for a given period should depend only on the time series values within this period However some indices are based on windows that can come across period boundaries The treatment of results obtained for such windows may raise questions and should be explained To simplify the discussion let us assume that you are computing annual indices based on a daily time series The Min Max moving average sum indices compute moving avera ges sums for moving n day windows in the time series n is specified by the user as a parameter The minimum maximum for a given year is computed over all n day windows whose last day is in the given year For example the 5 day average for the days Dec 28th 1988 Jan 1 1989 will be attributed to the year 1989 The temperature average for these 5 days was not known before Jan 1 1989 so it should not influence the index value for the year 1988 The situation is slightly more complicated with the Maz consec family of indices The carry over versions take into account all periods of consecutive days ending in a given year even if they start before that year and regardless of whether or not they continue further over to subsequent years For example suppose you
54. on t 1 AGGREGATION AND INDICES OF EXTREMES 29 Last below The last occurrence of a value below a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the last admissible value occurred or 0 if no value in the period satisfied the given criterion Last geq The last occurrence of a value greater than or equal to a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the last admissible value occurred or 0 if no value in the period satisfied the given criterion Last leq The last occurrence of a value less than or equal to a threshold If n denotes the length of a given period then the index value for the period is the value number e g day number between 1 and n when the last admissible value occurred or 0 if no value in the period satisfied the given criterion You can select from the list of indices and supply index parameters in the box below it A short description of the selected index is displayed for your convenience If there is more than a given percentage 1 by default of missing values in the period the aggregated value for this period is also treated as missing Otherwise the aggregated value is based on the existing values in that period Thus if missing values are present but sparse in the series the aggregated se
55. pe cannot be directly interpreted as it refers to transformed data The slope obtained from linear regression applied to untransformed data can be quite useful Resampling offers a way to compute significance without assuming a specific distribution of data It is straightforward and safer anyway Moreover the formulae available for the statistical tests are only approximations valid for long time series while boot strapping can be applied also when the length of the series is moderate Block resampling methods are a good solution if the data exhibits serial or seasonal dependence It is recommended to always use resampling to compute the final results or to double check their validity 4 Resampling Resampling offers an alternative way to compute the significance levels of test results It may allow you to use a test even if its as sumptions are not satisfied by the series Please refer to the previous sections for information on test assumptions Resampling is activated through the Test Resampling menu entry You can select the method and supply necessary parameters Resampling works by generating many random time series with distribution identical to that of your time series The test results obta ined for the original time series are compared to those obtained for the random series and significance is evaluated based on this comparison 4 RESAMPLING 25 Random series are generated either by permuting the series sampling its valu
56. r E is ok too Hydrospect always uses a dot as the decimal point regardless of the regional settings of your operating system in order to assure compatibility of data files between different systems Values and dates in each line must be separated by blank characters spaces or TABs commas or semicolons The column width does not have to be fixed in Hydrospect Fixed width data files can be read properly because multiple blank characters are treated as one A file may contain an arbitrary number of date and value columns but you will only be able to import one value column and multiple date columns at a time Missing values should preferably be represented as blank spaces delimited by commas or semicolons since multiple spaces or TABs are treated as one there is no way to have a blank space delimited by other blank spaces Optionally a special numeric constant e g 9999 00 or a similar constant of your choice may also be interpreted as a missing value That is the only way to specify missing values in a fixed width data file without delimiters Please note that missing values in date time columns are not allowed Data files often contain comments or headers that can be quite long and sometimes appear even in the middle of the file In such a case Hydrospect tries to find a common pattern among a majortiy of the lines in the file The lines following the pattern are treated as data lines and the other ones as comments It is importa
57. rest in change detection in flow data Kundzewicz 2004 Yet there are problems with the availability of appropriate data to use with the cho ice of methods to apply and finally with the interpretation of results The search for possibly weak changes in time series of hydrological data which are subject to certainly strong natural variability is a difficult task and the use of adequate baseline data being in short supply and of appropriate methodology is essential The idea that all countries convey their hydrological data to global data repositories is excellent but its implementation encounters con siderable practical problems Some countries for several reasons are reluctant to convey their data abroad thus hampering international studies and jeopardizing ability to evaluate the components of the wa ter balance of the world Yet if a uniform approach to change detection is taken then countries may study their data at home and exchange the results It is therefore essential that the methods of analysis used in different countries are comparable When undertaking a study aimed at change detection in a hydrolo gical variable one goes through the following steps the main stages in statistical testing Kundzewicz amp Robson 2004 e Decide what type of series to test depending on the issues of in terest e g monthly averages annual maxima deseasonalized data etc e Decide what types of change are of interest grad
58. ries If the significance level calculated by Hydrospect is 99 there is still a one percent probability of a result of the same magnitude occurring by pure chance It is essential to emphasize that all the tests included in Hydro spect are based on strong assumptions on the time series temporal independence for all the tests used and normal distribution for some tests the parametric ones Validity of these assumptions in particular cases guides our credibility in test results in particular in the regions of the test statistics where the hypothesis should be accepted rejec ted not communicated to the Hydrospect user and the confidence levels returned to the user If assumptions are not satisfied the tests can be only interpreted as exploratory data analysis tools rather than rigorous statistical methods A common mistake is to apply e g linear regression to a time se ries without checking if the series values are normally distributed and independent If the data is not normally distributed you may apply a parametric test to so called normal scores cf Section 3 of Chapter 4 page 31 and thus make a non parametric test out of a parametric one Normal scores linear regression is an example of a non parametric test Non parametric tests can be regarded as safer to use as they do not require the normal distribution assumption The drawback of this ap proach is that the result of such a test e g the linear regression slo
59. ries may be free of missing values You can define subperiods of fixed length by entering the length in the appropriate box the option Defined by period length will then be automatically selected The first subperiod will start at the begin ning of the series The last subperiod may be shorter than the given length The length you enter may be fractional in which case the leng ths of subsequent periods will occasionally differ by 1 so that the mean length will approach the value you have specified For example if you aggregate a time series of daily data and no date time information is available you can use the period of length 365 25 to define approxima tely annual periods Those periods may or may not be close to actual calendar or hydrological years depending on where your time series starts To have the aggregation start precisely at a certain point you can use subseries selection To perform monthly aggregation of daily data you can use periods of length 30 4375 If date time information is present you can define periods based on one of date time fields For example to perform monthly aggregation select the field denoting the month number in the date time informa tion To select a date time field use the small arrows spin buttons next to the time field number below the Defined by a date time field number option The option will then be automatically selected The 1 AGGREGATION AND INDICES OF EXTREMES 30 space to the right
60. statistic displayed is Kendall s sum commonly denoted as S divided by the square root of its variance under the hypothesis of independent and identically distributed observations If n is the length of the time series and a Qa a denote the time series values then the test statistic equals 1 vor where S gt sgn a a 1 lt i lt j lt n and _ n n 1 2n 5 unless there are ties equal values present in the data in which case the formula for variance given in 5 is used The significance level is valid under the null hypothesis of independent and identically distributed observations As an option Kendall s original tau the sum S divided by the maximum possible value of obtainable by a rearrangement of the time series values is also displayed If there are missing values in the time series the Mann Kendall s test is applied to the non missing values as if they were consecutive in the series Hydrospect employs a dedicated algorithm for computing this test efficiently in the order of n log n operations where n is the series length This test is particularly useful for detection of a gradual change or trend in time series 1 TESTS FOR CHANGES 19 1 5 Distribution free CUSUM This non parametric test is im plemented in general case as in 2 The test statistic is KS defined in 2 If n is the length of the time series and a aa a denote the time series values then the test statistic sat
61. threshold If t is the threshold entered as a parameter and a denotes subsequent time series values the index value equals a gt t where the sum is over a given period Sum deficits The sum of deficits below a threshold If t is the threshold entered as a parameter and a denotes subsequent time series values the index value equals Y G wi ai where the sum is over a given period Longest consec above The longest period of consecutive va lues above a threshold 1 AGGREGATION AND INDICES OF EXTREMES 28 Longest consec below The longest period of consecutive va lues below a threshold Longest consec geq The longest period of consecutive values greater than or equal to a threshold Longest consec leq The longest period of consecutive values less than or equal to a threshold Longest consec carry over The carry over versions of the Longest consec indices see the comments after this list Max moving average E g the highest n day temperature ave rage see also the comments after this list Min moving average E g the lowest n day temperature ave rage see also the comments after this list Max moving sum E g the highest n day precipitation sum see also the comments after this list Min moving sum E g the highest n day precipitation sum see also the comments after this list First above The first occurrence of a value above a threshold If n denotes the length o
62. ual trend or step change e Check out data assumptions e g use exploratory data analy sis or a formal test e Select a statistical test more than one is good practice This means selecting a test statistic and selecting a method for eva luating significance levels e Evaluate significance levels e Investigate and interpret results As stated above selecting more than one test is a good practice Hence results the usefulness of Hydrospect containing several tests which Guidelines for test selection Data are normally di This is an unlikely scenario for hydrological data stributed independent If applicable all tests used in the area should be and non seasonal suitable Any of the distribution free tests are suitable Te sts that are based on normality assumption can also be applied either by a first applying a nor mal scores or ranks transformation or b using a relevant test statistic and evaluating significance using resampling techniques For almost all the tests it will be necessary to extract the test statistic and then to evaluate si gnificance levels using block permutation or block bootstrap methods Without this test assump tions will not be met TABLE 1 Guidelines for test selection general appli cability of tests after Kundzewicz amp Robson 2004 Data are independent and non seasonal but are non normal Data are non normal and are not indepen
63. ubperiods by filling the Sub period starting points field e press Auto fill when the field is empty to fill it with values dividing the time series in three parts e enter one number period length in the field and press Auto fill to divide the series to periods of the given length e enter more numbers separated by spaces and press Auto fill to extend the sequence of numbers entered in the arithmetic progression based on the last two numbers There is also an option to compute the significance level valid under the null hypothesis of independent identically distributed values 2 Tests for serial dependence Before you use any of the change detection tests described in the previous section you have to make sure you are using an appropriate method and that the assumptions made by the methods are satisfied Typically change detection tests make two kinds of assumptions e independence of observations e g daily mean flows of a large river are very unlikely to be independent while annual mean flows often are e normally distributed observations 2 TESTS FOR SERIAL DEPENDENCE 22 From among the tests implemented in Hydrospect all assume indepen dence while the parametric ones assume normality The description of each test specifies its assumptions Hydrospect does not include tests for checking normallity Instead you are advised to always check your results using permutation and bootstrapping see the following sections for deta
64. ulate Hydrospect 1 0 menu option available through the File Data file reading mode menu entry To restore the standard behaviour please select the Standard option from the same menu The only difference between the two modes is how the lines ending with a comma or semicolon are treated Normally a line like 10 should be treated as consisting of two fields the second one being a missing value so data structure like the following 9 3 0 10 11 2 1 would be ok Hydrospect 1 0 would only recognize such a line properly if there were any spaces after the comma a comma or semicolon at the very end of the line wou d be ignored This unintentional beha viour was changed in Hydrospect 2 0 and now a comma or semicolon is always taken into account However if you relied on this undocumen ted behaviour in Hydrospect 1 0 you may have created used data files formatted like this 9 3 0 10 5 5 11 2 1 i e with some lines ending with a separator and other not so this is not a good practice In the Standard mode Hydrospect 2 0 would disregard some of those lines as not consistent with the common pat tern in the file Information about such disregarded ignored lines is displayed when you import a new data file see the previous section However when you open an existing document you may receive no warning of some lines being disregarded unless you choose to review 5 EXPORTING A TIME SERIES 14 and or reselect the import

world climate programme - water development, use and

Contents

Download Pdf Manuals

Related Search

Related Contents