Home

User Guide for the TIMSS International Database

1. sss 3 3 Relationship Between the Desired Populations and Exclusions sss 3 4 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Population 1 Written Assessment 3 9 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Population 2 Written Assessment essere trenes 3 10 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Performance AS e ME ete eee eer rre e eret terrere pter deed 3 11 Sample Information for TIMSS Population 1 Countries 3 12 Sample Information for TIMSS Population 2 Countries ssssseeee 3 13 Example Coding Guide for Short Answer Mathematics ltem sse 4 3 Example Coding Guide for Extended Response Mathematics ltem sssssss 4 4 TIMSS Within Country Free Response Coding Reliability Data sss 4 6 List of Deleted Cognitive Items eee IRR Der pete 4 8 Descriptive Statistics for the International Mathematics Achievement Scores for Population 1 Variable AIMATSCR uainne 6 5 Descriptive Statistics for the International Science Achievement Scores for Population T Variable AISCISCR terere re rte tree ten teen et tnt te ns 6 6 Descriptive
2. 0 to 5 years 2 6 to 10 years 3 11 to 20 years 4 Over 20 years value country 036 Australia 040 Austria 056 Belgium F1 057 Belgium Fr 100 Bulgaria 124 Canada 170 Colombia 196 Cyprus 200 Czech Republic 208 Denmark 826 England 250 France 280 Germany 300 Greece 344 Hong Kong 348 Hungary 352 Iceland 364 Iran Islamic Rep 372 Ireland 376 Israel 380 Italy 392 Japan 410 Korea 414 Kuwait 428 Latvia LSS 440 Lithuania 528 Netherlands 554 New Zealand 578 Norway 608 Philippines 620 Portugal 642 Romania 643 Russian Federation 827 Scotland 702 Singapore 201 Slovak Republic 890 Slovenia 717 South Africa 724 Spain 752 Sweden 756 Switzerland 764 Thailand 840 United States Now use the macro JACK to get the results include jack sas jack matwgt jkzone jkindic 75 idcntry idgrader btbgtaug bimatscr merged proc print data final noobs by idcntry idgrader where idgrader 2 var btbgtaug N matwgt mnx mnx_se pct pct_se format idcntry country btbgtaug exper idgrader grade 9 2 8 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES Figure 9 14 CHAPTER 9 SPSS Control Statements for Performing Analyses with Teacher Level Variables EXAMPLE2 SPS get file btmalll sys keep identry idteach idlink btbgtaug
3. SON SOUS SS Ss Ss za2v SS S Seno SS S Sor lt SNNNSNNNNSNN NSSSNSNNNNAS NSSSNSNNSNNAAS SNSSS NN NSNSNSSSNNNNNAAS SS SS NN NN NNNM NS SS NN NN NN N NNNM NS SN NN NN NN NN NN NNN NS NN NLL LLL LLL NNN NLS SSS SSS SSS SSS S lt lt LQNN NNN VNNNS NNN IN S SS NN NIS NN NIS NN NINN NN NIN NNN NIN NNN NINN NN NINN NON NPR GY NNN NNNSN v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v v SQN NA SQN NN SQN NN SQN NAN All TIMSS international data files are provided in ASCII format enabling users to access the data directly without having to go through intermediate steps All details of the file structure are provided in codebook files related to each of the data files listed in Table 7 1 The use of these codebooks is described in Section 7 3 TIMSS DATABASE USER GUIDE 7 328 CHAPTER 7 DATABASE FILES 7 2 1 Background Files There are three different types of TIMSS background files student teacher and school 7 2 1 1 Student Background File Students who participated in TIMSS were administered a background questionnaire with questions related to home background and school experiences The Student Background file contains students responses to these questions One version of the questionnaire was administered in Population 1 and two versions were administered in Population 2 In Population 2 one version was tailore
4. 8 Secondary 2 Secondary 2 8 8 8 2nd Year 8 2nd Grade Lower Secondary 2nd Grade Middle School 9 8 8 Secondary 2 Form 3 7 1st Year High School Grade 8 8 8 Secondary 2 Secondary 2 8 8 8 EGB Standard 6 7 7 8 Secondary 2 8 education In 4 of the 8 states territories Years of Formal Schooling Including Upper Grade 80r9 om o 0 0 00000000 OO ON 00000 CO OO Lower Grade Years of Formal Country s Name for a Country Schooling Including Lower Grade 1 Lower Grade Australia 70r8 70r8 Austria 3 Klasse 7 Belgium FI 1A 7 Belgium Fr 1A 7 Bulgaria 7 Yi Canada 7 7 Colombia 7 7 Cyprus 7 7 Czech Republic 4 7 Denmark 6 6 England Year 8 8 France 5 me 7 Germany 7 7 Greece Secondary 1 7 Hong Kong Secondary 1 7 Hungary 7 7 Iceland 7 7 Iran Islamic Rep 7 7 Ireland 1st Year 7 Israel Japan 1st Grade Lower Secondary 7 Korea Republic of 1st Grade Middle School 7 Kuwait Latvia 7 7 Lithuania 7 7 Netherlands Secondary 1 7 New Zealand Form 2 7 5 8 5 Norway 6 6 Philippines Grade 6 Elementary 6 Portugal Grade 7 7 Romania 7 7 Russian Federation 7 60r7 Scotland Secondary 1 8 Singapore Secondary 1 7 Slovak Republic 7 7 Slovenia 7 7 Spain 7 EGB 7 South Africa Standard 5 7 Sweden 6 6 Switzerland German 6 6 French and Italian 7 7 Thailand Secondary 1 7 United States 1 7 1 Years of schooling based on the number of years children in the grade level
5. Adams R J Wilson M R and Wang W C 1997 The multidimensional random coefficients multinomial logit Applied Psychological Measurement Adams R J Wilson M R and Wu M L 1997 Multilevel item responses models An approach to errors in variables regression Journal of Educational and Behavioral Statistics 22 1 46 75 Beaton A E 1987 Implementing the new design The NAEP 1983 84 technical report Report No 15 TR 20 Princeton NJ Educational Testing Service Beaton A E and Gonzalez E J 1997 Test curriculum matching analysis In M O Martin and D L Kelly Eds TIMSS technical report volume II Implementation and analysis Chestnut Hill MA Boston College Beaton A E Martin M O Mullis I V S Gonzalez E J Smith T A and Kelly D L 1996a Science achievement in the middle school years IEA s Third International Mathematics and Science Study Chestnut Hill MA Boston College Beaton A E Mullis I V S Martin M O Gonzalez E J Kelly D L and Smith T A 1996b Mathematics achievement in the middle school years IEA s Third International Mathematics and Science Study Chestnut Hill MA Boston College Bock R D and Aitken M 1981 Marginal maximum likelihood estimation of item parameters An application of the EM algorithm Psychometrika 46 443 459 Dempster A P Laird N M and Rubin D B 1977 Maximum likelihood from incomplete data via the EM algorithm Journal of
6. Although the achievement scores are computed at the individual level several summary variables of achievement are included in the School Background data files These correspond to the average achievement scores within the school in mathematics and science by grade level The variables are identified below ACLGMAT Mean Mathematics Achievement at the Lower Grade Population 1 ACLGSCI Mean Science Achievement at the Lower Grade Population 1 ACUGMAT Mean Mathematics Achievement at the Upper Grade Population 1 ACUGSCI Mean Science Achievement at the Upper Grade Population 1 BCLGMAT Mean Mathematics Achievement at the Lower Grade Population 2 BCLGSCI Mean Science Achievement at the Lower Grade Population 2 BCUGMAT Mean Mathematics Achievement at the Upper Grade Population 2 BCUGSCI Mean Science Achievement at the Upper Grade Population 2 BCEGMAT Mean Mathematics Achievement at the Extra Grade Population 2 BCEGSCI Mean Science Achievement at the Extra Grade Population 2 Note that although most countries tested across two grades per the target population definition Switzerland and Sweden opted to test students in an extra grade at Population 2 and therefore the variables BCEGMAT and BCEGSCI are set to missing in all but these two countries TIMSS DATABASE USER GUIDE 6 9 CHAPTER 6 ACHIEVEMENT SCORES 6 10 TIMSS DATABASE USER GUIDE Chapter 7 Content and Format of Database Files 7 1 I
7. OR UPCASE Type EX Then Do I In Item Recode I 10 thru 19 1 20 thru 29 2 30 thru 39 3 70 thru 79 0 nr 0 na sysmis 10m 0 other 0 Else 0 DoEnd IfEnd lenddefine SCOREIT Type MC Item BSMMA01 BSMSA10 BSMSBO05 BSMSBO06 BSMMBO8 BSMMBO09 BSMMB12 BSMMCO1 BSMMC03 BSMMCO6 BSMSC11 BSMSD06 BSMMD09 BSMMD10 BSMSE10 BSMSF02 BSMSF03 BSMSF05 BSMMF11 BSMSG07 BSMSH01 BSMSH03 BSMSH04 BSMSH06 BSMMI07 BSMMI09 BSMS113 BSMSI15 BSMSI16 BSMSI19 BSMSJ02 BSMMJ15 BSMMJ16 BSMMK03 BSMSK11 BSMSK16 BSMSK18 BSMML15 BSMMM02 BSMMM03 BSMMM04 BSMSM13 BSMSN06 BSMMN16 BSMMN17 BSMSO12 BSMMQ05 BSMSQ11 BSMSQ15 BSMSRO1 BSMMRO6 BSMMR11 RIGHT 1 nr 6 na 8 om 9 other 7 SCOREIT Type SA Item BSSMI04 BSSMI06 BSSSI18 BSSSJ03 BSSSJ09 BSSMJ12 BSSMJ13 BSSMK02 BSSMK05 BSSSK10 BSSSK19 BSSML16 BSSMM06 BSSMM08 BSSSM12 BSSSM14 BSSSNO7 BSSSN10 BSSMN13 BSSMN19 BSSMO06 BSSMO09 BSSSO10 BSSSO16 BSSSO17 BSSSP02 BSSSP03 BSSSP05 BSSSP06 BSSMP16 BSSMO10 BSSSO12 BSSSQ17 BSSSQ18 BSSSRO4 BSSSRO5 BSSMR13 BSSMR14 BSSMVO1 BSSMVO04 RIGHT 0 nr 96 na 98 om 99 other 90 TIMSS DATABASE USER GUIDE 9 43 CHAPTER 9 PERFORMING ANALYSES 9 44 TIMSS DATABASE USER GUIDE References Cited in Text Adams R J and Gonzalez E J 1996 TIMSS test design In M O Martin and D L Kelly Eds TIMSS technical report volume I Design and development Chestnut Hill MA Boston College
8. Pacific Educational Press Robitaille D F McKnight C C Schmidt W H Britton E D Raizen S A and Nicol C 1993 TIMSS monograph no 1 Curriculum frameworks for mathematics and science Vancouver B C Pacific Educational Press Rubin D B 1987 Multiple imputation for non response in surveys New York John Wiley amp Sons Schmidt W H McKnight C C Valverde G A Houang R T and Wiley D E 1997 Many visions many aims A cross national investigation of curricular intentions in school mathematics Dordrecht the Netherlands Kluwer Academic Publishers Schmidt W H Raizen S A Britton E D Bianchi L J and Wolfe R G 1998 Many visions many aims A cross national investigation of curricular intentions in school science Dordrecht the Netherlands Kluwer Academic Publishers TIMSS 1995 Guide to checking coding and entering the TIMSS data Chestnut Hill MA Boston College TIMSS 1996a TIMSS mathematics items Released set for population 2 seventh and eighth grades Chestnut Hill MA Boston College TIMSS 1996b TIMSS science items Released set for population 2 seventh and eighth grades Chestnut Hill MA Boston College TIMSS 1997a TIMSS mathematics items Released set for population 1 third and fourth grades Chestnut Hill MA Boston College TIMSS 1997b TIMSS science items Released set for population 1 third and fourth grades Chestnut Hill MA
9. e Data Access Control Files e Jackknife Statistics Program Files e Scoring Program Files The Data Access Control files are provided to convert the ASCII format raw data files into SAS data sets or SPSS system files A different control file is required for each data file and the control files are named so that the first three characters match the first three characters of the respective data file The Jackknife Statistics Program files are used to compute the percentage of students within defined subgroups and the mean value for each subgroup on specified continuous variables as well as the standard errors associated with these statistics using the jackknife repeated replication JRR method discussed in Chapter 8 The Scoring Program files are required to convert cognitive item response codes to the score values used in the computation of international scores with one file provided for the written assessment items and one for the performance assessment items For all program files two versions are provided one for SAS programs and one for SPSS programs The file extension SAS or SPS is used to identify the respective SAS and SPSS program files Table 7 14 lists all program files provided Table 7 14 Population 1 and Population 2 Program Files Data Files Population 2 Population 1 File Name File Name I Data Access Control Files Control Files for the Student Written Assessment File BSACONTR ASACONTR Control Files for Student Back
10. proceeding with sections 6 1 and 6 2 TIMSS DATABASE USER GUIDE 6 1 CHAPTER 6 ACHIEVEMENT SCORES ASMSTDR Standardized mathematics raw score Population 1 ASSSTDR Standardized science raw score Population 1 BSMSTDR Standardized mathematics raw score Population 2 BSSSTDR Standardized science raw score Population 2 Because of the difficulty in making any comparisons across the test booklets using only the number of raw score points obtained on a set of items raw scores were standardized by booklet to provide a simple score which could be used in comparisons across booklets in preliminary analyses The standardized score was computed so that the weighted mean score within each booklet was equal to 50 and the weighted standard deviation was equal to 10 These standardized raw scores were used in the initial item analysis for computing the discrimination coefficients for each of the items in the test This initial item analysis was conducted prior to scaling the test items The standardized raw scores can be found in the Student Background data files and in the Written Assessment data files ASMNRSC National Rasch Mathematics Score ML Population 1 ASSNRSC National Rasch Science Score ML Population 1 BSMNRSC National Rasch Mathematics Score ML Population 2 BSSNRSC National Rasch Science Score ML Population 2 The national Rasch scores were also designed for preliminary analyses These provided a basic
11. 4 5 4 5 9 9 9 8 7 Breadth Science 9 minutes Mathematics Free Response 9 minutes Science Free Response 9 minutes NX Z caduomouoztzEn nzxc rgogonumoosgtu TIMSS DATABASE USER GUIDE Dd CHAPTER 2 INSTRUMENTS Table 2 3 Ordering of Item Clusters Within Population 1 Booklets Booklet Cluster Order The Population 1 test booklets were designed to be administered in two consecutive testing sessions with a 15 20 minute break between the sessions The order of the clusters within the Population 1 booklets is shown in Table 2 3 All booklets contain mathematics and science items The core cluster appears in the second position in all booklets The rotation design used to assign cluster B through H to booklets 1 through 7 allows the estimation of all item covariances for the items in cluster A through H Each of the focus clusters occurs once in the first third and fifth positions in booklets 1 through 7 There are free response clusters in Part 1 as well as in Part 2 of each test booklet fourth and seventh cluster in each booklet Booklet 8 which contains three breadth clusters serves primarily to increase the content coverage of the tests 2 6 TIMSS DATABASE USER GUIDE INSTRUMENTS CHAPTER 2 Table 2 4 Distribution of Item Types Across Clusters Population 2 Number Mathematics Items Number Science Items Cluster Type Cluster Label Multiple Short Extended Multiple Short Extended Choice
12. Assignment of Performance Assessment Tasks to Stations Pulse Dice Magnets Calculator Shadows Batteries Folding and Cutting Rubber Band Packaging Solutions Population 2 Containers Population 1 Around the Bend Plasticine Table 2 7 Assignment of Students to Stations in the Performance Assessment Student Sequence Rotation 1 Stations Rotation 2 Stations Number 1 2 3 4 5 6 7 8 9 TIMSS DATABASE USER GUIDE 2 9 CHAPTER 2 INSTRUMENTS 2 6 Release Status for TIMSS Test Items Performance Tasks and Background Questionnaires Some TIMSS items have been released for unrestricted public use At Populations 1 and 2 all items in clusters I through Z are classified as public release and are available to secondary users TIMSS 1996a 1996b 1997a 1997b Information about how to obtain the released item sets is provided at the end of this chapter All of the performance assessment tasks are contained in the performance assessment international report Harmon et al 1997 AII student teacher and school questionnaires are also classified as public release and are available to secondary users in Supplements 1 and 2 of this User Guide 2 7 Questionnaires The student questionnaires were separate instruments from the test booklets According to TIMSS test administration procedures students were to be given a break after completing their test booklets Subsequent to the break test administrators were to dis
13. For the multiple choice items one digit numerical values of 1 5 are used to correspond to the response options a e For these items the correct response is included as part of the item variable label in the codebook files For the free response written assessment items and the performance assessment items two digit numerical codes are used that correspond to the diagnostic scoring rubrics used to determine fully correct partially correct and incorrect responses for each item As described in Chapter 4 the correctness score level may be determined by the first digit of these codes 3 3 points 2 2 points 1 1 point 7 or 9 0 points In addition to the correctness score information specific missing codes are also defined that are described in the section discussing missing codes Section 7 2 5 Since all cognitive item variables are included for all students in the assessment files regardless of which test booklet or performance assessment task they completed a Not Administered code is given to all items that were not included in the test booklet or performance assessment tasks assigned to each student 7 2 2 6 Analysis By Mathematics and Science Content Area Reporting Categories The TIMSS cognitive items measured student achievement in different areas within mathematics and the sciences and student performance was presented in the international reports in several different content area reporting categories In order to permit
14. Hungary Iceland Iran Islamic Republic Ireland Israel Japan Korea Kuwait Latvia Netherlands New Zealand Norway Portugal Scotland Singapore Slovenia Thailand United States Performance Assessment Australia Canada Cyprus Hong Kong Iran Islamic Republic Israel New Zealand Portugal Slovenia United States IN TRODUCTION Population 2 Written Assessment Australia Austria Belgium Bulgaria Canada Colombia Cyprus Czech Republic Denmark England France Germany Greece Hong Kong Hungary Iceland Iran Islamic Republic Ireland Israel Japan Korea Kuwait Latvia Lithuania Netherlands New Zealand Norway Philippines Portugal Romania Russian Federation Scotland Singapore Slovak Republic Slovenia South Africa Spain Sweden Switzerland Thailand United States Performance Assessment Australia Canada Colombia Cyprus Czech Republic England Hong Kong Iran Islamic Rep Israel Netherlands New Zealand Norway Portugal Romania Scotland Singapore Slovenia Spain Sweden Switzerland United States The Flemish and French education systems in Belgium participated separately TIMSS DATABASE USER GUIDE IN TRODUCTION CHAPTER 1I 1 2 Overview of TIMSS The Third International Mathematics and Science Study TIMSS was conducted in 1995 across more than 40 countries TIMSS represents the continuation of a long series of studies conducted by the International Association
15. a single teacher questionnaire was developed to address both mathematics and science So as not to overburden the teachers the classroom practices questions in the Population 1 teacher questionnaire pertain mostly to mathematics However teachers also were asked about how they spend their time in school and the atmosphere in their schools e g teaching loads collaboration policies responsibilities for decision making and the availability of resources TIMSS DATABASE USER GUIDE ss sl CHAPTER 2 INSTRUMENTS The teacher questionnaires were designed to provide information about the teachers of the students sampled in TIMSS The teachers who completed the TIMSS questionnaires do not constitute a sample from any definable population of teachers Rather they represent the teachers of a national sample of students The school questionnaires for each population sought information about the school s community staff students curriculum and programs of study and instructional resources and time At Populations 1 and 2 the school questionnaires also ask about the number of years students are taught by the same teacher A school questionnaire was to be completed by the principal headmaster or other administrator of each school that participated in TIMSS TIMSS Released Item Sets The TIMSS released items are available in four volumes TIMSS Mathematics Items Released Set for Population 1 Third and Fourth Grades TIMSS Scien
16. jki jkindic wgt matwgt idcntry idgrader btbgtaug n Sy Sort cases by idcntry idgrader temporary select if idgrader 2 report format list automatic matwgt mnx mnx se pct pct se break idcntry idgrader var btbgtaug label Belgium Fl Canada Czech Republic France Hong Kong Iran Islamic Rep Italy Kuwait Netherlands Philippines Russian Federation Slovak Republic Spain Thailand TIMSS DATABASE USER GUIDE CHAPTER 9 PERFORMING ANALYSES As before we first proceed to identify the variables relevant to the analysis in the corresponding files and review the documentation on the specific national adaptations to the questions of interest Supplements 1 2 3 Since we are using teacher level variables we need to look into the teacher file and the Student Teacher Linkage files to find the variables From the mathematics teacher file we extract the variable that contains the information on the mathematics teachers years of experience BTBGTAUG the variable that identifies the country IDCNTRY and the two variables that will allow us to link the teacher information to the student data IDTEACH and IDLINK In Population 2 there is one teacher file for the mathematics teachers and a second teacher file for the science teachers If the user wants to look only at mathematics teachers then the user will need to use the mathematics teacher file BTM lt COUNTRY gt
17. the sum of the variable SENWGT within each country will add up to 1 000 The variable SENWGT within each country is proportional to TOTWGT by the ratio of 1 000 divided by the size of the population These sampling weights can be used when international estimates are sought and the user wants to have each country contribute the same amount to the international estimate When this variable is used as the sampling weight for international estimates the contribution of each country is the same regardless of the size of the population HOUWGT House Weight This variable is computed as TOTWGT _ TOTWGT within each country The transformation of the weights will be different within each country but in the end the sum of the variables HOUWGT within each country will add up to the sample size for that country The variable HOUWGT is proportional to TOTWGT by the ratio of the sample size divided by the size of the population These sampling weights can be used when the user wants the actual sample size to be used in performing significance tests Although some statistical computer software allow the user to use the sample size as the divisor in the computation of standard errors others will use the sum of the weights and this results in severely deflated standard errors for the statistics if the TOTWGT is used as the weighting variable When performing analyses using such software we recommend using the variable HOUWGT as the weight variab
18. 1 if the interest is in the science teachers then the user will need to use the science teacher file BTS lt COUNTRY gt 1 but if the interest is in the mathematics and science teachers combined both these files need to be combined by appending or adding one file to the other In doing so it is important to keep in mind that although there are variables in common between these two files most of them are not On the other hand there is only one file for the Population 1 teachers where both mathematics and science teacher variables are found ATG lt COUNTRY gt 1 In each population there is only one student teacher link file where the sampling and achievement information is found In our example our teacher variable of interest years of teaching experience is a continuous variable However we want to categorize the teachers into 4 groups 0 to 5 years experience 6 to 10 years experience 11 to 20 years experience and More than 20 years experience While reading the Teacher file we use commands in SAS or SPSS to collapse the different values into four categories and we label them accordingly We then proceed to read the necessary information from the Student Teacher Linkage file From this file we keep the country identification IDCNTRY and the two variables that will allow us to link the student information to the teacher data IDTEACH and IDLINK We also keep the variable that indicates the grade for the student IDGRADER the mathematic
19. 4 New Zealand Standard 2 3 5 4 5 Standard 3 4 5 5 5 Norway 2 2 3 3 Portugal 3 3 4 4 Scotland Year 4 4 Year 5 5 Singapore Primary 3 3 Primary 4 4 Slovenia 3 3 4 4 Thailand Primary 3 3 Primary 4 4 United States 3 3 4 4 1 Years of schooling based on the number of years children in the grade level have been in formal schooling beginning with primary education International Standard Classification of Education Level 1 Does not include preprimary education Australia Each state territory has its own policy regarding age of entry to primary school In 4 of the 8 states territories students were sampled from grades 3 and 4 in the other four states territories students were sampled from grades 4 and 5 3 In the Netherlands kindergarten is integrated with primary education Grade counting starts at age 4 formerly kindergarten 1 Formal schooling in reading writing and arithmetic starts in grade 3 age 6 New Zealand The majority of students begin primary school on or near their 5th birthday so the years of formal schooling vary SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 Information provided by TIMSS National Research Coord 9 2 TIMSS DATABASE USER GUIDE SAMPLING Table 3 2 Grades Tested in TIMSS Population 2 CHAPTER Upper Grade Country s Name for Upper Grade 80r9 4 Klasse 2A amp 2P 2A amp 2P 8 Oo OO 7 Year 9 4 me 90 or 4 me Technologique 10
20. 5 Greece 130221 7 121910 7 Hong Kong 88590 8 88573 7 Hungary 118726 6 112436 1 Iceland 4212 4 4234 2 Iran Islamic Rep 1052795 3 935093 1 Ireland 68476 8 67643 8 Israel 60584 3 Japan 1562417 6 1641941 4 Korea 798409 3 810403 8 Kuwait 13093 0 Latvia LSS 17041 1 15414 0 Lithuania 36551 0 39700 0 Netherlands 175419 1 191662 7 New Zealand 48507 6 51133 3 Norway 51165 0 50223 8 Portugal 146882 0 137459 0 Romania 295348 5 296533 6 Russian Federation 2168163 5 2004792 2 Scotland 61938 0 64637 6 Singapore 36181 0 36538 5 Slovak Republic 83074 1 79766 4 Slovenia 28048 9 26010 7 South Africa 649180 0 766333 6 Spain 549032 2 547113 6 Sweden 96493 9 98193 1 Switzerland 66681 1 69732 5 Thailand 680225 3 657748 2 United States 3156846 8 3188296 6 3 14 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER c3 3 8 Weight Variables Included in the Student Data Files There are several sampling weight variables included in the student data files Some of these variables capture different aspects of the sampling process and others constitute the sampling weights themselves For the purpose of consistency the variable names across both populations are identical so the explanation that follows applies to the Population 1 and Population 2 data files The variables named in this section are included in the Student Background and the Performance Assessment data files Their values will vary across these files even for the same students because the
21. 5 Instrument Deviations and National Adaptations ssssseseseseeeeeeeenee 4 7 4 5 HogniliVe d ems rtt ads ete ce n dd Re ee QUII Ute doa ac 4 7 4 5 2 Backgrourid Questionnairellems ce tete itc ctetu tutt 4 10 TIMSS DATABASE USER GUIDE CONTENTS Chapter 5 TIMSS Scaling Procedures cssscssssssssccesecsssessscescssesessssecsssecessscesssseeseees D l Del The TIMSS Scaling Model 2 3 ttt ede te edet ee 5 2 The Unidimensional Random Coefficients Model sss 5 3 The Multidimensional Random Coefficients Multinomial Logit Model 5 4 The Population Model 5 5 Estiration i cet ete E e cero fecta Fonte De ete doe de e 5 6 latent Estimation and Prediction ie EU bene eheu teri ded 5 7 jDrawirng Plausible Values oett eH t Ue EOE ERR 5 8 Scaling Steps cue Une REDE Un E eee eae ees 5 8 1 Drawing The International Calibration Sample et 5 8 2 Standardizing the International Scale Scores ttn Chapter 6 Student Achievement Scores ss sessssessesesesesseseseseseesesesesscseseososeseosesesessesese T 6 1 6 2 Achievement Scores in the Student Files ccccccececessccesscccssscessecesseccsssecessccessecessseessscsssecesssessass 6 1 Achievement Scores in the School Background Files sss 6 9 Chapter 7 Content and Format of Database Files esas Z l 7 1 Introduction 7
22. 513 45 7 51 36 10 3 31 Slovenia Eighth Grade 0 to 5 years 839 82 536 53 23 22 3 85 1 90 6 to 10 years 4158 14 532 59 5 97 19 06 4 03 11 to 20 years 12025 34 541 67 5 48 55 13 4 98 Over 20 years 4789 01 550 44 6 17 21 96 3 82 9 32 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 In summary to perform analyses such as those in Figures 9 15 and 9 16 using the Student and Teacher Background data files the user needs to do the following Identify the variable or variables of interest in the corresponding teacher file and find out about any specific national adaptations to the variable Retrieve the relevant variable or variables from the teacher data files There is one teacher file for the teachers of the Population 1 students and two for the teachers of the Population 2 students If the user is interested in looking at both mathematics and science teachers combined for the Population 2 students then the files for these teachers need to be added or appended to each other e Retrieve the relevant variables from the Student Teacher Linkage file This includes the identification information for the country and teacher IDCNTRY IDTEACH and IDLINK the achievement score JRR replication information and the sampling weight If the analysis is to be based on mathematics teachers only then the weight variable to use is MATWGT If the analysis is to be based on the science teachers only then the weight variable to be used is SCIWGT
23. 8 In addition Chapter 9 includes sample analyses using these variables TIMSS DATABASE USER GUIDE Feld CHAPTER 7 DATABASE FILES 7 2 2 Assessment Files Student assessment files contain the student response data for the individual cognitive items in the TIMSS written assessment and the performance assessment Four student assessment files for each country are contained in the TIMSS International Database e Student written assessment file Population 1 e Student written assessment file Population 2 e Student performance assessment file Population 1 e Student performance assessment file Population 2 7 2 2 1 Written Assessment Files Students who participated in TIMSS were administered one of eight test booklets with questions in mathematics and science Some of these questions were multiple choice questions and some were open ended The responses to the open ended questions were coded using a two digit coding system described in Chapter 4 The written assessment data files contain the answers to the multiple choice questions and the codes assigned by the coders to the student responses Since under the TIMSS test design a student received only a fraction of the total test item pool the variables for the items that were not included in the version of the test that was administered to the student are coded as not administered The specific test booklet that was administered to the student is coded in the variable IDBOOK The writ
24. 9 29 Figure 9 15 Extract of SAS Computer Output for Performing Analyses with Teacher Level V riablesAEXAMPLE 2 3 rete ee reete ene eed tete e ede 9 31 Extract of SPSS Computer Output for Performing Analyses with Teacher Level Variables EXAMPLE 2 5i estet etr dert RE TOP TEUER 9 32 TIMSS DATABASE USER GUIDE V CONTENTS Figure 9 17 SAS Control Statements for Performing Analyses with School Level Variables FAB SSS seco nc atc resus ca a Eos ruptis tot RES EE P PUE 9 35 Figure 9 18 SPSS Control Statements for Performing Analyses with School Level Variables TEAM PLES SPS els or pae tare uU LM ce a este nese re 9 36 Figure 9 19 Extract of SAS Computer Output for Performing Analyses with School Level Variables EXAMPBLE 3 ihid eee cetaceans REOR NEED 9 37 Figure 9 20 Extract of SPSS Computer Output for Performing Analyses with School Level Variables EXAMPLES 5t cos oett tetro t tte oT AR Mee det Lote od 9 38 Table 9 3 Definitions of Response Codes for the Multiple Choice Items in the Written Assessment Datahiles eus ioco ser nee a rero e mc aer bte eot uo eu esee le M UE 9 39 Table 9 4 Definition of Response Codes for the Open Ended Items in the Written Assessment and Performance Assessment Data Files sse 9 40 Figure 9 21 Extracted Sections of SAS Control Code Used to Convert Cognitive Item Response Codes to Correctness Score Levels sss 9 42 Figure 9 22 Extracted Sections of SPSS Control
25. Africa Spain Jos Antonio Lopez Varona Instituto Nacional de Calidad y Evaluaci n C San Fernando del Jarama No 14 28002 Madrid Spain ACKNOWLEDGMENTS Sweden Ingemar Wedman Anna Hofslagare Kjell Gisselberg Umea University Department of Educational Measurement S 901 87 Umea Sweden Switzerland Erich Ramseier Amt Fiir Bildungsforschung der Erziehungsdirektion des Kantons Bern Sulgeneck Stra e 70 Ch 3005 Bern Switzerland Thailand Suwaporn Semheng Institute for the Promotion of Teaching Science and Technology 924 Sukhumvit Road Bangkok 10110 Thailand United States William Schmidt Michigan State University Department of Educational Psychology 463 Erikson Hall East Lansing MI 48824 1034 United States TIMSS DATABASE USER GUIDE ACKNOWLEDMENTS TIMSS ADVISORY COMMITTEES The International Study Center was supported in its work by several advisory committees The International Steering Committee provided guidance to the International Study Director on policy issues and general direction of the study The TIMSS Technical Advisory Committee provided guidance on issues related to design sampling instrument construction analysis and reporting ensuring that the TIMSS methodologies and procedures were technically sound The Subject Matter Advisory Committee ensured that current thinking in mathematics and science education were addressed by TIMSS and was instrumental in the development of the TI
26. Answer Response Choice Answer Response gt Core 12 Minutes Focus 12 minutes Breadth Mathematics and Science 22 minutes 6 6 6 6 6 6 6 7 7 7 9 7 7 7 9 9 7 Noawof WAN DAN OD DD DD OD OD N HA NYONYNNY HA NY DY PD Mathematics Free Response 10 minutes Science Free Response 10 minutes N XxZzZ cadaommouozzmn nzxc rgonmoosugu The booklet design for Population 2 is very similar to that for Population 1 Of the 26 clusters in Population 2 eight take 12 minutes ten take 22 minutes and eight take 10 minutes The core cluster cluster A comprising six mathematics and six science multiple choice items appears in the second position in every booklet The seven focus clusters appear in at least three booklets and the ten breadth clusters appears in only one booklet The eight free response clusters each containing 10 minutes of short answer and extended response items were each assigned to two booklets Tables 2 4 and 2 5 show the number of items in each cluster and the assignment of clusters to booklets respectively TIMSS DATABASE USER GUIDE 2 7 CHAPTER 2 INSTRUMENTS Table 2 5 Ordering of Clusters Within Population 2 Booklets Booklet Cluster Order 2 5 Performance Assessment The TIMSS performance assessment was administered at Populations 1 and 2 to a subsample of students in the upper grades that participated in the written assessment Harmon and Kelly 1996 Harmon
27. Boston College TIMSS DATABASE USER GUIDE REFERENCES Wilson M R 1992 The ordered partition model An extension of the partial credit model Applied Psychological Measurement 16 3 Wolter K M 1985 Introduction to variance estimation New York Springer Verlag Wu M L Adams R J and Wilson M 1997 Conquest Generalized item response modelling software Manual Melbourne Australian Council for Educational Research TIMSS DATABASE USER GUIDE Acknowledgments TIMSS was truly a collaborative effort among hundreds of individuals around the world Staff from the national research centers the international management advisors and funding agencies worked closely to design and implement the most ambitious study of international comparative achievement ever undertaken TIMSS would not have been possible without the tireless efforts of all involved The TIMSS performance assessment was an integral part of the study and one that required a great deal of additional resources and effort for all involved in that component Below the individuals and organizations are acknowledged for their contributions to TIMSS Given that implementing TIMSS has spanned more than seven years and involved so many people and organizations this list may not pay heed to all who contributed throughout the life of the project Any omission is inadvertent TIMSS also acknowledges the students teachers and school principals who contribut
28. Colombia Germany Germany Romania Romania Slovenia Slovenia Countries with unapproved sampling procedures at the classroom level Denmark Denmark Greece Greece Thailand South Africa Thailand Countries with unapproved sampling procedures at classroom level and not meeting other guidelines Israel Kuwait South Africa Countries with unapproved sampling procedures at school level Phillipines Phillipines Met guidelines for sample participation rates only after replacement schools were included 1 National Desired Population does not cover all of International Desired Population Because coverage falls below 65 Latvia is annotated LSS for Latvian Speaking Schools only National Defined Population covers less than 90 percent of National Desired Population 3 TIMSS was unable to compute sampling weights for the Philippines SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER Figure 3 4 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Performance Assessment Eighth Grade Fourth Grade Countries satisfying guidelines for sample participation rates grade selection and sampling procedures Canada Canada Cyprus Cyprus Czech Republic Iran Islamic Republic Iran Islamic Republic t New Zealand New Zealand Portugal Norway Portugal S
29. D 5 And for a response vector we have f x 0 0 8 exp x B8 Ag 1 with V 0 amp X orto as 7 The difference between the unidimensional model and the multidimensional model is that the ability parameter is a scalar 0 in the former and a D by one column vector 0 in the latter TIMSS DATABASE USER GUIDE 552g CHAPTER 5 SCALING Likewise the scoring function of response k to item i is a scalar b in the former whereas it is a D by 1 column vector b in the latter 5 4 The Population Model The item response model is a conditional model in the sense that it describes the process of generating item responses conditional on the latent variable 0 The complete definition of the TIMSS model therefore requires the specification of a density f 6 0 for the latent variable 0 We usea to symbolize a set of parameters that characterize the distribution of 0 The most common practice when specifying unidimensional marginal item response models is to assume that the students have been sampled from a normal population with mean m and variance s That is f a f 0 10 exp cur 8 0 gt 0 gt Ino 20 or equivalently 0 u E 9 where E N 0 0 Adams Wilson and Wu 1997 discuss how a natural extension of 8 is to replace the mean m with the regression model Y p where Y is a vector of u fixed and known values for student n and b is the corresponding vector of regressio
30. Dehairs Rijksuniversiteit Ghent Vakgroep Onderwijskunde amp The Ministry of Education Henri Dunantlaan 2 B 9000 Ghent Belgium Belgium French Georges Henry Christian Monseur Universit de Li ge B32 Sart Tilman 4000 Li ge 1 Belgium Bulgaria Kiril Bankov Foundation for Research Communication Education and Informatics Tzarigradsko Shausse 125 Bl 5 1113 Sofia Bulgaria Canada Alan Taylor Applied Research amp Evaluation Services University of British Columbia 2125 Main Mall Vancouver B C V6T 1Z4 Canada Past National Research Coordinator Colombia Carlos Jairo Diaz Universidad del Valle Facultad de Ciencias Multitaller de Materiales Didacticos Ciudad Universitaria Mel ndez Apartado Aereo 25360 Cali Colombia Cyprus Constantinos Papanastasiou Department of Education University of Cyprus Kallipoleos 75 P O Box 537 Nicosia CY 1789 Cyprus Czech Republic Jana Strakova Vladislav Tomasek Institute for Information on Education Senovazne Nam 26 111 21 Praha 1 Czech Republic Denmark Peter Weng Peter Allerup Borge Prien The Danish National Institute for Educational Research 28 Hermodsgade Dk 2200 Copenhagen N Denmark England Wendy Keys Derek Foxman National Foundation for Educational Research The Mere Upton Park Slough Berkshire SL1 2DQ England France Anne Servant Minist re de l Education Nationale 142 rue du Bac 75007 Paris France Jo
31. Eighth Grade Community Type 1 303 99 605 69 2 14 38 28 Community Type 2 38258 26 526 10 5 00 48 05 3 66 Community Type 3 7440 07 537 12 20 98 9 34 2 78 Community Type 4 33618 58 551 57 5 22 42 22 4 01 Belgium F1 Eighth Grade Community Type 2 22954 95 562 50 7 24 33 59 4 82 Community Type 3 16854 23 584 66 7 79 24 66 4 43 Community Type 4 28525 47 553 32 13 93 41 74 6 07 Belgium Fr Eighth Grade Community Type 2 10906 33 515 15 18 88 23 31 5 03 Community Type 3 10109 43 518 49 10 38 21 61 4 91 Community Type 4 25776 11 539 30 5 98 55 09 5 44 United States Eighth Grade Community Type 1 90271 12 2466 31 21 13 3 26 1 66 Community Type 2 587437 27 509 64 6 10 21 19 3 39 Community Type 3 898598 98 513 36 6 50 32 42 4 11 Community Type 4 1195464 02 498 18 7 25 43 13 4 40 Slovenia Eighth Grade Community Type 1 1630 53 537 02 18 13 7 50 2 34 Community Type 2 5435 88 521 50 6 88 25 00 2 93 Community Type 3 4860 04 559 01 6 20 22 35 4 23 Community Type 4 9815 66 541 87 4 32 45 15 3 81 9 38 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 In summary to perform analyses such as those in Figures 9 17 and 9 18 using the Student and School Background files the user needs to do the following e Identify the variable or variables of interest in the student file and find out about any specific national adaptations to the variable e Retrieve the relevant variables from the student files including the achievement score sampling weigh
32. Grade TS value seetv 2 Less than 1 Hr 3 1 to 2 hrs 4 3 to 5 hrs i 5 More than 5 hrs value country 036 Australia 040 Austria 056 Belgium Fl d 057 Belgium Fr 100 Bulgaria 124 Canada T 170 Colombia 196 Cyprus 200 Czech Republic 208 Denmark 826 England 250 France M 280 Germany 300 Greece 344 Hong Kong 348 Hungary 352 Iceland 364 Iran Islamic Rep 372 Ireland 376 Israel 380 Italy p 392 Japan 410 Korea 414 Kuwait n 428 Latvia LSS 440 Lithuania 528 Netherlands 554 New Zealand 578 Norway 608 Philippines 620 Portugal 642 Romania 643 Russian Federation 827 Scotland 702 Singapore 201 Slovak Republic 890 Slovenia 717 South Africa 724 Spain 752 Sweden 756 Switzerland 764 Thailand 840 United States Now use the macro JACK to get the results include jack sas jack totwgt jkzone jkindic 75 idcntry idgrader bsbgdayl bimatscr student proc print data final noobs by idcntry idgrader where idgrader 2 var bsbgdayl N totwgt mnx mnx se pct pct se format idcntry country idgrader grade bsbgdayl seetv TIMSS DATABASE USER GUIDE 9 23 CHAPTER 9 PERFORMING ANALYSES Figure 9 10 SPSS Control Statements for Performing Analyses with Student Level Variables EXAMPLE1 SPS get file bsgalll sys keep idcntry
33. If the analysis is to be based on the science and mathematics teachers combined then the weight variable to be used is TCHWGT Merge the variables from the teacher data files into the Student Teacher Linkage files using the variables IDCNTRY IDTEACH and IDLINK e Use the macro JACK with the corresponding arguments and parameters Print out the result file 9 6 Performing Analyses with School Level Variables Although the students in the TIMSS samples were selected from within a sample of schools the school level variables should only be analyzed with respect to the number of students attending schools of one type or another In other words the school level data should not be analyzed to make statements about the number of schools with certain characteristics but rather to make statements about the number of students attending schools with one characteristic or another When school level variables are analyzed we recommend that the user merge the selected school level variables with the student level file and then use the sampling and weight information contained in the student level file to make the desired statements The examples presented in this section describe how this can be accomplished using SAS or SPSS Lets us say that we want to find out the percent of eighth graders that attend schools located in a certain geographical area of the country BCBGCOMM and their average achievement in mathematics As in the previous exampl
34. Procedures The principal method by which student achievement is reported in TIMSS is through scale scores derived using Item Response Theory IRT scaling With this approach the performance of a sample of students in a subject area can be summarized on a common scale or series of scales even when different students have been administered different items The common scale makes it possible to report on relationships between students characteristics based on their responses to the background questionnaires and their overall performance in mathematics and science Because of the need to achieve broad coverage of both subjects within a limited amount of student testing time each student was administered relatively few items within each of the content areas of each subject In order to achieve reliable indices of student proficiency in this situation it was necessary to make use of multiple imputation or plausible values methodology Further information on plausible value methods may be found in Mislevy 1991 and in Mislevy Johnson and Muraki 1992 The proficiency scale scores or plausible values assigned to each student are actually random draws from the estimated ability distribution of students with similar item response patterns and background characteristics The plausible values are intermediate values that may be used in statistical analyses to provide good estimates of parameters of student populations Although intended for use in pla
35. Psychometrika 59 149 176 Lie S Taylor A and Harmon M 1996 Scoring techniques and criteria In M O Martin and D L Kelly Eds TIMSS technical report volume I Design and development Chestnut Hill MA Boston College Martin M O and Kelly D L Eds 1996 TIMSS technical report volume I Design and development Chestnut Hill MA Boston College Martin M O and Kelly D L Eds 1997 TIMSS technical report volume IT Implementation and analysis Chestnut Hill MA Boston College Martin M O and Mullis I V S Eds 1996 TIMSS Quality assurance in data collection Chestnut Hill MA Boston College Martin M O Mullis I V S Beaton A E Gonzalez E J Smith T A and Kelly D L 1997 Science achievement in the primary school years IEA s Third International Mathematics and Science Study Chestnut Hill MA Boston College Maxwell B 1996 Translation and cultural adaptation of the survey instruments In M O Martin and D L Kelly Eds TIMSS technical report volume I Design and development Chestnut Hill MA Boston College Mislevy R J 1991 Randomization based inference about latent variables from complex samples Psychometrika 56 177 196 Mislevy R J Beaton A E Kaplan B and Sheehan K M 1992 Estimating population characteristics from sparse matrix samples of item responses Journal of Educational Measurement 29 2 133 161 Mislevy R J and Sheehan K M 198
36. Rasch score for preliminary analyses within countries but could not be used for international comparisons since each country has been assigned the same mean score The national Rasch scores were computed by standardizing mathematics and science logit scores to have a weighted mean of 150 and a standard deviation of 10 within each country The logit scores were computed using the Quest Rasch analysis software Quest provides maximum likelihood ML estimates of a scaled score based on the Rasch model for the performance of the students on a set of items The computation took into account the varying difficulty of the items across test booklets and the performance and ability of the students responding to each set of items These logit scores were obtained using item difficulties that were computed for each country using all available item responses for the country and centering the item difficulty around zero When computing the item difficulties responses marked as not reached were treated as items that were not administered This avoids giving inflated item difficulties to the items located at the end of the test in cases where students systematically do not reach the end of the test These item difficulties were then used to compute logit scores for each student When computing the student logit scores the responses marked as not reached were treated as incorrect responses This avoided unfairly favoring students who started answering the te
37. Teachers Reports on Their Years of Teaching Experience Mathematics Upper Grade Eighth Grade 0 5 Years 6 10 Years 11 20 Years More than 20 Years Country Australia Austria Belgium FI Belgium Fr Canada Colombia Cyprus Czech Republic Denmark England France Germany Greece Hong Kong Hungary Iceland Iran Islamic Rep Ireland Israel Japan Korea Kuwait Latvia LSS Lithuania Netherlands New Zealand Norway Portugal Romania Russian Federation Scotland Singapore Slovak Republic Slovenia Spain Sweden Switzerland Thailand United States Eighth grade in most countries see Table 2 for more information about the grades tested in each country Countries shown in italics did not satisfy one or more guidelines for sample participation rates age grade specifications or classroom sampling procedures see Figure A 3 Background data for Bulgaria and South Africa are unavailable Because population coverage falls below 65 Latvia is annotated LSS for Latvian Speaking Schools only Standard errors appear in parentheses Because results are rounded to the nearest whole number some totals may appear inconsistent An r indicates teacher response data available for 70 84 of students An s indicates teacher response data available for 50 69 of students oot POO OI coco walt aNoaocla ROG 1 550 89 So ola o Qj on 20 w ROAR o O9 o o m o o Ww O O qv 0o Q A 0 O 0 O jo
38. The data collected during these visits are presented in Martin and Mullis 1996 together with extensive documentation of the quality of the translation activities population sampling scoring data checking and database construction 4 1 Data Collection and Field Administration For the sake of comparability all testing was conducted at the end of the school year The four countries on a Southern Hemisphere school schedule Australia Korea New Zealand and Singapore tested in September through November of 1994 which was the end of the school year in the Southern Hemisphere The remaining countries tested their students at the end of the 1994 95 school year most often in May and June of 1995 In addition to selecting the sample of students to be tested the NRCs were responsible for working with the School Coordinators translating the test instruments assembling and printing the test booklets and packing and shipping the necessary materials to the designated School Coordinators They also were responsible for arranging for the return of the testing materials from the school sites to the national center preparing for and implementing the free response scoring entering the results into data files conducting on site quality assurance observations for a 1096 sample of schools and preparing a report on survey activities TIMSS DATABASE USER GUIDE 4 1 CHAPTER 4 DATA COLLECTION PROCEDURES The Survey Operations Manual for Populations
39. U S National Center for Education Statistics the U S National Science Foundation the IEA and the Canadian government Each participating country provides funding for the national implemenation of TIMSS Boston College is an equal opportunity affirmative action employer Printed and bound in the United States Contents Chapter 1 Overview of TIMSS and the Database User Guide 1 1 1 1 Overview of the International Database sse 1 1 1 2 Overview oETIMSS s cette dtr dei etc ET OT Gc En 1 3 3 HMSS International Reports ttr tee e bt e e let met 1 5 14 Contents ofthe Database i trt LR Rete te ecd Du ea eed e 1 5 1 5 Conterits of the User Guides eds i e cie e tee diee tint ie dudes 1 6 1 6 Management and Operations of TIMSS ssssssssssesesee eerte neret 1 8 TZ Additional Resources 21 cerae DURO Oe Boc abe A eee edd 1 8 Chapter 2 TIMSS Instruments and Booklet Design eere 2 1 2 4 Introductio s o ere eit e D dte OM t de e de 2 1 2 2 The TIMSS Mathematics and Science Content sss 2 1 2 9 The TIMSSMfems nest eret hte pt e wi ete tree etta 2 3 2 4 Organization of the Test Booklets ccecessesssesesesteeseseseseseeeeecseseeeseeeseeeaeaseeeesesesaeacacaneseneeeeeaes 2 4 2 5 Peformance Assessment 1 ete ded re cath URE AA eres 2 8 2 6 Release Status for TIMSS Test Items Performance Tasks and Background Questi
40. Vl Vx Task M2 Calculator BSPM25 Item 5 no code 11 Recoded 11 codes used in some countries Error in coding guide valid codes listed as 10 12 19 Population 2 Task M5 Packaging BSPM51 Two versions of task used across countries original asked for Item 1 ASPM51 2 OR 3 boxes revised asked for 3 Item changed to 2 point value Populations 2 amp 1 for report tables changed codes for 3 correct boxes 30 31 to 2 point codes 22 23 Task S5 Solutions BSPS52A Administrator notes not coded consistently across countries Item 2A invalid 99 codes blank used in several countries recoded to Population 2 not administered Item omitted from report table but kept in data file Task S5 Solutions BSPS54 Coding guide revised based on reports of problematic scoring Item 4 during training development Population 2 Task S6 Containers ASPS61A Administrator notes not coded consistently across countries Item 1A invalid 99 codes blank used in several countries recoded to Population 1 not administered Performance Assessment Items TIMSS DATABASE USER GUIDE Z 27 CHAPTER 7 DATABASE FILES Other Variables in the Reliability Files In addition to the coding reliability variables the reliability files also include identification variables to aid in case identification Some tracking variables are also included that were used in conducting of the coding reliability study within each country including the reliability
41. When creating the replicate weights the following procedure is followed 1 Each sampled student is assigned a vector of 75 weights or We J where h takes values from 1 to 75 2 The value of Ws is the overall sampling weight which is simply the product of the final school weight the appropriate final classroom weight and the appropriate final student weight as defined in the chapter on sampling and sampling weights TIMSS DATABASE USER GUIDE 8 3 CHAPTER 8 SAMPLING VARIANCE 3 The replicate weights for a single case are then computed as Ws Wg tk where the variable k for an individual i takes the value k 2 u if the record belongs to zone h and k 1 otherwise In TIMSS a total of 75 replicate weights were computed regardless of the number of actual zones within each country If a country had fewer than 75 zones then the replicate weights W where h was greater than the number of zones within the country were each the same as the overall sampling weight Although this involves some redundant computation having 75 replicate weights for each country has no effect on the size of the error variance computed using the jackknife formula but facilitates the computation of standard errors for a number of countries at one time 8 4 TIMSS DATABASE USER GUIDE Chapter 9 Performing Analyses with the TIMSS Data Some Examples This chapter presents some basic examples of analyses that can be performed with the TIMSS Int
42. amp v NSARSSRAS S GSS SSAGS ONS SEV EoNoyossloveagn aes oo SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUIDE 9 3 CHAPTER 9 PERFORMING ANALYSES 9 1 Contents of the CDs There are two CDs that accompany this User Guide one CD containing the Population 1 data and one containing the Population 2 data Each CD has the following internal file structure e A main directory identifying the population POP 1 or POP 2 e Within each main directory there are five sub directories DATA Contains ASCII data files PROGRAMS Contains SPSS and SAS programs CODEBOOK Contains codebooks ALMANACS Contains data almanacs TCMA Contains Test Curriculum Matching Analysis Data The directory names within each CD and the file names follow the DOS naming convention file names with up to eight characters followed by a three character extension as in FILENAME EXT Files with the same names are related to each other and the extension identifies their function The extensions used in the files contained in the CDs are indicated in Table 9 1 below Table 9 1 Three letter Extension Used to Identify the Files Contained in the CD Extension Description SAS Control File or program SPSS Control File or program ASCII data file Almanac Codebook in printout format Codebook in machine readable format The DATA sub directory contains the TIMSS d
43. and the analysis conducted using a student level file Students in the extra grade are those students above the upper grade in the TIMSS sample Sweden and Switzerland tested students at this grade level and those data are included in the database TIMSS DATABASE USER GUIDE 3 19 Chapter 4 Data Collection Materials Processing Scoring and Database Creation Each country participating in TIMSS was responsible for collecting its national data and processing the materials in accordance with the international standards In each country a national research center and National Research Coordinator NRC were appointed to implement these activities One of the main ways in which TIMSS sought to achieve uniform project implementation was by providing clear and explicit instructions on all operational procedures Such instructions were provided primarily in the form of operations manuals supported where possible by computer software systems that assisted NRCs in carrying out the specified field operations procedures Forms accompanying some of the manuals served to document the implementation of the procedures in each country Many of these forms were used to track schools students and teachers and to ensure proper linkage of schools students and teachers in the database As part of the TIMSS quality assurance efforts there also was a program of site visits by trained Quality Assurance Monitors representing the International Study Center
44. any modified items This review process was instituted for the majority of the background questionnaire items and in particular for any of the items that were used for reporting the contextual data in the international reports Each item deviation or national adaptation was individually reviewed in order to determine which of the following actions should be taken e National data deleted as not being internationally comparable e National data recoded to match the international version e National data retained with some documentation describing modifications a pe TIMSS DATABASE USER GUIDE DATA COLLECTION PROCEDURES CHAPTER 4 Whenever possible national data were retained by recoding to match as closely as possible the international version of the items and or by documenting minor deviations NRCs were contacted to resolve questions regarding the international comparability of revised items and no changes were made to the international data files without first informing the NRC and receiving confirmation whenever possible A summary of all available documentation for the deleted or modified background questionnaire items in the international data files is provided in Supplement 3 This documentation was sent to all NRCs for review and verification prior to release of the international data files TIMSS DATABASE USER GUIDE 4 1 CHAPTER 4 DATA COLLECTION PROCEDURES 4 12 TIMSS DATABASE USER GUIDE Chapter 5 TIMSS Scaling
45. below in Figure 4 1 In this guide students received one point for a correct answer and then there were multiple diagnostic codes for incorrect answers The rubrics for more complicated items were correspondingly more complicated having categories for full and partial credit As shown in Figure 4 2 on both parts of this extended mathematics task students received two points for a fully correct answer and one point for a partially correct answer In some cases the scoring guides include three points Figure 4 1 Example Coding Guide for Short Answer Mathematics Item L16 Find x if 10x 15 5x 20 Answer Correct ee Sed p Treue 90 Crossed out erased illegible or impossible to interpret 39 BLANK TIMSS DATABASE USER GUIDE 4 3 CHAPTER 4 DATA COLLECTION PROCEDURES Figure 4 2 Example Coding Guide for Extended Response Mathematics Item Length 6cm In the space below draw a new rectangle whose length is one and one half times the length of the rectangle above and whose width is half the width of the rectangle above Show the length and width of the new rectangle in centimeters on the figure b What is the ratio of the area of the new rectangle to the area of the first one Show your work Note There is no distinction made between responses with and without units A Codes for Drawing Correct Re
46. et al 1997 The performance tasks permitted students to demonstrate their ability to make record and communicate observations to take measurements or collect experimental data and present them systematically to design and conduct a scientific investigation or to solve certain types of problems A set of 13 such hands on activities was developed and used with subsamples of students at fourth and eighth grades Eleven of the tasks were either identical or similar across populations and two tasks were different Of these two one task was administered to the Population 1 fourth graders and one was administered to Population 2 eighth graders The 12 tasks administered at each population were presented at nine different stations Each station required about 30 minutes working time Each student was assigned to three stations by a sequence number for a total testing time of 90 minutes Because the complete circuit of nine stations occupies nine students students participating in the performance assessment were sampled in sets of nine However the complete rotation of students required two sets of 9 or 18 students to assure that each task was paired with each other task at least once Taken together Tables 2 6 and 2 7 show the stations each student visited and the tasks completed according to the rotation assignment either Rotation 1 or Rotation 2 and sequence number 2 8 TIMSS DATABASE USER GUIDE INSTRUMENTS CHAPTER 2 Table 2 6
47. five values are provided in the International Database iei TIMSS DATABASE USER GUIDE Chapter 6 Student Achievement Scores The TIMSS international database contains several student level achievement scores These scores were computed at different stages of the study to serve specific purposes This chapter presents a description of these achievement scores how they were derived how they were used by TIMSS and how they can be used by users of the database For identification purposes the first letter for the variable name identifies the population for which the score was computed The scores computed for Population 1 have the letter A as the first character in their name and scores for Population 2 have the letter B as the first character This convention was followed with other background and derived variables and with the files included in the database 6 1 Achievement Scores in the Student Files Six types of achievement scores are included in the student data files raw scores standardized raw scores national Rasch scores plausible values international EAP scores and international proficiency scores Each type is described below ASMSCPT Number of raw score points obtained on the mathematics items Population 1 ASSSCPT Number of raw score points obtained on the science items Population 1 BSMSCPT Number of raw score points obtained on the mathematics items Population 2 BSSSCPT Number of raw score points o
48. guess if they do not know the answer In the free response items students were asked to construct their own responses to the test questions by writing or drawing their answers These included short answer items and items where students were asked to provide extended responses The free response items were scored using the two digit coding system developed by TIMSS see Chapter 4 At Population 1 there were 102 mathematics items including 79 multiple choice items 15 short answer items and 8 extended response items The science test contained 97 items of which 13 were classified as requiring short answers and 10 as requiring more extended responses In all there is a total pool of 235 unique testing minutes in Population 1 118 for mathematics and 117 for science At Population 2 the overall pool of cognitive items contained 151 mathematics items including 125 multiple choice items 19 short answer items and 7 extended response items There were 135 science items including 102 multiple choice items and 33 free response items Population 2 contained a total of 396 unique testing minutes 198 for mathematics and 198 for science 2 4 Organization of the Test Booklets At each population the test items were allocated to 26 different clusters labeled A through Z Also at each population the 26 clusters were assembled into eight booklets Each student completed one booklet At Population 1 students were given 64 minutes to complete their book
49. have a test in SQ2 s forms general and math related questions mathematics Population 2 Student Questionnaire variables only in SQ2 form SQ2G SQ2G 31C How often do you have a test in questions related to integrated or general science science Population 2 Student Questionnaire variables in only the SQ2 s SQ2S SQ2S 31C How often do you have a test in form questions related to specific science subject biology areas Population 2 Mathematics Teacher Questionnaire TQM2A TQM2A17A Familiarity with National TQM2B Curriculum Guide for TQM2C Mathematics TQM2D Population 2 Science Teacher Questionnaire TQS2A TQS2A17A Familiarity with National TQS2B Curriculum Guide for Science TQS2C TQS2D Population 2 School Questionnaire SCQ2 SCQ2 1 In what type of community is school located International Background Variable Names The naming system for the background variables permits the determination of the population and questionnaire based on 7 or 8 digit codes according to the general definitions given in Table 7 4 TIMSS DATABASE USER GUIDE Zeli CHAPTER 7 DATABASE FILES Table 7 4 International Background Variable Naming Conventions Character Values Position Definition Student Variables Teacher Variables School Variables Population A Population 1 A Population 1 A Population 1 B Population 2 B Population 2 B Population 2 Questionnaire Type S T C Background Variable B B B Subj
50. have been in formal schooling beginning with primary education International Standard Classification of Education Level 1 Does not include preprimary Australia Each state territory has its own policy regarding age of entry to primary school students were sampled from grades 7 and 8 in the other four states territories students were sampled from grades 8 and 9 3 Indicates that there is a system split between the lower and upper grades In Cyprus system split occurs only in the large or city schools In Switzerland there is a system split in 14 of 26 cantons New Zealand The majority of students begin primary school on or near their 5th birthday so the years of formal schooling vary 5 Russian Federation 70 of students in the seventh grade have had 6 years of formal schooling 70 in the eighth grade have had 7 years of formal schooling SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 Information provided by TIMSS National Research Coordinators TIMSS DATABASE USER GUIDE 3 3 CHAPTER 3 SAMPLING TIMSS used a grade based definition of the target populations In a few cases TIMSS components were administered only to the upper grade of these populations i e the performance assessment was conducted at the upper grade and some background questions were asked of the upper grade students only However two adjacent grades were chosen to ensure extensive coverage of the same age cohort for most c
51. idstud idgrader jkindic jkzone totwgt bsbgdayl bimatscr recode bsbgdayl 1 222 8 9 sysmis else copy select if not missing bsbgdayl value labels idgrader 1 Seventh Grade 2 Eighth Grade vox bsbgdayl 2 Less than 1 Hr 3 1 to 2 hrs x 4 3 to 5 hrs v 5 More than 5 hrs idecntry 036 Australia 040 Austria 056 Belgium F1 057 Belgium Fr 100 Bulgaria 124 Canada 170 Colombia 196 Cyprus 200 Czech Republic 208 Denmark 826 England 250 France 280 Germany 300 Greece 344 Hong Kong H 348 Hungary 352 Iceland 364 Iran Islamic Rep 372 Ireland 376 Israel 380 Italy 3 392 Japan 410 Korea 414 Kuwait 428 Latvia LSS 440 Lithuania 528 Netherlands i 554 New Zealand 578 Norway 608 Philippines s 620 Portugal 642 Romania 643 Russian Federation 827 Scotland 702 Singapore 201 Slovak Republic 890 Slovenia 717 South Africa 724 Spain 752 Sweden 756 Switzerland 764 Thailand 840 United States Now use the macro JACK to get the results include jack sps jack cvar identry idgrader bsbgdayl dvar bimatscr njkr 75 jkz jkzone jki jkindic wgt totwgt Sort cases by idcntry idgrader temporary select if idgrader 2 report format list automatic var 7 bsbgdayl label totwgt mnx mnx se pct pct se break idcntry idgrader 9 24 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES F
52. in the WORK FINAL data set These would be IDCNTRY IDGRADER and ITSEX There is one unique occurrence for each combination of the categories for these variables TIMSS DATABASE USER GUIDE 9 1 3 CHAPTER 9 PERFORMING ANALYSES Weight Variable Contains the estimate in the population that belongs to the group defined by the specific combination of the classification variable categories In our example this variable is called TOTWGT N Contains the number of cases in the group defined by the specific combination of categories for the classification variables MNX Contains the weighted mean of the variable DVAR for the group defined by the corresponding combination of classification variable categories MNX SE Contains the standard error of the mean for variable specified in DVAR computed using the JRR method for computing the standard error PCT Contains the weighted percent of people in the group for the classification variable listed last within the specific combination of the categories defined by the groups defined initially In our example we would obtain the percent of boys and girls within each combination of country and grade PCT SE Contains the standard error of PCT computed using the JRR method for computing the standard error The file resulting from using this macro can then be printed using a SAS procedure of choice An example call to this macro and a subset of the resulting file is presented in Figure 9 6 In
53. mathematics and science MATWGT Adjusted weight for mathematics teacher data SCIWGT Adjusted weight for science teacher data TCHWGT Adjusted weight for total teachers mathematics and science The MATWGT SCIWGT and TCHWGT variables contain mathematics science and total teacher weights adjusted for the total number of teachers of each type for each student For example if a student has three mathematics teachers the adjusted weight for each mathematics teacher MATWGT will be equal to one third so that each mathematics teacher contributes equally to the teacher based data for students See Mullis and Smith 1996 for more detailed information about the coding reliability procedures and results Z 28 TIMSS DATABASE USER GUIDE DATABASE FILES CRAP IE RS X Other Variables in the Student Teacher Linkage Files The linkage files also contain the identification variables required to identify cases and link the student and teacher files In addition some tracking and achievement score variables are also included The codebooks for Student Teacher Linkage files contain a complete list of all variables included 7 2 5 Missing Codes in the International Data Files All values assigned to variables in the TIMSS international data files are numeric and a subset of the numeric values for each of the variable types is reserved for specific codes related to different categories of missing data The missing categories defined below are ass
54. of Population 2 Items into Mathematics Content Area Reporting Categorie Setel ette ente de eie Dit Pe dee rie olds 7 2 Classification of Population 2 Items into Science Content Area Reporting Categories 7 22 Recodes Made to Free Response ltem Codes in the Written Assessment and Performance Assessment Items Gio od RR ARTHRITIS 7 26 Population 1 and Population 2 Codebook Files sss 7 33 File Structure of Machine Readable Codebook Files ses 7 34 Example Printout of a Codebook Page ccssssssessseessseseseeescseseseseseenssaeaeeensseeseeeneaeaeaeans 7 36 Population 1 and Population 2 Program Files sse 7 38 Data Almanac Riles suco teet ette tempted ee eat 7 39 Example Data Almanac Display for Categorical Variable sss 7 40 Example Data Almanac Display for Continuous Variable sss 7 41 Sample Table for Student Level Analysis Taken From the TIMSS International Report Mathematics Achievement in the Middle School Years sss 9 2 Sample Table for Teacher Level Analysis Taken From the TIMSS International Report Mathematics Achievement in the Middle School Years sss 9 3 Three letter Extension Used to Identify the Files Contained in the CD uu ccs 9 4 Extract from SAS Control Code for Creating a Student Background SAS Data Set 9 6 Extract from SPSS Control Code for Creating a St
55. once to obtain the statistic for the full sample and up to 75 times to obtain the statistics for each of the jackknife replicates J The number of times a statistic needs to be computed for a given country will depend on the number of implicit strata or sampling zones defined for the sample Note that when using the JRR technique for the estimation of sampling variability the approach will appropriately reflect the combined effect of the between and within sampling zone contributions to the variance Doubling and zeroing the weights of the selected units within the strata or zones is accomplished effectively with the creation of replicate weights which are then used in the calculations The next chapter shows how this approach allows standard statistical software such as SAS or SPSS to be used to compute JRR estimates of sampling variability The replicate weight approach requires the user to temporarily create a new set of weights for each pseudo replicate sample Each replicate weight is equal to k times the overall sampling weight where k can take values of zero one or two depending on whether or not the case is to be removed from the computation left as it is or have its weight doubled The value of k for an individual student record for a given replicate depends on the assignment of the record to the specific PSU and zone 8 2 Construction of Sampling Zones for Sampling Variance Estimation An important step in applying the JRR te
56. probability of selection for a student for each of these samples was different The performance assessment sample was selected as a sub sample of the written assessment sample and therefore the values for the different weighting factors and their adjustments are computed separately for the students within each file The meaning and interpretation of the weights in each of the files remains the same The weighting factors and their adjustments factors included in the student level data files are as follows WGTFACI School Weighting Factor This variable corresponds to the inverse of the probability of selection for the school where the student is enrolled WGTADJ1 School Weighting Adjustment This is an adjustment that is applied to WGTFAC1 to account for non participating schools in the sample If we were to multiply WGTFACI by WGTADJ1 we would obtain the sampling weight for the school adjusted for non participation WGTFAC2 Class Weighting Factor This is the inverse of the probability of selection of the classroom within the school In most cases the value of this variable is an integer but it could take other values when more than one classroom is selected in the school Although most countries selected classrooms within schools this was not always the case When a country selected students within the school without first selecting a specific classroom or when there was only one classroom at the target grade the value of this variable is s
57. provide contextual and explanatory information TIMSS expanded beyond the already substantial task of measuring achievement in two subject areas by also including a thorough investigation of curriculum and how it is delivered in classrooms around the world In addition extending the work of previous IEA studies TIMSS included a performance assessment Continuing the approach of previous IEA studies TIMSS addressed three conceptual levels of curriculum The intended curriculum is composed of the mathematics and science instructional and learning goals as defined at the system level The implemented curriculum is the mathematics and science curriculum as interpreted by teachers and made available to students The attained curriculum is the mathematics and science content that students have learned and their attitudes towards these subjects To aid in interpretation and comparison of results TIMSS also collected extensive information about the social and cultural contexts for learning many of which are related to variation among educational systems Nearly 50 countries participated in one or more of the various components of the TIMSS data collection effort including the curriculum analysis To gather information about the intended curriculum mathematics and science specialists within each participating country worked section by section through curriculum guides textbooks and other curricular materials to categorize aspects of these materials in ac
58. reports A list of all these deleted items as well as any items omitted in the test instruments for specific countries is given in Table 4 2 TIMSS DATABASE USER GUIDE 4 7 CHAPTER 4 DATA COLLECTION PROCEDURES Table 4 2 List of Deleted Cognitive Items ltem Subject Variable Name Population 2 Written Assessment All Mathematics BSMMM09 Austria Mathematics BSMMM05 Belgium FR Science BSMSNO5 Colombia Science BSMSO12 Cyprus Science BSMSF03 Science BSMSH01 Science BSMSJ01 Science BSMSJ07 Science BSMSO 11 Science BSMSQ15 Denmark Mathematics BSMMN16 France Science BSMSNO05 Science BSMSQ16 Greece Mathematics BSSMVO1 Hungary Mathematics BSMML17 Mathematics BSMMQO09 Science BSESZO1 Science BSESZO1 Iceland Mathematics BSMMO04 Science BSSSQ17 Iran Science BSMSF04 Science BSMSNO09 Israel Science BSMSB05 Japan Mathematics BSMMuJ11 Mathematics BSMMN17 Mathematics BSEMUO2 Mathematics BSEMUO2 Kuwait Mathematics BSMMD10 Mathematics BSMMF07 Mathematics BSMMIO1 Science BSMSJ05 Mathematics BSMMJ18 Mathematics BSMMO04 Latvia Science BSMSE12 Lithuania Mathematics BSMMF1 1 Philippines Mathematics BSMMI03 Science BSMSJ02 Romania Mathematics BSMMA03 Russian Federation Science BSMSI19 Singapore Mathematics BSEMU02 Mathematics BSEMU02 Slovak Republic Science BSSSK10 Thailand Mathematics BSMMk04 Science BSMSP04 Science BSMSQ16 United States Science BSMSI13 4 8 TIMSS DATABASE USER GUIDE DATA COLLECTION P
59. school files together data merged merge student school by idcntry idschool if nmiss bcbgcomm 0 proc format library work value grade 1 Seventh Grade 2 Eighth Grade B value comm 1 3 Community Type2 Community Type 1 2 4 Community Type 4 Community Type 3 value country 036 Australia 040 Austria 056 Belgium F1 057 Belgium Fr 100 Bulgaria 124 Canada T 170 Colombia 196 Cyprus 200 Czech Republic h 208 Denmark 826 England 250 France F 280 Germany 300 Greece 344 Hong Kong x 348 Hungary 352 Iceland 364 Iran Islamic Rep 372 Ireland 376 Israel 380 Italy b 392 Japan 410 Korea 414 Kuwait i 428 Latvia LSS 440 Lithuania 528 Netherlands 554 New Zealand 578 Norway 608 Philippines 620 Portugal 642 Romania 643 Russian Federation 827 Scotland 702 Singapore 201 Slovak Republic M 890 Slovenia 717 South Africa 724 Spain H 752 Sweden 756 Switzerland 764 Thailand d 840 United States Now use the macro JACK to get the results include jack sas jack totwgt jkzone jkindic 75 idcntry idgrader bcbgcomm bimatscr merged proc print data final noobs where idgrader 2 by idcntry idgrader var bcbgcomm N totwgt mnx mnx se pct pct se format idcntry countryl idgrader grade bcbgcomm comm TIMSS DATABASE USER GUIDE
60. sse 9 33 9 7 Scoring the llems eere ee e re pepe n erp d 9 39 TIMSS DATABASE USER GUIDE iii CONTENTS Tables and Figures Table 1 1 Figure 2 1 Table 2 1 Table 2 2 Table 2 3 Table 2 4 Table 2 5 Table 2 6 Table 2 7 Table 2 8 Table 3 1 Table 3 2 Figure 3 1 Figure 3 2 Figure 3 3 Figure 3 4 Table 3 3 Table 3 4 Figure 4 1 Figure 4 2 Table 4 1 Table 4 2 Table 6 1 Table 6 2 Table 6 3 Table 6 4 Table 7 1 Table 7 2 Table 7 3 Table 7 4 Table 7 5 Table 7 6 Countries Participating in TIMSS at Population 1 and 2 Data Included in Database 1 2 The Major Categories of the TIMSS Curriculum Frameworks sss 22 Mathematics and Science Content Area Reporting Categories sse 2 3 Distribution of Item Types Across Clusters Population 1 sse 2 5 Ordering of Item Clusters Within Population 1 Booklets sse 2 6 Distribution of Item Types Across Clusters Population 2 sss 2 7 Ordering of Clusters Within Population 2 Booklets sse 2 8 Assignment of Performance Assessment Tasks to Stations 2 9 Assignment of Students to Stations in the Performance Assessment s sess 2 9 Countries Administering the Specialized and Non Specialized Versions of the Population 2 Student Questionnaire sss 2 11 Grades Tested in TIMSS Population 1 ssssssseerrres 32 Grades Tested in TIMSS Population 2
61. that captures whether the case is to be dropped or have its weight doubled for the corresponding replicate weight The name of this variable in all TIMSS files is JKINDIC NJKR This indicates the number of replicate weights to be generated when computing the JRR error estimates When conducting analyses using the data from all countries the value of NJKR should be set to 75 for the student school and teacher background data and 42 for the performance assessment data The user working with the data for only one country should set the NJKR argument to as many replicates there were in the country see Table 9 2 for the maximum number of replicates by country If the data from two or more countries is being used for an analysis then the larger number of jackknife zones should be used When in doubt about what number to set the NJKR parameter it should be set to 75 The error variance will always be estimated correctly if more replicate weights than necessary are computed but will be underestimated if the user specifies less replicate weights than necessary CVAR This lists the variables that are to be used to classify the students in the data file This can be a single variable or a list of variables The maximum number of variables will depend mostly on the computer resources available to the user at the time It is recommended to always include the variable that identifies the country At least one variable has to be specified usually IDCN
62. that participated in TIMSS at Population 1 and Population 2 and for which internationally comparable data are available At Population 1 there are nine different data file types and at Population 2 there are ten different file types reflecting achievement data collected from students both for the written assessment and performance assessment background data collected from students teachers and schools student teacher linkage information and coding reliability data For each file type a separate data file is provided for each participating country that collected the information Table 7 1 shows the Population 1 and Population 2 data files provided Table 7 1 TIMSS Population 1 and Population 2 Data Files File Type Population 2 File Name Population 1 File Name Student Written Assessment File BSA lt Country gt 1 DAT ASA lt Country gt 1 DAT Student Background File BSG lt Country gt 1 DAT ASG lt Country gt 1 DAT Teacher Background File s BTM lt Country gt 1 DAT Mathematics ATG lt Country gt 1 DAT BTS lt Country gt 1 DAT Science School Background File BCG lt Country gt 1 DAT ACG lt Country gt 1 DAT Student Teacher Linkage File BLG lt Country gt 1 DAT ALG lt Country gt 1 DAT Student Written Assessment Reliability File BSR lt Country gt 1 DAT ASR lt Country gt 1 DAT Student Performance Assessment File BSP lt Country gt 1 DAT ASP lt Country gt 1 DAT Student Performance Assessment Reliability File BSQ lt Country gt 1 DAT ASQ lt Country
63. this example the macro computes the percent of boys and girls by grade and by country and their mean achievement in mathematics The listing presented in Figure 9 6 is interpreted in the following way The first line shows that there were 3039 students in the sample for IDCNTRY 36 Australia in IDGRADER 1 Seventh grade and who had ITSEX 1 Girls It is estimated that there are 123649 seventh grade girls in Australia their mean mathematics score is 500 with a standard error of 4 3 We can also tell from this line of data that it is estimated that 52 percent of the seventh graders in Australia are girls and the standard error of this percent is 2 3 The second line shows the same information but for the seventh grade boys ITSEX 2 It is estimated that there are 114646 seventh grade boys in Australia their mean mathematics score is 495 with a standard error of 5 2 We can also tell from this line of data that it is estimated that 48 percent of the seventh graders in Australia are boys and the standard error of this percent is 2 3 9 14 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 6 SAS Control Code and Extract of Output File for Using the Macro JACK SAS options nocenter Read the variables from the student file data student set bsgalll keep idcntry idgrader jkindic jkzone totwgt itsex bimatscr where ITSEX gt 0 include jack sas jack totwgt jkzone jkindic 75 idcntry idgrader itsex bimatscr stu
64. this is that any nationally defined diagnostic codes employed in individual countries 7 or 8 were recoded to the other category 9 within the same correctness level prior to the computation of the Code Agreement Variables TIMSS DATABASE USER GUIDE E225 CHAPTER 7 DATABASE FILES Table 7 11 Recodes Made to Free Response ltem Codes in the Written Assessment and Performance Assessment Items Variable Recodes Comment All Items Country specific diagnostic codes recoded to other categories within the score level y Viv ve BSMMK08 71 Training team found it difficult to distinguish between the 70 and 71 codes both codes combined in 70 Only 20s have positive point biserial correlation change to 1 point item codes BSESL04 20 21 29 10 11 12 19 BSESM11 10 11 12 13 20 21 22 23 24 25 30 31 BSESY01 20 21 22 29 10 11 19 BSESY02 21 BSSSJ03 19 BSSSM12 19 BSES014 20 29 10 11 19 BSSSQ18 19 29 BSSML16 19 BSSMM06 19 BSSMM08 19 BSSMQ10 19 BSSMR13 74 Only 30s have positive point biserial correlation change to 1 point item codes Only 20s have positive point biserial correlation change to 1 point item codes Typographical error in category 21 in coding guide Typographical error in coding guide Typographical error in coding guide Only 20s have positive point biserial correlation Typographical error in coding guide Typographical error in coding guide Typogra
65. 0 T L S vvt9 08 T 0 0L ezodebuts 0 6 OL 0 9 0 v ore o z o T v LZ9Z 08 IZI ZETE pue 3oog 0 TI 0 8 0 L 0 v o z oT 0 0 vp 8LEZ 6 6S 0592 Tebnaaod 0 6 0 9 o s o e O T O T 0 T ZE 6502 9 z 61zc Aemzon 0 6 0 8 0 9 ore o z o T 0 1 6 E T6IZ 6S S vosc pue eoz MeN 0 ZI 0 9 0 0 0 0 0 0 0 0 0 6 T 812 o SZZ 06LZ spue 3eui3eN 0 0T 0 6 0 9 0 0 z OT 0 T v L681 SG 8 vSO0Z SST eTazeT t t g g 969Z 8L LLLZ eoioy g i f s i s i i 0 0 90 v 90 F ueder 0 OT 0 8 0 9 0 v 0 z O T OT s 669Z vv GE 6882 pue e4I 0 ZI 0 6 0 9 ore oT 0 1 0 1 6 z09Z 00 v I9 doy orue sI ULTI 0 0I 0 8 o L o p O T O T 0 0 I v 6SvI Iv 8v 869T puereor O TT 0 6 S 9 0 S I 0 T OST EP OPLZ ZST vS 8 0 Azebuny OET 0 6 0 8 0 v 0 z o t 0 0 8 v 8v vL cv 96 v buoy buoH O LTE 0 8 0 9 0 v ore O T 0 T s y 08SZ SIT SG6Z 90998195 0 6 0 8 0 9 0 v 0 T O T O T 0 LZLZ 0L TL 9S0 pueTbug 0 6 0 8 o s 0 v o z 0 1 0 1 Iv OSTE 9E vz 9GZE oTTqndey uoezo O EL 0 6 0 9 0 v o z 0 I 0 I s 808Z OPT cc 80 snaddp 0 6 0 8 0 9 0 v 0 z O T O T Y 6LS9 ZET IEZ v6SL epeuej 0 0I 0 8 0 4 o s ore O T O T o s 8812 9v TI 97SZ eTazsny 0 6 0 8 0 9 0 v 0 z o i 0 0 vp 802v 6S 921 IvLY erriezisnvy Xen 06d 0 uerpew I Old UTW ueoW Iddy TWO upv Soseo Azyunog ON 30N epezb 41eMOT xNHOLVOIGNI HOVUD5 N S I S uorqeooT zNugd5gHSV cz ad3unoo o3 sued no ueuA no ede PTO MOH uora3sen eseqeqeq euoraeugieq
66. 1 and 2 was prepared by the IEA Data Processing Center for the NRCs and their colleagues who were responsible for implementing the TIMSS data collection and processing procedures It describes the activities and responsibilities of the NRCs from the moment the international testing materials arrived at the national center to the moment the cleaned data sets and accompanying documentation were sent to the IEA Data Processing Center In addition to detailed within school sampling instructions the manual included Procedures for translating and assembling the test instruments and questionnaires Instructions for obtaining cooperation from the selected schools e Explicit procedures for packing and sending materials to the schools e Preparations for test administration e Instructions for data entry and verification Included in this manual were a set of Survey Tracking Forms that were completed at various stages of the study to track schools classrooms and students and ensure proper linkage among them in the database Each school was asked to appoint a coordinator to be the liaison between the national research center and the school The School Coordinator Manual describes the steps the School Coordinator followed once the testing materials arrived at the school until they were returned to the NRC Essentially the School Coordinator was responsible for receiving the shipment of testing materials from the national center and providing for their se
67. 13 90 12 07 4 62 54 Belgium Fr Eighth Grade Less than 1 Hr 19212 63 536 03 4 20 33 34 1 26 1 to 2 hrs 25151 10 535 67 4 88 43 65 1 77 3 to 5 hrs 9549 30 522 29 3 98 16 57 1 32 More than 5 hrs 3705 76 445 04 9 01 6 43 98 United States Eighth Grade Less than 1 Hr 687980 70 503 69 5 73 22 20 19 1 to 2 hrs 1239797 59 512 94 5 10 39 98 91 3 to 5 hrs 767004 30 501 23 4 22 24 74 59 More than 5 hrs 405491 75 460 77 4 58 13 08 95 Slovenia Eighth Grade Less than 1 Hr 5838 91 546 07 4 11 22 64 1 06 1 to 2 hrs 13852 82 540 94 3 37 53 72 1 09 3 to 5 hrs 4992 78 540 09 4 66 19 36 87 More than 5 hrs 1101 55 517 72 9 93 4 27 42 9 26 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CUR AGP OLE R 09 9 5 Performing Analyses with Teacher Level Variables When analyzing the teacher data it is first necessary to link the students with their corresponding teachers Each student record in the Student Background data file can contain a link to as many as six different teachers in the Teacher Background data file To facilitate the linking between students and their teachers in the teacher file the Student Teacher Linkage file was created and is part of the International Database This file is called ALG lt COUNTRY gt 1 in Population 1 and BLG COUNTRY 1 in Population 2 The Student Teacher Linkage file contains one record for each student by teacher combination with the corresponding identification variables Each record also contains th
68. 2 DatajFiles zie eee coats Ate tA e PDA Background File srenti ieie HERE RE RR OI RH UBRO RE E NIU deg 72 131 SiudentB ckground Filez dette et PER LERRA LIRE RUE DA F232 Teacher Backgrouna tilesu o pee e aide ba peter s 7 2 1 3 School Background File 7 2 1 4 Identification Variables 7 2 1 5 Achievement Scores ET 7 2 1 6 linking and Tracking V More 7 2 1 7 International Background Variables icu AE 7 2 1 8 Variables Derived from Student Teacher and School laadid Data FDP Sampling Matldbless a meei Bh voles e e e a a A N a tus 7 2 2 Ass ssmietil Fes ie RR RR RE RU ERR EA 7 2 2 1 Written Assessment Files 7 2 2 2 Performance Assessment Files 7 2 2 3 Cognitive tem Veiriable Names ctc tette bet bte tetti bet etait n 7 2 2 4 Performance Assessment Task Identifications 7 2 2 5 Cognitive Item Response Code Values n 7 2 2 6 Analysis By Mathematics and Science Content Area Ed PN 7 2 27 Release Status of TIMSS Test Items and Performance Tasks 7 2 2 8 Other Variables in the Student Assessment Files tts 17 7 2 2 9 School Performance Assessment Files i ttttttttetant T 2 3 Coding Reliabiliy Flles ha eto t I DAR RH RE E dtt eset 7 2 4 Student Teacher Linkage Files tct 7 2 5 Missing Codes in the International Data Files tte 7 2 6 National Data Issues Affecting the Usage of International Data Files TIMSS DATABASE USER GUIDE CONTENTS 7 9
69. 23 584 657 7 7854 24 6642 4 42650 Community Type 4 996 28525 47 553 321 13 9251 41 7438 6 06839 COUNTRY ID Belgium Fr GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 2 518 10906 33 515 153 18 8832 23 3082 5 02550 Community Type 3 357 10109 43 518 489 10 3821 21 6051 4 90721 Community Type 4 1252 25776 11 539 305 5 9846 55 0867 5 44131 COUNTRY ID United States GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 1 132 90271 12 466 310 21 1273 3 2568 1 65686 Community Type 2 961 587437 27 509 643 6 0996 21 1936 3 38785 Community Type 3 1784 898597 98 513 360 6 4977 32 4196 4 10878 Community Type 4 3116 1195464 02 498 183 7 2497 43 1300 4 40161 COUNTRY ID Slovenia GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 1 163 1630 53 537 019 18 1296 7 4994 2 34076 Community Type 2 520 5435 88 521 496 6 8803 25 0016 2 92850 Community Type 3 492 4860 04 559 011 6 1974 22 3531 4 22613 Community Type 4 1092 9815 66 541 874 4 3193 45 1459 3 81019 TIMSS DATABASE USER GUIDE 9 37 CHAPTER 9 Figure 9 20 PERFORMING ANALYSES Extract of SPSS Computer Output for Performing Analyses with School Level Variables EXAMPLE 3 GENNTYPE OF COUNTRY ID COMMUNITY TOTWGT MNX MNX SE PCT PCT SE Australia Eighth Grade Community Type 2 33600 29 512 91 11 33 17 16 3 71 Community Type 3 98242 75 531 36 7 25 50 16 4 36 Community Type 4 64010 75 528 81 7 74 32 68 4 41 Austria
70. 32468 2 32468 2 13471 2 13471 63905 63905 79216 79216 76051 76051 70983 70983 C C N29 9 HEB ES EB ES 1 20563 1 20563 0 77280 0 77280 0 76075 0 76075 0 90855 0 90855 already presented one example in the previous section when explaining how to use the two macros provided with the data files We now proceed to work out another example where all the steps are undertaken including the invocation of the corresponding SAS and SPSS macro For example suppose we want to replicate one of the results presented in the international report We are interested in looking at the eighth graders reports on the hours spent each day watching television and videos and their achievement in mathematics These are the results that are presented in Figure 9 1 earlier in this chapter TIMSS DATABASE USER GUIDE 9 21 CHAPTER 9 PERFORMING ANALYSES To do this we need to undertake several steps After reviewing the codebooks and the questionnaire information we find out if the students were asked a question about the number of hours they watch television see Supplement 2 The data collected from this variable are captured in the variable BSBGDAY TI and this variable is found in the Student Background data file Our next step is to review the documentation of national adaptations to the questionnaires to ensure that there were no deviations listed for this variable see Supplement 3 If no changes were made we can continue w
71. 4 Godeboolerlles ie eta e ORI Heg d inn dite tmd ten 7 33 7 3 1 Accessing the Codebook Files tto 7 33 7 32 USING theCodebooks etc REND E Ead 7 34 FA JProgramiFil s c oic ostisedeiehofedeteneere v etege tete et aitdoetettees 7 38 7 5 Data Almanacs e o coe eeu d ee eun 7 39 7 6 TestCurriculum Matching Analysis Data Files sss 7 42 Chapter 8 Estimating Sampling Variance esesessosesesesessoseseseseseososesesescssoseseseseseoses 8 1 8 1 Computing Error Variance Using the JRR Method sss 8 1 8 2 Construction of Sampling Zones for Sampling Variance Estimation see 82 8 3 Computing the JRR Replicate Weights sse 8 3 Chapter 9 Performing Analyses with the TIMSS Data Some Examples 9 1 9 L Contents of the CDs onte edet f dial bte c ote te taedet 9 4 9 2 Creating SAS Data Sets and SPSS System Files 9 5 9 3 Computing JRR Replicate Weights and Sampling Variance Using SPSS and SAS uu eee 9 8 9 3 1 SAS Macro for Computing Mean and Percents with Corresponding Standard Errors JACK SAS 9 8 9 3 2 SPSS Macro for Computing Mean and Percents with Corresponding Standard Errors JACK SPS 9 16 9 4 Performing Analyses with Student Level Variables sss 9 21 9 5 Performing Analyses with Teacher Level Variables sssssseeeee 9 27 9 6 Performing Analyses with School Level Variables
72. 6 to 10 years 435 4158 14 532 594 5 9679 19 0633 4 02549 11 to 20 years 1214 12025 34 541 671 5 4834 55 1310 4 97920 Over 20 years 542 4789 01 550 436 6 1702 21 9555 3 82312 TIMSS DATABASE USER GUIDE 9 3 CHAPTER 9 Figure 9 16 PERFORMING ANALYSES Extract of SPSS Computer Output for Performing Analyses with Teacher Level Variables EXAMPLE 2 GEN YEARS BEEN COUNTRY ID TEACHING MATWGT MNX MNX_SE PCT PCT_SE Australia Eighth Grade 0 to 5 years 35747 90 517 04 8 50 17 66 2 32 6 to 10 years 39265 42 528 25 11 59 19 40 2 56 11 to 20 years 71548 18 541 14 8 37 35 34 2 75 Over 20 years 55882 22 533 10 8 48 27 60 2 61 Austria Eighth Grade 0 to 5 years 4573 10 515 60 19 73 7 16 2 31 6 to 10 years 8429 89 545 78 9 49 13 20 2 48 11 to 20 years 32826 23 553 93 6 68 51 39 4 00 Over 20 years 18041 87 549 28 8 77 28 25 3 64 Belgium F1 Eighth Grade 0 to 5 years 6915 80 555 76 17 95 9 59 2 76 6 to 10 years 6351 73 589 84 14 52 8 81 2 22 11 to 20 years 23216 69 553 86 13 43 32 19 4 75 Over 20 years 35647 25 574 55 10 56 49 42 4 91 Belgium Fr Eighth Grade 0 to 5 years 3051 78 536 16 12 29 8 17 3 15 6 to 10 years 2844 35 528 20 13 80 7 62 2 31 11 to 20 years 11466 42 558 39 6 98 30 70 5 21 Over 20 years 19988 70 543 10 6 39 53 52 4 84 United States Eighth Grade 0 to 5 years 684804 90 483 55 6 34 24 66 3 43 6 to 10 years 383442 47 488 37 9 76 13 81 2 69 11 to 20 years 706610 59 500 84 7 30 25 44 3 25 Over 20 years 1002577 37
73. 7 Marginal estimation procedures In A E Beaton Ed Implementing the new design The NAEP 1983 84 technical report Report No 15 TR 20 Princeton NJ Educational Testing Service Mislevy R J and Sheehan K M 1989 The role of collateral information about examinees in item parameter estimation Psychometrika 54 4 661 679 TIMSS DATABASE USER GUIDE REFERENCES Mullis I V S and Smith T S 1996 Quality control steps for free response scoring In M O Martin and I V S Mullis Eds TIMSS Quality assurance in data collection Chestnut Hill MA Boston College Mullis I V S Jones C O and Garden R A 1996 Training sessions for free response scoring and administration of the performance assessment In M O Martin and I V S Mullis Eds TIMSS Quality assurance in data collection Chestnut Hill MA Boston College Mullis I V S Martin M O Beaton A E Gonzalez E J Kelly D L and Smith T A 1997 Mathematics achievement in the primary school years IEA s Third International Mathematics and Science Study Chestnut Hill MA Boston College Mullis I V S and Martin M O 1997 Item analysis and review In M O Martin and D L Kelly Eds TIMSS technical report volume II Implementation and analysis Chestnut Hill MA Boston College Robitaille D F Ed 1997 National contexts for mathematics and science education An encyclopedia of education systems participating in TIMSS Vancouver B C
74. 9 35 CHAPTER 9 PERFORMING ANALYSES Figure 9 18 SPSS Control Statements for Performing Analyses with School Level Variables EXAMPLE3 SPS get file bcgalll sys keep identry idschool bcbgcomm sort cases by idcntry idschool save outfile school get file bsgalll sys keep identry idschool idstud idgrader jkindic jkzone totwgt bimatscr select if idgrader 2 sort cases by idcntry idschool save outfile student Now merge the two files match files file student table school by identry idschool select if not missing bcbgcomm execute Define the format for the variables used value labels idgrader 1 Seventh Grade 2 Eighth Grade bcbgcomm 1 Community Type 1 2 Community Type 2 3 Community Type 3 4 Community Type 4 identry 036 Australia 040 Austria 056 Belgium Fl E 057 Belgium Fr 100 Bulgaria 124 Canada p 170 Colombia 196 Cyprus 200 Czech Republic us 208 Denmark 826 England 250 France 280 Germany 300 Greece 344 Hong Kong 348 Hungary 352 Iceland 364 Iran Islamic Rep 372 Ireland 376 Israel 380 Italy 392 Japan 410 Korea 414 Kuwait 428 Latvia LSS 440 Lithuania 528 Netherlands s 554 New Zealand 578 Norway 608 Philippines 620 Portugal 642 Romania 643 Russian Federation 827 Scotland 702 Singapore 201 Slov
75. CE IEA Third International Mathematics and Science Study TIMSS 1994 95 6 6 TIMSS DATABASE USER GUIDE ACHTEWVEM ENT SCO RES CHAPTER 6 Table 6 3 Descriptive Statistics for the International Mathematics Achievement Scores for Population 2 Variable BIMATSCR Lower Grade Upper Grade Standard Deviation Australia 497 90 529 63 Austria 509 17 539 43 Belgium Fl 557 62 565 18 Belgium Fr 507 14 526 26 Bulgaria 513 80 539 66 Canada 493 99 527 24 Colombia 368 51 384 76 Cyprus 445 67 473 59 Czech Republic 523 39 563 75 Denmark 464 85 502 29 England 476 15 505 73 France 492 19 537 83 Germany 484 38 509 16 Greece 439 89 483 90 Hong Kong 563 60 588 02 Hungary 501 80 537 26 Iceland 459 43 486 78 Iran Islamic Rep 400 95 428 33 Ireland 499 69 527 40 Israel 521 59 Japan 571 09 604 77 Korea 577 06 607 38 Kuwait 3 392 18 Latvia LSS 461 59 493 36 Lithuania 428 19 477 23 Netherlands 515 98 540 99 New Zealand 471 70 507 80 Norway 460 66 503 29 Portugal 423 15 454 45 Romania 454 42 481 55 Russian Federation 500 93 535 47 Scotland 462 89 498 46 Singapore 601 04 643 30 Slovak Republic 507 84 547 11 Slovenia 498 17 540 80 South Africa 347 51 354 14 Spain 447 97 487 35 Sweden 477 47 518 64 Switzerland 505 55 545 44 Thailand 494 77 522 37 United States 475 68 499 76 Minimum Maximum Mean Standard Minimum Maximum SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUID
76. Code Used to Convert Cognitive Item Response Codes to Correctness Score Levels esses 9 43 vi TIMSS DATABASE USER GUIDE Chapter 1 Overview of TIMSS and the Database User Guide 1 1 Overview of the International Database This User Guide accompanies the TIMSS International Database for the Primary and Middle School Years TIMSS Populations 1 and 2 The database provided on two compact disks contains achievement data written test and performance assessment and student teacher and school background data collected in 42 countries in 1995 Table 1 1 lists for each of Populations 1 and 2 the countries for which written assessment and performance assessment data are included in the International Database Each of these countries gave the IEA permission to release its national data The TIMSS International Database contains the following for each country for which internationally comparable data are available e Mathematics and science proficiency scale scores e Students responses to cognitive mathematics and science items e Students responses to hands on performance tasks e Students background questionnaire data e Mathematics and science teacher background questionnaire data e School background questionnaire data e Test curriculum matching analysis data Sampling weights e International codebooks e SPSS and SAS control statement files Data almanacs Given the size and complexity of TIMSS and the ps
77. E 6 7 CHAPTER 6 ACHIEVEMENT SCORES Table 6 4 Descriptive Statistics for the International Science Achievement Scores for Population 2 Variable BISCISCR Lower Grade Upper Grade Standard T i Mean dS Minimum Maximum Mean Standard Minimum Maximum Deviation Australia 504 37 544 60 Austria 518 76 557 68 Belgium FI 528 67 550 28 Belgium Fr 442 05 470 57 Bulgaria 530 85 564 83 Canada 499 19 530 89 Colombia 387 48 411 09 Cyprus 419 92 462 57 Czech Republic 532 93 573 94 Denmark 439 02 478 30 England 512 02 552 07 France 451 46 497 65 Germany 499 50 531 33 Greece 448 65 497 35 Hong Kong 495 26 522 12 Hungary 517 90 553 68 Iceland 461 96 493 55 Iran Islamic Rep 436 33 469 74 Ireland 495 21 537 78 Israel z 524 48 Japan 531 04 570 98 Korea 535 02 564 87 Kuwait 429 57 Latvia LSS 434 91 484 76 Lithuania 403 06 476 45 Netherlands 517 25 560 07 New Zealand 480 96 525 45 Norway 483 19 i 527 19 Portugal 427 93 479 63 Romania 451 58 486 05 Russian Federation 483 96 538 08 Scotland 468 14 517 18 Singapore 544 70 607 33 Slovak Republic 509 67 544 41 Slovenia 529 93 560 05 South Africa 317 14 325 90 Spain 477 15 517 05 Sweden 488 43 535 40 Switzerland 483 66 521 74 Thailand 492 90 525 43 United States 508 24 534 45 SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 6 8 TIMSS DATABASE USER GUIDE ACHIEVEMENT SCORES CHAPTER 6 6 2 Achievement Scores in the School Background Files
78. For categorical variables all possible response options are listed Any missing codes described in Section 7 2 5 are also included for either numerical or categorical variables For example for multiple choice cognitive items the code options are a b c d e while for the free response cognitive items the code options are the two digit numerical codes described in Chapter 4 Option The sixth column Option includes a textual description of each type of response option For variables containing numeric data it contains an explanation of the values contained in the variable Location Format The seventh column Location Format presents the location and format of each variable in the raw data files The format is the pattern used to write each value of a numeric or categorical variable with a general structure of XX XX N C X The numbers preceding the slash indicate the location of the variable and refer to its position in the raw data file starting ending column positions The N or C after the slash identifies the variable as numerical or categorical The numeric code after the slash indicates the length of the values and the number of decimal places associated with each variable e g 2 0 2 digits 0 decimal places 6 2 six digits two decimal places TIMSS DATABASE USER GUIDE y 37 CHAPTER 7 DATABASE FILES 7 4 Program Files Three different types of program files are provided for use in analyses of the TIMSS data files
79. International Association for the Evaluation of Educational Achievement User Guide for the TIMSS International Database Primary and Middle School Years Population 1 and Population 2 Data Collected in 1995 Edited by Eugenio J Gonzalez Teresa A Smith with contributions by Heiko Jungclaus Dirk Hastedt Dana L Kelly Ina V S Mullis Michael O Martin Knut Schwippert Jens Brockmann Ray Adams Pierre Foy Ce Shen September 1997 TIMSS International Study Center Boston College Chestnut Hill MA USA 1997 International Association for the Evaluation of Educational Achievement IEA User Guide for the TIMSS International Database Primary and Middle School Years 1995 Assessment Edited by Eugenio J Gonzalez and Teresa A Smith To obtain additional copies of the TIMSS International Database and User Guide contact the IEA International Association for the Evaluation of Educational Achievement The IEA Secretariat Herengracht 487 1017 BT Amsterdam The Netherlands Tel 31 20 625 36 25 Fax 31 20 420 71 36 email Department IEA nl For more information about TIMSS contact the TIMSS International Study Center TIMSS International Study Center Campion Hall 323 CSTEEP School of Education Boston College Chestnut Hill MA 02167 United States email timss bc edu http wwwcsteep bc edu timss Funding for the international coordination of TIMSS is provided by the
80. JACK SPS This macro can be used to compute weighted percents and means within categories Although the user can compute weighted percent and mean estimates using other basic SPSS commands the macro JACK SPS also computes the JRR error estimate for these means and percents The control code for the macro JACK SPS is presented in Figure 9 7 Figure 9 7 SPSS Macro for Computing Mean and Percents with Corresponding JRR Standard Errors JACK SPS set mprint on define jack cvar charend dvar charend njkr charend jkz Icharend jki Icharend wgt charend weight off sort cases by cvar compute k 1 compute wgtx wgt dvar vector rwgt njkr vector rwgtx njkr loop i 1 to njkr if jkz Zi and jki 0 rwgt i wgt 0 if jkz Zi and jki 1 rwgt i wgt 2 if jkz lt gt i rwgt f i wgt 1 compute rwgtx i rwgt i dvar end loop llet nway tmpjckl aggregate outfile nway presorted break cvar k wgt rwgtl to concat rwgt njkr wgtx rwgtxl to concat rwgtx njkr sum wgt rwgtl to concat rwgt njkr wgtx rwgtxl to concat rwgtx njkr n n wgt llet cvarl null tdo i lin cvar llet cvarl concat i blank 1 cvarl doend llet cvar2a concat cvarl blank 1 k llet cvar2b concat tail cvarl blank 1 k tdo i lin cvar2a lif cvar2b null then break ifend get
81. MCO3 BSMMCO6 BSMSC11 BSMSD06 BSMMD09 BSMMD10 BSMSE10 BSMSF02 BSMSF03 BSMSF05 BSMMF11 BSMSG07 BSMSHO1 BSMSHO3 BSMSHO4 BSMSHO6 BSMMIO7 BSMMI09 BSMSI13 BSMSI15 BSMSI16 BSMSI19 BSMSJO02 BSMMJ15 BSMMJ16 BSMMK03 BSMSK11 BSMSK16 BSMSK18 BSMML15 BSMMMO02 BSMMM03 BSMMM04 BSMSM13 BSMSN06 BSMMN16 BSMMN17 BSMSO12 BSMMQ05 BSMSQ11 BSMSQ15 BSMSRO1 BSMMRO6 BSMMR11 LET SHORTA BSSMI04 BSSMI06 BSSSI18 BSSSJ03 BSSSJ09 BSSMJ12 BSSMJ13 BSSMK02 BSSMK05 BSSSK10 BSSSK19 BSSML16 BSSMM06 BSSMM08 BSSSM12 BSSSM14 BSSSNO07 BSSSN10 BSSMN13 BSSMN19 BSSMO06 BSSMO09 BSSSO10 BSSSO16 BSSSO17 BSSSP02 BSSSP03 BSSSP05 BSSSP06 BSSMP16 BSSMQ10 BSSSQ12 BSSSQ17 BSSSQ18 BSSSRO4 BSSSRO5 BSSMR13 BSSMR14 BSSMV01 BSSMV04 ARRAY ARIGHT amp ARIGHT ARRAY EXTEND amp SHORTA DO OVER ARIGHT SCOREIT ARIGHT MC 1 6 8 9 7 END DO OVER EXTEND SCOREIT SHORTA SA 96 98 99 90 END 9 42 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 22 Extracted Sections of SPSS Control Code Used to Convert Cognitive Item Response Codes to Correctness Score Levels SET MPRINT ON DEFINE SCOREIT Type charend Item charend RIGHT charend nr charend na charend om charend other charend I UPCASE Type MC Then Do I In Item Recode I RIGHT 1 nr 0 na sysmis 10m 0 other 0 Else 0 DoEnd IfEnd If UPCASE Type SA
82. MSS tests The Free Response Item Coding Committee developed the coding rubrics for the free response items The Performance Assessment Committee worked with the Performance Assessment Coordinator to develop the TIMSS performance assessment The Quality Assurance Committee helped to develop the quality assurance program International Steering Committee Tjeerd Plomp Chair the Netherlands Lars Ingelstam Sweden Daniel Levine United States Senta Raizen United States David Robitaille Canada Toshio Sawada Japan Benny Suprapto Brotosiswojo Indonesia William Schmidt United States Technical Advisory Committee Raymond Adams Australia Pierre Foy Canada Andreas Schleicher Germany William Schmidt United States Trevor Williams United States Sampling Referee Keith Rust United States Subject Area Coordinators Robert Garden New Zealand Mathematics Graham Orpwood Canada Science Special Mathematics Consultant Chancey Jones TIMSS DATABASE USER GUIDE Subject Matter Advisory Committee Svein Lie Chair Norway Antoine Bodin France Peter Fensham Australia Robert Garden New Zealand Geoffrey Howson England Curtis McKnight United States Graham Orpwood Canada Senta Raizen United States David Robitaille Canada Pinchas Tamir Israel Alan Taylor Canada Ken Travers United States Theo Wubbels the Netherlands Free Response ltem Coding Committee Svein Lie Chair Norway Vladimir Burjan Slov
83. ORE BSASCORE e Performed Assessment Files ASPSCORE BSPSCORE When using this code the user must first consider the recoding scheme that is desired For example under certain circumstances the user might want to recode the not reached responses as incorrect codes 6 and 96 whereas under other circumstances the user might want to recode these responses as not administered or invalid In the case of TIMSS not reached responses were recoded as not administered and effectively as missing responses for the purpose of calibrating the items when setting the international scale But the not reached responses were then recoded as incorrect when scoring the item for the individual countries and for the purpose of calculating the scale scores for the individuals By default the scoring program provided with the database recodes the items coded as not reached and those left blank coded as incorrect responses To use these macros the user needs to include them as part of the SAS or SPSS programs used for the analysis This is done by using the INCLUDE statement in the corresponding program In the case of SAS the scoring program code should be included as part of a DATA step that reads the items that are to be recoded When using SPSS the scoring program code should be included after the system file containing the item responses has been read into memory and becomes the working file Both of these programs recode the items onto themselves so if the use
84. ROCEDURES CHAPTER 4 Table 4 2 Continued ltem Subject Variable Name Population 1 Written Assessment Cyprus Science ASMSB04 England Mathematics ASEMT01 Hong Kong Mathematics ASMMJ05 Hungary Science ASMSD01 Mathematics ASEMT04 Iceland Science ASMSB03 Iran Mathematics ASMMK04 Japan Mathematics ASMMGO03 Mathematics ASMMJ03 Mathematics ASMMLO3 Mathematics ASMMLO6 Science ASMSP06 Korea Mathematics ASMMAO01 Science ASMSR08 Latvia Mathematics ASMMGO01 Mathematics ASMMGO02 Thailand Science ASMSB04 Mathematics ASSMUO4 Science ASESX01 Populations 1 and 2 Performance Assessment Colombia Task M4 Mathematics BSPM46 Item 6 Pop2 Cyprus Task M2 Mathematics ASPM25 Item 5 Pop1 Cyprus Task S6 Science ASPS61A Item 1A Pop1 TIMSS DATABASE USER GUIDE 4 9 CHAPTER 4 DATA COLLECTION PROCEDURES 4 5 2 Background Questionnaire Items As was the case with the test instruments the international versions of the student teacher and school background questionnaires were also developed in English and then translated into other languages by the TIMSS countries While the intent of TIMSS is to provide internationally comparable data for all variables there are many contextual differences among countries so that the international version of the questions are not always appropriate in all countries Therefore the international versions of the questionnaires were designed to provide an opportunity for individual countries to modify some questions or respon
85. S macro language The user with some programming experience in either one of these statistical packages will be able to make the necessary modifications to the macros to obtain the desired results When using these macros the code assumes that the user has already created a system file in SPSS or a data set if using SAS that contains the variables necessary for the analysis As part of this chapter we also describe the control files included in the CD that can be used to create SAS and SPSS data set system files Documentation regarding the computational methods used to obtain any derived variables included in the international reports is presented in Supplement 4 TIMSS DATABASE USER GUIDE 9 CHAPTER 9 PERFORMING ANALYSES Figure 9 1 Sample Table for Student Level Analysis Taken From the TIMSS International Report Mathematics Achievement in the Middle School Years Ho PT EX Students Reports on the Hours Spent Each Day Watching Television and Videos Mathematics Upper Grade Eighth Grade Less than 1 Hour 1 to 2 Hours 3 to 5 Hours More than 5 Hours Country Australia Austria Belgium Fl Belgium Fr Canada Colombia Cyprus Czech Republic Denmark England France Germany Greece Hong Kong Hungary Iceland Iran Islamic Rep Ireland Israel Japan Korea Kuwait Latvia LSS Lithuania Netherlands New Zealand Norway Portugal Romania Russian Federation Scotland Singapore Slovak Republic Slovenia Spain
86. Score Points in Mathematics Population 1 Number of Raw Score Points in Science TIMSS DATABASE USER GUIDE CHAPTER 7 DATABASE FILES BSMSCPT Population 2 Number of Raw Score Points in Mathematics BSSSCPT Population 2 Number of Raw Score Points in Science School Level Achievement Score Variables In addition to these student level scores the following variables containing the mean of the overall mathematics and science scale scores for the lower and upper grades of each school are provided in the School Background files ACLGMAT Population 1 Mean Overall Mathematics Achievement for Lower Grade ACUGMAT Population 1 Mean Overall Mathematics Achievement for Upper Grade ACLGSCI Population 1 Mean Overall Science Achievement for Lower Grade ACUGSCI Population 1 Mean Overall Science Achievement for Upper Grade BCLGMAT Population 2 Mean Overall Mathematics Achievement for Lower Grade BCUGMAT Population 2 Mean Overall Mathematics Achievement for Upper Grade BCLGSCI Population 2 Mean Overall Science Achievement for Lower Grade BCUGSCI Population 2 Mean Overall Science Achievement for Upper Grade 7 2 1 6 Linking and Tracking Variables Information about students teachers and schools provided on the survey tracking forms is included in linking or tracking variables These variables have prefixes of IL or IT Some of the important linking and tracking variables are listed below Variables Included in the
87. Statistics for the International Mathematics Achievement Scores for Population 2 Variable BIMATSCR sse nnne 6 7 Descriptive Statistics for the International Science Achievement Scores for Population 2 Variable BISCISCR sse 6 8 TIMSS Population 1 and Population 2 Data Files sss 7 2 Country Identification and Inclusion Status in Population 1 and Population 2 Data Files 7 3 Background Questionnaire Item Field Location Format Conventions 7 11 International Background Variable Naming Conventions sss 7 12 International Report Table Figure Location Reference Definition for Derived Variables 7 13 Variable Name Definitions for the Written Assessment and Performance Assessment Ifems 2 on criticae tes dis trece t eei decret c o c me ence ode E Means oben 7 17 TIMSS DATABASE USER GUIDE Table 7 7 Table 7 8 Table 7 9 Table 7 10 Table 7 11 Table 7 12 Table 7 13 Figure 7 1 Table 7 14 Table 7 15 Figure 7 2 Figure 7 3 Figure 9 1 Figure 9 2 Table 9 1 Figure 9 3 Figure 9 4 Figure 9 5 Table 9 2 Figure 9 6 Figure 9 7 Figure 9 8 Figure 9 9 Figure 9 14 Figure 9 16 CONTENTS Classification of Population 1 Items into Mathematics Content Area Reporting Categories oett trem em ere ed ena I ERE e Ee e pe ERE g Se Rn 7 19 Classification of Population 1 Items into Science Content Area Reporting Categories 7 20 Classification
88. Student Background Files ITSEX Gender of each student ITBIRTHM Month of birth of each student ITBIRTHY Year of birth of each student ITDATEM Month of testing for each student ITDATEY Year of testing for each student Survey Tracking forms are listings of students teachers or schools used for sampling and administration purposes Some additional linking and tracking variables included in the international background files but not listed above are indicated in the codebooks described in Section 7 3 TIMSS DATABASE USER GUIDE DATABASE FILES ITLANG ITPART C HAZ TER Z Language of testing for each student Data are included for six countries that administered the TIMSS test in more than one language Canada 1 English 2 French Norway 1 Bokmal 2 Nynorsk South Africa 1 English 2 Afrikaans Spain 1 Castellano 3 Catalan 4 Gallego 8 Valenciano Switzerland 1 German 2 French 3 Italian Romania 348 Hungarian 642 Romanian Participation status variable indicating whether each student participated in any TIMSS session only those students with ITPART equal to 3 participated in a TIMSS session are included in the Student Background files ITPARTI ITPART2 ITPART3 Separate participation status variables for students ILRELIAB related to the specific sessions as applicable first half testing session second half testing session and questionnaire session Linking variable indicating t
89. Sweden Switzerland Thailand United States i Eighth grade in most countries see Table 2 for more information about the grades tested in each country Standard errors appear in parentheses Because results are rounded to the nearest whole number some totals may appear inconsistent Countries shown in italics did not satisfy one or more guidelines for sample participation rates age grade specifications or classroom sampling procedures see Figure A 3 Background data for Bulgaria and South Africa are unavailable Because population coverage falls below 65 Latvia is annotated LSS for Latvian Speaking Schools only A tilde indicates insufficient data to report achievement NOR Ob O MO ooa amp 0 Q Q Oo 0 o co SSStssstssssisssszsssss c M Oo do ss BDAGGSRINAVNADGNAGAGEDGNSNVIGCANZSSSGaGosoa B Se ozooNeoreess es eue FPoys oe eoynswperosn 1 o 9979 NW590070gco7UNNEAXOOWN ONRETIESRETMOTONOODRESGREKRE 2 v e amp B amp omiiioniou obosloscoimsoesselsoooisolemsimMc AS5BSO0z UWzwepogzSGzzOUOSUTZUUozuugozzoozusos5 do d 5 cio lilio 5l le5NooeesGlleuiisLsoelMeslMop EIC ERS a SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 9 2 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 2 Sample Table for Teacher Level Analysis Taken From the TIMSS International Report Mathematics Achievement in the Middle School Years
90. TRY 9 1 8 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 DVAR This is the variable for which means are to be computed Only one variable has to be listed here If the user wants to examine for example results in mathematics and science then the macro needs to be invoked separately to generate each table Although in most cases the continuous variable of interest will be an achievement variable this can actually be any other continuous variable Unlike the SAS macro JACK SAS the JACK macro in SPSS does not require that the data file that contains the data of interest be specified when calling the macro By default SPSS uses the current working file This needs to be read with the GET FILE command prior to invoking the macro The simplest and most straightforward way is to invoke the macro using the conventional SPSS notation for invoking macros This involves listing the macro name followed by the corresponding list of arguments for the analysis each separated by a slash For example if the macro is invoked as get file BSGALL1 jack cvar IDCNTRY IDGRADER ITSEX dvar BIMATSCR jkz JKZONE jki JKINDIC njkr 75 WGT TOTWGT it will compute the mean mathematics achievement and its corresponding standard error for boys and girls by grade within each country using the variable TOTWGT as the sampling weight It will also compute the percent of boys and girls by grade within the country and it
91. This resulted in equal selection probabilities within national samples for the students in the calibration samples The Populations 1 and 2 international calibration samples contained 14 700 and 23 100 students respectively 5 8 2 Standardizing the International Scale Scores The plausible values produced by the scaling procedure were in the form of logit scores that were on a scale that ranged generally between 3 and 3 For reporting purposes these scores were mapped by a linear transformation onto an international scale with a mean of 500 and a standard deviation of 100 Each country was weighted to contribute the same when the international mean and standard deviation were set with the exception of the countries that tested only one grade Countries that tested only one grade had only half the contribution of TIMSS DATABASE USER GUIDE Saf CHAPTER 5 SCALING the remaining countries The contribution of the students from each grade within each country was proportional to the number of students at each grade level within the country The transformation applied to the logit scores was 0j 0 Si 500 100 s 6j where S is the standardized scale score for student i in plausible value j in country k Ois the logit score for the same student 0 is the weighted average across all countries on plausible value j and SD is the standard deviation across all countries on plausible value j Note that while t
92. Y1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 1488 687980 70 503 686 5 72799 22 1953 0 75219 1 to 2 hrs 2668 1239197 59 512 935 5 10220 39 9783 0 90504 3 to 5 hrs 1727 767004 30 501 228 4 22001 24 7447 0 58673 More than 5 hrs 1001 405491 75 460 767 4 57929 13 0818 0 95070 COUNTRY ID Slovenia GRADE Eighth Grade BSBGDAY1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 614 5838 91 546 071 4 11421 22 6437 1 06473 1 to 2 hrs 1442 13852 82 540 938 3 36504 53 7221 1 09384 3 to 5 hrs 516 4992 78 540 094 4 65821 19 3623 0 86774 More than 5 hrs 112 1101 55 517 721 9 92666 4 2719 0 42142 TIMSS DATABASE USER GUIDE 9 25 CHAPTER 9 Figure 9 12 Extract of SPSS Computer Output for Performing Analyses with Student Level Variables EXAMPLE 1 PERFORMING ANALYSES GENNOUTSIDE SCHLNWATCH TY OR COUNTRY ID VIDEOS TOTWGT MNX MNX SE PCT PCT SE Australia Eighth Grade Less than 1 Hr 52718 94 538 88 6 00 23 53 92 1 to 2 hrs 92637 64 538 58 4 13 41 34 84 3 to 5 hrs 59663 57 527 54 3 80 26 62 84 More than 5 hrs 19073 65 487 01 5 47 8 51 61 Austria Eighth Grade Less than 1 Hr 21112 51 539 82 5 42 25 29 1 40 1 to 2 hrs 44420 93 545 82 4 21 53 21 1 06 3 to 5 hrs 14158 96 539 47 5 20 16 96 98 More than 5 hrs 3795 76 496 85 8 64 4 55 57 Belgium F1 Eighth Grade Less than 1 Hr 18055 42 579 95 6 67 24 35 1 24 1 to 2 hrs 38846 19 575 12 6 15 52 40 1 18 3 to 5 hrs 13808 20 534 66 7 12 18 63 1 03 More than 5 hrs 3426 62 5
93. ach questionnaire type are listed in Table 7 15 Table 7 15 Data Almanac Files Almanac File Contents ASGALM LIS Student background data Population 1 ATGALM LIS Teacher background data Population 1 ACGALM LIS School background data Population 1 BSGALM LIS Student background data Population 2 BTMALM LIS Mathematics teacher background data Population 2 BTSALM LIS Science teacher background data Population 2 BCGALM LIS School background data Population 2 There are two types of displays in the data almanacs depending on whether the item is a categorical variable or a continuous variable The display for categorical variables includes the sample size the count of students who were not administered the question the count of students choosing each of the options on the question and the count of students who did not choose any of the valid options to the questions In cases where the question did not apply the count of students to whom the question did not apply is also presented in the almanac An example of a categorical variable almanac display is shown in Figure 7 2 TIMSS DATABASE USER GUIDE 7 39 CHAPTER 7 Figure 7 2 Example Data Almanac Display for Categorical Variable Third International Mathematics and Science Study Student Background Variables Population 1 Unweighted data almanac for the TIMSS International Database Question Are you a girl or a boy ASBGSEX Location SQ1 2 R GRADE INDICATOR Lo
94. ach student in mathematics and science The need for plausible values arises from the fact that any student was administered only a fraction of the items in the assessment as described in Chapter 2 Time constraints did not allow for all the items to be administered to each student A plausible value is an estimate of how the individual student would have performed on a test that included all possible items in the assessment see Chapter 5 Since no student responded to all items this estimate is based on the responses to the items that were included in the test booklet that the student actually took and the performance of students with similar characteristics Plausible values have been shown to improve the estimation of population parameters They were developed during the analysis of the 1983 84 NAEP data in order to improve estimates of population distributions The general theory of the NAEP plausible values can be attributed to Mislevy Mislevy and Sheehan 1987 1989 based on Rubin s work Rubin 1987 on multiple imputations Within a subject area and across the sample one set of plausible values can be considered as good as another Each of these sets is equally well designed to estimate population parameters although the estimates will differ somewhat This difference is attributable to imputation error Five sets of plausible values are provided so that analyses may be replicated as many as five times Results which vary from replicatio
95. administered to a student or the code assigned by the coders to the student s response or report of a task The user might want to work with these item data after they are recoded to the right wrong format in the case of multiple choice items or to the level of correctness in the case of the open ended items and performance assessment tasks To this effect we have included in the CD a set of programs in SPSS and SAS that will allow the user to recode the items from the written assessment and from the performance assessment to their right wrong or correctness level format Each of these programs contains a macro called SCOREIT and the necessary call to this macro so that all the items in the corresponding file are scored These programs will convert the response option codes for multiple choice items to dichotomous score levels 0 or 1 based on scoring keys For the open ended items the two digit diagnostic codes will be converted to the corresponding correctness score level 3 2 1 0 based on the value of the first digit as described in Chapter 4 9 40 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 The filenames in SAS and SPSS have been kept constant except for the last three characters of the file name As defined previously in Table 7 14 four files are included to provide control code to perform the recodes of the test items in the written and performance assessment in Populations 1 and 2 e Written Assessment Files ASASC
96. ained from the first coder e Second Code Variable cognitive item response codes obtained from second coder e Response Code Agreement Variable degree of agreement between the two codes It should be noted that the Second Code Variable data were used only to evaluate the within country coding reliability Only the first codes contained in the student assessment files were used in computing the achievement scores reflected in the Student Background files and the international reports Z 24 TIMSS DATABASE USER GUIDE DATABASE FILES CRAP IHE RS X Reliability Variable Names The variable names for the Original Code Second Code and Agreement Code variables are based on the same general naming system as that for the Original Code variables shown in Table 7 6 The three reliability variables may be differentiated by the second character in the variable name e Original Code Variable Second Character S e g ASSSROI BSPM42 e Second Code Variable Second Character R e g ARSSROI1 BRPM42 e Agreement Code Variable Second Character I e g AISSRO1 BIPM42 Reliability Variable Code Values The values contained in both the Original Code and Second Code variables are the two digit diagnostic codes obtained using the TIMSS scoring rubrics The Agreement Code variable has three different values depending on the degree of agreement between the two coders 0 Identical codes both digits in the diagnostic codes are identical 1 Id
97. ak Republic Kjell Gisselberg Sweden Galina Kovalyova Russian Federation Nancy Law Hong Kong Josette Le Coq France Jan Lokan Australia Curtis McKnight United States Graham Orpwood Canada Senta Raizen United States Alan Taylor Canada Peter Weng Denmark Algirdas Zabulionis Lithuania Performance Assessment Committee Derek Foxman England Robert Garden New Zealand Per Morten Kind Norway Svein Lie Norway Jan Lokan Australia Graham Orpwood Canada ACKNOWLEDGMENTS TIMSS DATABASE USER GUIDE ACKNOWLEDGMENTS Quality Assurance Committee Jules Goodison United States Hans Pelgrum The Netherlands Ken Ross Australia Editorial Committee David F Robitaille Chair Canada Albert Beaton International Study Director Paul Black England Svein Lie Norway Rev Ben Nebres Philippines Judith Torney Purta United States Ken Travers United States Theo Wubbels the Netherlands TIMSS DATABASE USER GUIDE
98. ak Republic d 890 Slovenia 717 South Africa 724 Spain 752 Sweden 756 Switzerland 764 Thailand x 840 United States Now use the macro to get the results include jack sps jack cvar identry idgrader bcbgcomm dvar bimatscr njkr 75 jkz jkzone jki jkindic wgt totwgt Sort cases by idcntry idgrader temporary select if idgrader 2 report format list automatic var bcbgcomm label totwgt mnx mnx se pct pct se break idcntry idgrader 9 36 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES Figure 9 19 CHAPTER Extract of SAS Computer Output for Performing Analyses with School Level Variables EXAMPLE 3 COUNTRY ID Australia GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 2 866 33600 29 512 906 11 3257 17 1558 3 71297 Community Type 3 3184 98242 75 531 360 7 2504 50 1613 4 36371 Community Type 4 2177 64010 75 528 813 7 7368 32 6829 4 41223 COUNTRY ID Austria GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 1 40 303 99 605 685 2 1439 0 3818 0 27798 Community Type 2 908 38258 26 526 105 5 0042 48 0505 3 65878 Community Type 3 287 7440 07 537 116 20 9813 9 3444 2 77784 Community Type 4 1212 33618 58 551 566 5 2165 42 2233 4 01114 COUNTRY ID Belgium Fl GRADE Eighth Grade BCBGCOMM N TOTWGT MNX MNX SE PCT PCT SE Community Type 2 898 22954 95 562 497 7 2354 33 5920 4 81714 Community Type 3 729 16854
99. ally after sorting by a set of implicit stratification variables This resulted in the implicit creation of 75 strata with two schools selected per stratum The jackknife repeated replication JRR method is suitable for estimating sampling errors in the TIMSS design because it provides approximately unbiased estimates of the sampling error arising from the complex sample selection procedure for estimates such as percentages totals and means In addition this method can also be readily adapted to the estimation of sampling errors for parameters estimated using other statistical modeling procedures such as percent correct technology The general use of the JRR entails systematically assigning pairs of schools to sampling zones and the random selection of one of these schools to have its contribution doubled and the other zeroed so as to construct a number of pseudo replicates of the original sample The statistic of interest is computed once for all of the original sample and once more for each of the pseudo replicate samples The variation between the original sample estimate and the estimates from each of the replicate samples is the jackknife estimate of the sampling error of the statistic 8 1 Computing Error Variance Using the JRR Method When implementing the JRR method in TIMSS for each country we assume that there were up to 75 strata or zones H within each country each one containing two sampled schools selected independently W
100. an for the mathematics items they were still quite good For the 26 countries at Population 2 the averages across items for the correctness score ranged from 88 to 100 and the overall average across the 26 countries was 9596 The overall average for diagnostic score agreement was 87 with a range of 73 to 98 At Population 1 the average agreement for correctness was 94 ranging from 83 to 99 The average exact agreement on diagnostic codes was 87 overall with a range of 72 to 90 TIMSS also conducted a special cross country coding reliability study at Population 2 in which 39 scorers from 21 of the participating countries coded common sets of student responses In this study an overall average percentage of exact agreement of 92 for correctness scores and 8046 for diagnostic codes was obtained For more details about the TIMSS free response scoring refer to e Scoring Techniques and Criteria Lie Taylor and Harmon 1996 Training Sessions for Free Response Scoring and Administration of the Performance Assessment Mullis Jones and Garden 1996 Guide to Checking Coding and Entering the TIMSS Data TIMSS 1995 e Quality Control Steps for Free Response Scoring Mullis and Smith 1996 4 3 Data Entry To maintain equality among countries very little optical scanning and no image processing of item responses was permitted Essentially all of the student test information was recorded in the student booklets or on separa
101. arranging for data collection coding and data entry preparing the data files for submission to the IEA Data Processing Center contributing to the development of the international reports and preparing national reports The way in which the national centers operated and the resources that were available varied considerably across the TIMSS countries In some countries the tasks were conducted centrally while in others various components were subcontracted to other organizations In some countries resources were more than adequate while in others the national centers were operating with limited resources Of course across the life of the project some NRCs have changed This list attempts to include all past NRCs who served for a significant period of time as well as all the present NRCs All of the TIMSS National Research Coordinators and their staff members are to be commended for their professionalism and their dedication in conducting all aspects of TIMSS This list only includes information for those countries for which data are included in the International Database TIMSS DATABASE USER GUIDE ACKNOWLEDMENTS Australia Jan Lokan Raymond Adams Australian Council for Educational Research 19 Prospect Hill Private Bag 55 Camberwell Victoria 3124 Australia Austria Guenter Haider Austrian IEA Research Centre Universitat Salzburg Akademiestrabe 26 2 A 5020 Salzburg Austria Belgium Flemish Christiane Brusselmans
102. ata files in ASCII format The different data file types that are in this directory are described in Chapter 7 Each of these files has two corresponding control files in the PROGRAMS sub directory as shown in Table 7 14 One of these two files reads the ASCII data and creates a SAS data set the second one reads the ASCII data and creates an SPSS system file There are several other programs in this directory The other programs that can be found in this directory are the following JACK SAS and JACK SPS Both of these are macro programs one in SAS and one in SPSS that can be used to compute the weighted percent of students within defined groups and their mean on a 9 4 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES C HAPTER 9 specified continuous variable These macros also generate replicate weights and compute JRR sampling variance for the percent and mean estimates Although in general the continuous variable of choice will be one of the achievement scores in either mathematics or science this continuous variable can be any other continuous variable in the file How to use each of these macro programs is described later in this chapter EXAMPLEI SAS EXAMPLE2 SAS and EXAMPLE3 SAS EXAMPLEI SPS EXAMPLE2 SPS and EXAMPLE3 SPS These are the programs used in the examples presented later in this chapter These programs are included only in the CD for Population 2 although the same examples can be easily adapted to perform the same ana
103. ata in other files e Identification variables Linking and tracking variables e Sampling variables e Score variables The codebooks for student assessment files include a complete list of all of these variables as well as item and code value labels to assist in the interpretation of the student assessment data 7 2 2 9 School Performance Assessment Files The School Performance Assessment files for Population 1 and Population 2 contain school level information relevant to the administration of the performance assessment In these files there is one record for each school that participated in the performance assessment in each country containing identification variables required to link to other files In addition a number of tracking variables are included with specific information provided by the performance assessment administrators in each school The variables included in the School Performance Assessment files are the following ITROTAT Rotation scheme used ITPASI Adequacy of room ITPAS2 Time allotted per station ITPAS3 Adequacy of equipment and material ITPAS4 Missing information from manual ITPASS Clearness of instructions For the performance assessment files additional tracking variables are included related to student participation in the performance assessment such as to which rotation and sequence number each student was assigned ITROTAT ITSEQ and students participation status for each performance assessm
104. ategory Data Fractions and Algebra Measurement Geomet Representation Proportionality Number Sense ii X Analysis and i Probability 51 items 27 items 18 items 23 items 20 items 11 items Item M09 was excluded from Population 2 data files due to problematic item statistics TIMSS DATABASE USER GUIDE 7 21 CHAPTER 7 DATABASE FILES Table 7 10 Classification of Population 2 Items into Science Content Area Reporting Categories Items in Content Area Reporting Category Environmental Issues and the Nature of Science Earth Science Life Science Physics Chemistry 22 items 40 items 40 items 19 items 14 items 7 22 TIMSS DATABASE USER GUIDE DATABASE FILES C ACP TRE RS 7 2 2 7 Release Status of TIMSS Test Items and Performance Tasks To aid in the interpretation of the student cognitive item response data a large number of TIMSS items have been released to the public along with their scoring guides For both Populations 1 and 2 written assessment items in Clusters I through Z and the entire set of performance assessment tasks and coding guides are available See Chapter 2 for information on how to obtain the released item sets 7 2 2 8 Other Variables in the Student Assessment Files In addition to the written assessment and performance assessment item variables a number of other variables described in previous sections are included for each student to aid in case identification and in linking to the d
105. ation number to identify each student within each country Since teachers may teach more than one class the IDTEACH and IDLINK combinations in the Teacher Background files provide a unique identification for each teacher teaching a specific class Teacher background variables are linked to appropriate students using the Student Teacher Linkage file described in Section 7 2 4 The variable IDSCHOOL contained in all three background files is a unique identification number for each school within a country that may be used to link school background data to corresponding students or teachers In the Teacher Background data file the identification variable IDSUBJCT may be used to identify teachers as either mathematics only science only or mathematics science teachers For Population 2 separate background files are provided for mathematics and science teachers At Population 1 a single questionnaire was given and only one Teacher Background file is 7 O TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTERS X provided including all Population 1 teachers In some countries however a portion or even all of the students were taught mathematics and science by different teachers Therefore the IDSUBJCT variable was used to identify the appropriate teachers to include in background data tables in the international mathematics and science reports for Population 1 7 2 1 5 Achievement Scores Student Level Achievement Score Variables Variables
106. bimatscr select if not missing itsex Now use the macro to get the results include jack sps jack cvar idcntry idgrader itsex dvar bimatscr njkr 75 jkz jkzone jki jkindic wgt totwgt sort cases by idcntry idgrader list var all IDCNTRY IDGRADER ITSEX N TOTWGT MNX PCT 36 1 Jy 3039 123648 61 500 451 51 8891 36 1 2 2560 114645 55 495 142 48 1109 36 2 1 3722 114806 50 531 990 49 6409 36 2 2 3529 116467 28 527 357 50 3591 40 1 1 1545 45945 00 508 568 53 2220 40 1 2 1358 40382 01 509 963 46 7780 40 2 J 1321 42574 48 535 625 50 0342 40 2 2 1385 42516 32 543 581 49 9658 56 1 1 1344 31717 08 558 544 49 4211 56 1 2 1424 32460 16 556 717 50 5789 56 2 1 1437 37480 16 567 222 49 9276 56 2 2 1457 37588 84 563 139 50 0724 840 1 1 1976 1587912 47 473 310 50 3006 840 T 2 1910 1568934 35 478 069 49 6994 840 2 1 3561 1574689 22 497 464 49 3897 840 2 2 3526 1613607 42 501 996 50 6103 890 1 1 1486 14410 14 495 847 51 3907 890 1 2 1411 13630 25 500 643 48 6093 890 2 1 1381 13341 15 536 878 51 3542 890 2 2 1324 12637 53 544 918 48 6458 9 4 Performing Analyses with Student Level Variables Many analyses of the TIMSS data can be undertaken using student level data We have CHAPTER 9 MNX SE 4 5 4 5 27647 4 58177 4 54757 3 23439 4 4 7 8 w UU Ul 32185 20426 58168 10432 71205 47097 43010 78875 74234 74972 47104 21162 3 3 3 3 22189 48741 29050 76508 PCT SE 2
107. bing how specific national deviations were handled in the international derived variables is also summarized in Supplement 4 for each of the report variables The response option definitions for all derived variables as well as their associated international report table and figure references also are included in the student and teacher codebook files which are described in Section 7 3 Derived Variables Based on School Background Data In addition to the derived reporting variables based on the student and teacher background data several additional derived variables are included in the school background files for Population 1 and Population 2 These variables are listed below with the questionnaire locations of the school background variables used to obtain them SCQ1 SCQ2 Population 1 or Population 2 School Questionnaire Item Number ACDGTENR BCDGTENR Total school enrollment sum of boys and girls reported in SCQI 16A SCQ2 17A ACDGLTER BCDGTENR Total lower grade enrollment sum of boys and girls reported in SCQ1 16E SCQ2 17E ACDGLTRT BCDGLTRT Total students repeating lower grade sum of boys and girls reported in SCQ1 16F SCQ2 17F ACDMLTER BCDMLTER Total lower grade students studying mathematics sum of boys and girls reported in SCQ1 16I 5SCQ2 17T ACDSLTER BCDSLTER Total lower grade students studying science sum of boys and girls reported in SCQ1 16J SCQ2 17J ACDGUTER BCDGUTER Total upper grade enrollment sum of boys a
108. booklet set to which each student was assigned ITBSET and the identification of the first and second coders ITCODE ITCODE2 7 2 4 Student Teacher Linkage Files General Information The Student Teacher Linkage files for Population 1 and Population 2 contain information required to link the student and teacher files and to compute appropriately weighted teacher level data using the student as the unit of analysis Example analyses using these data files are described in Chapter 9 The Student Teacher Linkage files contain one entry per student teacher linkage combination in the data In many cases students are linked to more than one mathematics and or science teacher and in these cases there will be one record for each student teacher link This is particularly true in Population 2 where the majority of students have separate teachers for science and mathematics In addition in some countries many students may also have more than one teacher for each of the two subject areas For instance if three teachers are linked to a student there are three entries in the file corresponding to that student Variables Required for Computing Teacher Level Data The following important variables required to compute appropriately weighted teacher level data are included in the linkage files NMTEACH Number of mathematics teachers for each student NSTEACH Number of science teachers for each student NTEACH Number of total teachers for each student
109. btained on the science items Population 2 After the items were recoded as right or wrong or to their level of correctness in the case of the open ended items raw scores were computed by adding the number of points obtained across the items for the subject Multiple choice items received a score of either 1 or 0 Open ended response items receive score points from 0 to 3 depending on their coding guide Open ended items with a first digit of 7 or 9 indicating an incorrect incomplete answer were given zero points The value of the first digit of the code determines the number of score points assigned to an open ended item A description of the algorithm used to score the items can be found in Chapter 9 in the section Scoring the Items Although these scores can be used to compare students performances on the same booklet they should not be used to compare students performances across different booklets Different booklets contain different numbers of items for each subject and the specific items contained in one booklet had varying difficulties It is recommended that these scores be used only to verify whether the items have been recoded correctly when a user decides to recode the items to their level of correctness Raw scores can be found in the Student Background data files and in the Written Assessment data files For the user not familiar with the data files included in the International Database we recommend reading Chapter 7 before
110. ce Items Released Set for Population 1 Third and Fourth Grades TIMSS Mathematics Items Released Set for Population 2 Seventh and Eighth Grades TIMSS Science Items Released Set for Population 2 Seventh and Eighth Grades To obtain copies of the TIMSS released item sets contact the International Study Center TIMSS International Study Center Campion Hall Room 323 CSTEEP Boston College Chestnut Hill MA 02467 United States Phone 1 617 552 4521 Fax 1 617 552 8419 e mail timss bc edu The TIMSS released items also are available on the World Wide Web at wwwcsteep bc edu timss 2 12 TIMSS DATABASE USER GUIDE Chapter 3 Sampling and Sampling Weights This chapter describes the selection of school and student samples and the sampling weights included on the TIMSS data files The TIMSS sample design is fully detailed in Foy Rust and Schleicher 1996 and Foy 19972 The weighting procedures are described in Foy 1997b 3 1 The Target Populations The selection of valid and efficient samples is crucial to the quality and success of an international comparative study such as TIMSS For TIMSS National Research Coordinators worked on all phases of sampling with staff from Statistics Canada In consultation with the TIMSS Sampling Referee Keith Rust WESTAT Inc staff from Statistics Canada reviewed the national documentation on sampling plans sampling data sampling frames and sample execution T
111. ce of student scores in analyses plausible values are designed primarily to estimate population parameters and are not optimal estimates of individual student proficiency This chapter provides details of the IRT model used in TIMSS to scale the achievement data For those interested in the technical background of the scaling the chapter describes the model itself the method of estimating the parameters of the model and the construction of the international scale 5 1 The TIMSS Scaling Model The scaling model used in TIMSS was the multidimensional random coefficients logit model as described by Adams Wilson and Wang 1997 with the addition of a multivariate linear model imposed on the population distribution The scaling was done with the ConQuest software Wu Adams and Wilson 1997 that was developed in part to meet the needs of the TIMSS study The multidimensional random coefficients model is a generalization of the more basic unidimensional model TIMSS DATABASE USER GUIDE Sus CHA RYT ESR 5 SCALING 5 2 The Unidimensional Random Coefficients Model Assume that items are indexed i J with each item admitting K 1 response alternatives k 0 1 K Use the vector valued random variable X D OD UNE M 1 where X 1 if response to item i is in category j O otherwise to indicate the K 1 possible responses to item i A response in category zero is denoted by a vector of zeroes This effe
112. ces precluded several countries from providing this information At Population 2 a very high percentage of exact agreement was observed on the mathematics items for all countries with averages across items for the correctness score ranging from 97 to 100 and an overall average of 99 across all 26 countries The corresponding average across all countries for diagnostic code agreement was 95 with a range of 89 to 99 At Population 1 the average agreement for correctness on the mathematics items was very similar 97 overall for the 16 countries ranging from 88 to 100 For diagnostic codes the overall average was 93 and the range was from 79 to 99 TIMSS DATABASE USER GUIDE 4 5 CHAPTER 4 DATA COLLECTION PROCEDURES Table 4 1 TIMSS Within Country Free Response Coding Reliability Data Correctness Score Agreement Diagnostic Code Agreement Range Across Range Across Countries of Average Overall Average Countries of Average Percent of Exact Percent of Exact Agreement Overall Average Percent of Exact Percent of Exact Agreement Agreement Across All Items Min Max Across ltems Agreement Mathematics Population 1 Population 2 Science Population 1 Population 2 Note Overall averages are based on all countries providing coding reliability data Population 1 data are based on 16 countries Population 2 data are based on 26 countries While the percentages of exact agreement for science items were somewhat lower th
113. chniques to the estimation of sampling variability consists of assigning the schools to implicit strata known as sampling zones Most TIMSS sample designs in the participating countries called for a total of 150 sampled schools per target population Each of these 150 schools were assigned to one of 75 sampling zones These zones were constructed by pairing the sequentially sampled schools and assigning these pairs to a sampling zone Since schools were generally sorted by a set of implicit stratification variables the resulting assignment of sequentially sampled schools to sampling zones takes advantage of any benefit due to this implicit stratification In cases when more than 75 pairs of schools were sampled within a country schools were then assigned to sampling zones in a way such that no more than 75 sampling zones were defined In some cases this meant assigning more than two schools to the same sampling zone Sampling zones were constructed within explicit strata In cases when there was an odd number of schools in an explicit stratum either by design or because of school level non response the students in the remaining school were randomly divided into two quasi schools for the purposes of calculating the jackknife standard error Each zone then consisted of a 8 2 TIMSS DATABASE USER GUIDE SAMPLING VARIANCE CHAPTER 8 pair of schools or quasi schools When computing replicate weights for the estimation of JRR sampling error
114. computed The adjusted sampling weights are included in the International Database As a consequence any computations using these new weights may be slightly different from those shown in the international report tables for Population 1 Bulgaria The student teacher and school background data submitted by Bulgaria were not deemed internationally comparable and are thus not included in the International Database Israel Due to the use of unapproved school sampling procedures for the performance assessment the performance assessment results for Israel in the international report are based on unweighted data at both the fourth and eighth grades Harmon et al 1997 Thus the sampling weights have been set to 1 for all cases in the performance assessment file for Israel TIMSS DATABASE USER GUIDE 7 3 CHAPTER 7 DATABASE FILES New Zealand There are some incorrect Population 1 student teacher linkages for students in multi grade classes containing both lower and upper grade students In some of these classes different teachers for mathematics and or science are linked to the students in the other grade included in their composite class As a result some students in these classes were given not only the correct linkages to their own teacher s but also some incorrect linkages to teacher s of students in the other grade in their class The student teacher linkage problems were discovered after the production of the international rep
115. cordance with detailed specifications derived from the TIMSS mathematics and science curriculum frameworks Robitaille et al 1993 Initial results from this component of TIMSS can be found in two companion Countries on a Southern Hemisphere school schedule Australia Korea New Zealand and Singapore tested students in September November 1994 All other countries tested students in 1995 TIMSS DATABASE USER GUIDE 3 CHAPTER IN TRODUCTION volumes Many Visions Many Aims A Cross National Investigation of Curricular Intentions in School Mathematics Schmidt et al 1997 and Many Visions Many Aims A Cross National Investigation of Curricular Intentions in School Science Schmidt et al 1998 To collect data about how the curriculum is implemented in classrooms TIMSS administered a broad array of questionnaires which also collected information about the social and cultural contexts for learning Questionnaires were administered at the country level about decision making and organizational features within the educational systems The students who were tested answered questions pertaining to their attitudes towards mathematics and science classroom activities home background and out of school activities The mathematics and science teachers of sampled students responded to questions about teaching emphasis on the topics in the curriculum frameworks instructional practices textbook use professional training and educatio
116. cotland Singapore Spain Sweden Switzerland Countries not satisfying guidelines for sample participation Australia Australia England Hong Kong Netherlands United States United States Countries not meeting age grade specifications high percentage of older students Colombia Slovenia Romania Slovenia Countries with small sample sizes Hong Kong Countries with unapproved sampling procedures Israel Israel Met guidelines for sample participation rates only after replacement schools were included 1 National Desired Population does not cover all of International Desired Population German speaking car National Defined Population covers less than 90 percent of National Desired Population before allowing f performance assessment sample exclusions 3 School level exclusions for performance assessment exceed 25 of National Desired Population SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUIDE 3i 3 CHAPTER 3 SAMPLING 3 7 Sampling Weights Appropriate estimation of population characteristics based on the TIMSS samples requires that the TIMSS sample design be taken into account in all analyses This is accomplished in part by assigning a weight to each respondent where the sampling weight properly accounts for the sample design takes into account any stratification or disproportional sampling of subgroups and includes adjustments for non response T
117. cros compute the percentage of students within a subgroup defined by a set of classification variables the JRR standard error of this percentage the mean for the group on a variable of choice and the JRR standard error of that mean Each of these macros operate as follows 1 Computes a set of 75 replicate weights for the record using the procedure described in the previous chapter 2 Aggregates or summarizes the file computing the sum of the weights for each category the sum of the weights overall and the sum of a weighted analysis variable 3 Computes the percent of people within each group their mean on the analysis variable and their corresponding standard errors In SPSS the resulting working file contains the corresponding statistics In SAS the file FINAL contains the corresponding statistics When using each of these macros the user needs to specify a set of classification variables one analysis variable the number of replicate weights to be generated the variable that contains the sampling information such as JKZONE and JKINDIC and the sampling weight that is to be used for the analysis When using the SAS macro the user will also need to specify the data file that contains the data that is to be processed In SPSS the macro uses the current working file which is the file that was last read 9 3 1 SAS Macro for Computing Mean and Percents with Corresponding Standard Errors JACK SAS The CD containing the TIMSS Internat
118. ct of three independent events selecting the school the classroom and the student To obtain the probability of selection for an individual student we need to multiply three selection probabilities school classroom and student and their respective adjustment factors The resulting product of these three probabilities gives us the individual probability of selection for the student Inverting this probability give us the sampling weight for the student The same result is achieved by multiplying the different weights of the different selection stages school classroom student Three versions of the students sampling weight are provided in the user database All three give the same figures for statistics such as means and proportions but vary for statistics such as totals and population sizes Each one has particular advantages in certain circumstances TOTWGT Total Student Weight This is obtained by simply multiplying the variables WGTFAC1 WGTADJ1 WGTFAC2 WGTFAC3 and WGTADJ3 for the student The sum of these weights within a sample provides an estimate of the size of the population Although this is a commonly used sampling weight it sometimes adds to a very large number and to a different number within each country This is not always desirable For example if we want to compute a weighted estimate of the mean achievement in the population across all countries using the variable TOTWGT as our weight variable will lead each cou
119. ctively makes the zero category a reference category and is necessary for model identification The choice of this as the reference category is arbitrary and does not affect the generality of the model We can also collect the X together into the single vector X X X Xj which we call the response vector or pattern Particular instances of each of these random variables are indicated by their lower case equivalents x x and x The items are described through a vector ET amp amp 5 of p parameters Linear combinations of these are used in the response probability model to describe the empirical characteristics of the response categories of each item These linear combinations are defined by design vectors ap j zlee kS 1 K each of length p which can be collected to form a design matrix A pd EMI Adopting a very general approach to the definition of items in conjunction with the imposition of a linear model on the item parameters allows us to write a general model that includes the wide class of existing Rasch models An additional feature of the model is the introduction of a scoring function which allows the specification of the score or performance level that is assigned to each possible response to each item To do this we introduce the notion of a response score b which gives the performance level of an observed response in category j of item i The b can be collected in a vecto
120. curity before and after the test date The School Coordinator administered the teacher questionnaires arranged the testing accommodations trained Test Administrators and arranged for make up sessions when necessary The Test Administrator Manual covered the procedures from the beginning of testing to the return of the completed tests questionnaires and tracking forms to the School Coordinator The manual includes a Test Administration Script to be read by the Test Administrator The Test Administrators were responsible for activities preliminary to the testing session including maintaining the security of the test booklets and ensuring adequacy of supplies and the testing environment Activities during the testing session included distribution of the test booklets to the appropriate students using the Student Tracking Form timing of the testing and breaks and accurately reading the test administration script 4 2 Free Response Scoring Upon completion of each testing session the School Coordinator shipped the booklets questionnaires and forms to the National Research Center The NRCs then organized the instruments for scoring and data entry These procedures were designed to maintain identification information that linked students to schools and teachers minimize the time and effort spent handling the booklets ensure reliability in the free response scoring and document the reliability of the scoring Since approximately one third of th
121. d States DATABASE Sample Size 4741 2526 7594 3308 3256 3056 2955 4396 3038 1698 3361 2889 4306 2777 2054 2790 2504 2219 2650 3132 7030 2521 2870 3819 USER Third Grade Sum of Maximum Weights Jacknifing Zones 232333 3 74 86043 5 68 371166 0 75 9740 3 74 116051 5 73 531682 1 67 98999 8 75 83846 6 62 116778 6 75 3734 6 75 1391859 1 75 58502 8 73 1388749 3 74 607007 1 75 15120 7 59 171561 2 52 48385 9 75 49035 8 70 114774 8 72 59393 4 65 41904 0 75 27453 4 61 883764 8 75 3643393 3 59 GUIDE Sample Size CHAPTER 3 Fourth Grade Sum of Maximum Weights Jacknifing Zones 237328 1 91390 7 389159 8 9995 4 120406 0 534922 4 106181 0 89901 4 117228 1 3739 3 1433314 5 60496 9 66967 0 1438464 9 615004 3 24071 0 18882 5 173406 9 52254 3 49896 4 133186 2 59053 7 41244 0 27685 1 864525 4 3563795 3 CHAPTER 3 SAMPLING Table 3 4 Sample Information for TIMSS Population 2 Countries Seventh Grade Eighth Grade Country Sample Sum of Maximum Sample Sum of Maximum Size Weights Jacknifing Zones Size Weights Jacknifing Zones Australia 238294 2 231349 2 Austria 89592 6 86739 4 Belgium FI 64177 2 75069 0 Belgium Fr 49897 7 59269 8 Bulgaria 140979 0 147094 0 Canada 377731 5 377425 8 Colombia 619462 0 527145 4 Cyprus 10033 0 9347 0 Czech Republic 152491 8 152493 7 Denmark 44980 0 54172 1 England 465457 5 485280 1 France 860657 2 815509 8 Germany 742346 4 726087
122. d for educational systems where science is taught as an integrated subject non specialized version The second version was tailored for educational systems where the sciences biology earth science physics and chemistry are taught separately specialized version Table 2 8 in Chapter 2 shows which countries administered the specialized and non specialized versions of the questionnaire Although most countries chose to use one version of the questionnaire some countries opted to use both The variable ITQUEST identifies the version of the questionnaire that was administered to the students in Population 2 For students who were administered the non specialized version all questions that were given only in the specialized version were coded as not administered For students who were assigned a specialized version of the questionnaire all questions that were asked only in the non specialized version were coded as not administered The Student Background files also contain a series of identification variables link variables sampling variables achievement variables and the derived variables that were used for the creation of the international reports 7 2 1 2 Teacher Background File The mathematics and science teachers of the students who were sampled for TIMSS were administered at least one questionnaire with questions pertaining to their background and their teaching practices in the classes of the sampled students Each teacher was asked to re
123. dent proc print data final noobs IDCNTRY IDGRADER ITSEX N TOTWGT MNX PCT MNX SE 36 1 1 3039 123648 61 500 451 51 8891 4 32185 36 1 2 2560 114645 55 495 142 48 1109 5 20426 36 2 1 3722 114806 50 531 990 49 6409 4 58168 36 2 2 3529 116467 28 527 357 50 3591 5 10432 40 1 1 1545 45945 00 508 568 53 2220 3 27647 40 1 2 1358 40382 01 509 963 46 7780 4 58177 40 2 1 1321 42574 48 535 625 50 0342 4 54757 40 2 2 1385 42516 32 543 581 49 9658 3 23439 56 1 1 1344 31717 08 558 544 49 4211 4 71205 56 1 2 1424 32460 16 556 717 50 5789 4 47097 56 2 1 1437 37480 16 567 222 49 9276 7 43010 56 2 2 1457 37588 84 563 139 50 0724 8 78875 840 1 1 1976 1587912 47 473 310 50 3006 5 74234 840 1 2 1910 1568934 35 478 069 49 6994 5 74972 840 2 1 3561 1574689 22 497 464 49 3897 4 47104 840 2 2 3526 1613607 42 501 996 50 6103 5 21162 890 1 1 1486 14410 14 495 847 51 3907 3 22189 890 1 2 1411 13630 25 500 643 48 6093 3 48741 890 2 1 1381 13341 15 536 878 51 3542 3 29050 890 2 2 1324 12637 53 544 918 48 6458 3 76508 PCT SE 32468 32468 13471 13471 63905 63905 79216 79216 76051 76051 70983 70983 WWNHNRPRPRPENNNN 20563 20563 77280 77280 76075 76075 90855 90855 ocOoooooonmniu TIMSS DATABASE USER GUIDE CHAPTER 9 PERFORMING ANALYSES 9 3 2 SPSS Macro for Computing Mean and Percents with Corresponding Standard Errors JACK SPS The CD containing the TIMSS International Database also contains an SPSS macro program called
124. dents take courses in specific sciences such as biology chemistry earth science or physics specialized version Slightly more than half the countries used the general questionnaire and slightly less than half used the questionnaire developed for use with specific science curriculum areas Table 2 8 presents the countries that administered the specialized and non specialized versions PENO TIMSS DATABASE USER GUIDE INSTRUMENTS CHAPTER 2 Table 2 8 Countries Administering the Specialized and Non Specialized Versions of the Population 2 Student Questionnaire Non Specialized Version Science Specialized Version Science as as an Integrated Subject Separate Subjects Australia Belgium Flemish Austria Belgium French Canada Czech Republic Colombia Denmark Cyprus France England Germany Hong Kong Greece Iran Hungary Ireland Iceland Israel Latvia Japan Lithuania Korea Netherlands Kuwait Portugal New Zealand Romania Norway Russian Federation Philippines Slovak Republic Scotland Slovenia Singapore Sweden Upper Grade South Africa Spain Sweden Lower Grade Switzerland Thailand United States The teacher questionnaires for Population 2 addressed four major areas teachers background instructional practices students opportunity to learn and teachers pedagogic beliefs There are separate questionnaires for teachers of mathematics and of science Since most Population 1 teachers teach all subjects
125. e classrooms of students were sampled Generally in each school one classroom was sampled from each target grade although some countries opted to sample two classrooms at the upper grade in order to be able to conduct special analyses Most participants tested all students in selected classrooms and in these instances the classrooms were selected with equal probabilities The few participants who chose to subsample students within selected classrooms sampled classrooms with PPS As an optional third sampling stage participants with particularly large classrooms in their schools could decide to subsample a fixed number of students per selected classroom This was done using a simple random sampling method whereby all students in a sampled classroom were assigned equal selection probabilities 3 4 Performance Assessment Subsampling For the performance assessment TIMSS participants were to sample at least 50 schools from those already selected for the written assessment and from each school a sample of either 9 or 18 upper grade students already selected for the written assessment This yielded a sample of about 450 students in the upper grade of each population eighth and fourth grades in most countries in each country For the performance assessment in the interest of ensuring the quality of administration countries could exclude additional schools if the schools had fewer than nine students in the upper grade or if the schools were in a remo
126. e the first step in our analysis is to locate the variables of interest in the specific codebook and file We find the variable BCBGCOMM in the School Background file and the student weights and scores in the Student Background file We then proceed to review the documentation on national adaptations and discover that Australia has modified this variable slightly to fit their particular context At this time we could proceed in one of two ways we could exclude Australia from our analysis or we could label the variable accordingly so that we will not be making incorrect inferences about the TIMSS DATABASE USER GUIDE 9 33 CHAPTER 9 PERFORMING ANALYSES specific groups In the latter case since we want to also explore the results for Australia we take the precaution of labeling the values for variable BCBGCOMM in a generic way before we proceed with the analysis After these considerations we then proceed to read the School Background file and keep only the variables that are relevant to our analysis In this case we keep the country identification IDCNTRY and school identification IDSCHOOL We keep these variables because these are the variables that will allow us to merge the school data to the student data We also keep from the School Background file the variable of interest in this case BCBGCOMM We then read the variables of interest from the student data file First we read the identification of the country and the school IDCNTRY a
127. e TIMSS frameworks used in developing the TIMSS tests were subject matter content and performance expectations During test development each item was coded as to the content and performance expectations with which it is associated The TIMSS item classification system permits an item to draw on multiple content areas and to involve more than one performance expectation so that an item may have several content and performance codes However in constructing the tests only the principal code was used for each of the two dimensions For example an item may be coded for content as uncertainty and probability principal code and proportionality problem secondary code When that item was selected for the test only the principal code was considered Because of limitations in resources for data collection a number of the detailed categories in the frameworks were combined into a few mathematics and science content reporting categories In the analysis each item in the TIMSS test was included in one reporting category based on its principal content code Table 2 1 presents the reporting categories for the mathematics and science content areas used in the international reports The classification of items into each mathematics and science reporting category are shown in Tables 7 7 through 7 10 in Chapter 7 Table 2 1 Mathematics and Science Content Area Reporting Categories Population 1 Mathematics Whole numbers Fractio
128. e coverage falls below 65 Latvia is annotated LSS for Latvian Speaking Schools only National Defined Population covers less than 90 percent of National Desired Population SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUIDE 3 9 CHAPTER 3 SAMPLING Figure 3 3 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Population 2 Written Assessment Eighth Grade Seventh Grade Countries satisfying guidelines for sample participation rates grade selection and sampling procedures t Belgium Fl Latvia LSS Belgium Fr Latvia LSS Canada Lithuania Belgium Fl Lithuania Cyprus New Zealand Canada New Zealand Czech Republic Norway Cyprus Norway t England Portugal Czech Republic Portugal France Russian Federation England Russian Federation Hong Kong Singapore France Scotland Hungary Slovak Republic Hong Kong Singapore Iceland Spain Hungary Slovak Republic Iran Islamic Rep Sweden Iceland Spain Ireland 1 Switzerland Iran Islamic Rep Sweden Japan United States Ireland 1 Switzerland Korea Japan United States Korea Countries not satisfying guidelines for sample participation Australia Australia Austria Austria Belgium Bulgaria Bulgaria Netherlands Netherlands Scotland Countries not meeting age grade specifications high percentage of older students Colombia
129. e data from all the countries we selected from the system file or SAS data set only those variables that are relevant to our analysis Although in general this analysis is quite feasible with a powerful desktop computer the user needs to keep in mind that computing and storage requirements for these type of analysis are quite demanding In general to perform analyses such as those in figures 9 9 and 9 10 using the Student Background data files the user needs to do the following e Identify the variable or variables of interest in the student file and find out about any specific national adaptations to the variable e Retrieve the relevant variables from the data files including the achievement score sampling weights JRR replication information and any other variables used in the selection of cases e Use the macro JACK with the corresponding arguments and parameters Print out the result file 9 22 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 9 SAS Control Statements for Performing Analyses with Student Level Variables EXAMPLE1 SAS options nocenter Read the variables from the student file data student set bsgalll keep idcntry idgrader jkindic jkzone totwgt bsbgdayl bimatscr where idgrader 2 and nmiss bsbgdayl 0 Select only 8th graders if bsbgdayl 1 then bsbgdayl 2 Define the format for the variables used proc format library work value grade 1 Seventh Grade 2 Eighth
130. e grade Weighted overall response rates were computed and reported by grade both with and without replacement schools Response rates and school and student sample sizes for the written and performance assessments are available in Mullis et al 1997 Martin et al 1997 Beaton et al 19962 Beaton et al 1996b and Harmon et al 1997 3 6 Compliance with Sampling Guidelines Figures 3 2 and 3 3 for Populations 1 and 2 respectively indicate the degree to which countries complied with the TIMSS sampling guidelines Countries were considered to have met the TIMSS guidelines if they achieved acceptable participation rates 8596 of both the schools and students or a combined rate the product of school and student participation of 7596 with or without replacement schools and if they complied with the TIMSS guidelines for grade selection and classroom sampling Countries that met these guidelines are shown in the top panels of Figures 3 2 and 3 3 Countries that met the guidelines only after including replacement schools are identified Countries not reaching at least 50 school participation without the use of replacement schools or that failed to reach the sampling participation standard even with the inclusion of replacement schools are shown in the second panels To provide a better curricular match several countries elected to test students in the seventh and eighth grades the two grades tested by most countries even
131. e language speakers These are students who cannot read or speak the language of the test and so could not overcome the language barrier of testing Typically students who had received less than one year of instruction in the language of the test were excluded but this definition was adapted in different countries Some countries opted to test students in more than one language 3 2 School Sample Selection In the first stage of sampling representative samples of schools were selected from sampling frames comprehensive lists of all eligible students The TIMSS standard for sampling precision required that all population samples have an effective sample size of at least 400 students for the main criterion variables To meet the standard at least 150 schools were to be selected per target population However the clustering effect of sampling classrooms rather than students was also considered in determining the overall sample size for TIMSS Because the magnitude of the clustering effect is determined by the size of the cluster and the intraclass correlation TIMSS produced sample design tables showing the number of schools to sample for a range of intraclass correlations and minimum cluster size values Some countries needed to sample more than 150 schools Countries however were asked to sample 150 schools even if the estimated number of schools to sample fell below 150 Information about design effect and effective sample size can be found in Gon
132. e number of mathematics and science teachers for the student and a set of weights that can be used when conducting analyses with these data In order to simplify the merging process for analyses that link teacher variables to student achievement student achievement scores sampling weights and JRR replication information have been added to the Student Teacher Linkage file For such analyses it is necessary only to merge the Teacher Background file with the Student Teacher Linkage file For analyses linking teacher variables to other student variables it is necessary also to merge the Student Background files to the other two Conducting analyses with the teacher data requires some extra steps that are not required when analyzing the student or school background data The code in SAS and SPSS for the example that follows is presented in Figures 9 13 and Figure 9 14 For our example we want to find out about the years of experience of the mathematics teachers that teach the eighth grade students in each of the TIMSS countries In particular we want to find out what percent of eighth grade students are taught by teachers of specific groups based on their years of experience and the mean achievement in mathematics of the students taught by those teachers These are the results that are presented in Figure 9 2 earlier in this chapter The TIMSS teacher files do not contain representative samples of teachers within a country Rather these are the teache
133. e used to reproduce the values shown in the international report tables applying the appropriate teacher or student filters and weights Some exceptions are noted in Supplement 4 which lists all derived variables used to produce the international reports along with their corresponding international table and figure reference location The order of variables in the index and in the Student Background file codebooks is based on the referenced table or figure in the international report The nomenclature used to indicate the international table or figure location reference for each derived variable is based on 7 to 9 digit codes according to the general definitions given in Table 7 5 Table 7 5 International Report Table Figure Location Reference Definition for Derived Variables Character Position Definition Value s Population A Population 1 B Population 2 International Report Type MR Mathematics Report SR Science Report Table Figure Reference T a b Table Number F a b Figure Number column section Chapter Number column section if applicable For example BMRT5 9a Population 2 Math Report Table 5 9 first major section Derived Variable Names The derived variable names are based on 7 or 8 digit codes according to the same general definitions as are used for the international background variables see Table 7 4 In the case of the derived variables however the third character is a D derived variab
134. e written test time and all of the performance assessment time was devoted to free response items the free response scoring was complex 4 2 TIMSS DATABASE USER GUIDE DATA COLLECTION PROCEDURES CHAPTER 4 The free response items were scored using item specific rubrics Scores were represented by two digit codes The first digit designates the correctness level of the response The second digit combined with the first represents a diagnostic code used to identify specific types of approaches strategies or common errors and misconceptions This coding approach was used with all free response items including both the short answer and extended response items The number of points specified in each rubric varies by item since each item is unique in terms of answer approach and types of misconceptions generated by students Most items are worth one point In these rubrics correct student responses were coded as 10 11 12 and so on through 19 and earned one score point The type of response in terms of the approach used or explanation provided is denoted by the second digit In all of the guides incorrect student responses were coded as 70 71 and so on through 79 and earned zero score points However as in the approach used for correct scores the second digit in the code represents the type of misconception displayed incorrect strategy used or incomplete explanation given An example guide for a short answer mathematics item is shown
135. eacher or school background questionnaires that were considered not applicable in some countries were not included in their questionnaires Background questionnaire items were mistranslated or not internationally comparable In some cases questions in the international version of the questionnaires were mistranslated or modified to fit the national situation Whenever possible modified background questionnaire items were recoded to match as closely as possible the international version of the items This could not be done in all cases however and some national data were recoded to not administered in order to include only the internationally comparable data Background questionnaire items that have been omitted or where internationally comparable data are not available for some countries are documented in Supplement 3 of this user guide 7 30 TIMSS DATABASE USER GUIDE DATABASE FILES CRAP TE RS 7 Not Applicable Response Codes 6 96 996 The Not Applicable Response Codes are used only for the background questionnaire items in which responses are dependent on a filter question If a dependent question was not applicable to a respondent because he she answered a filter question negatively the dependent question s have been coded to not applicable Also if a respondent was not meant to answer a variable because of its logical relationship to other variables in the questionnaire design these variables also have been recoded
136. ear for National Research Coordinators and their staff members 1 3 TIMSS International Reports The International Database contains the data that were published in 1996 and 1997 in a series of reports prepared by the TIMSS International Study Center at Boston College Mathematics Achievement in the Primary School Years IEA s Third International Mathematics and Science Study Mullis et al 1997 Science Achievement in the Primary School Years IEA s Third International Mathematics and Science Study Martin et al 1997 Mathematics Achievement in the Middle School Years IEA s Third International Mathematics and Science Study Beaton et al 19962 Science Achievement in the Middle School Years IEA s Third International Mathematics and Science Study Beaton et al 1996b Performance Assessment in IEA s Third International Mathematics and Science Study Harmon et al 1997 1 4 Contents of the Database The International Database provided in two compact disks includes more than 3000 variables in more than 500 files One disk contains Population 1 data and the other disk contains Population 2 data The files included on each disk are briefly described below Data Files These files include the written assessment data performance assessment data background questionnaire data coding reliability data information to link students and teachers and sampling weights Codebook Files The codebook files contain all information related
137. ect Area G General G General G General M Mathematics C Classroom M Mathematics S General Science M Mathematics S Science B Biology S Science C Chemistry E Earth Science P Physics or Physical Science RK Abbreviated Question Reference For example BSBGEDUM Population 2 students reports of mother s highest level of education BTBSCLTM Population 2 science teaches reports of minutes in last science lesson ACBMTEAW Population 1 schools reports of how many hours during the school week teachers have for teaching mathematics 7 2 1 8 Variables Derived from Student Teacher and School Background Data General Information In addition to the background variables contained in the student teacher and school questionnaires a number of derived variables based on the student and teacher background variables were computed for use in the international reports These derived variables many of which use combined response options or data from more than one item are also included in the International Database for use in secondary analyses There are also several derived variables based on school background variables that are included in the international school background file although these were not presented in the international reports 7 2 TIMSS DATABASE USER GUIDE DATABASE FILES CRAP TE Rs 7 Use of Derived Variables in the International Reports The derived variables may in general b
138. ed their time and effort to the study MANAGEMENT AND OPERATIONS Since 1993 TIMSS has been directed by the International Study Center at Boston College in the United States Prior to this the study was coordinated by the International Coordinating Center at the University of British Columbia in Canada Although the study was directed centrally by the International Study Center and its staff members implemented various parts of TIMSS important activities also were carried out in centers around the world The data were processed centrally by the IEA Data Processing Center in Hamburg Germany Statistics Canada was responsible for collecting and evaluating the sampling documentation from each country and for calculating the sampling weights The Australian Council for Educational Research conducted the scaling of the achievement data International Study Center 1993 Albert E Beaton International Study Director Michael O Martin Deputy International Study Director Ina V S Mullis Co Deputy International Study Director Eugenio J Gonzalez Director of Operations and Data Analysis Dana L Kelly Research Associate Teresa A Smith Research Associate Cheryl L Flaherty Research Associate Maryellen Harmon Performance Assessment Coordinator Robert Jin Computer Programmer Ce Shen Computer Programmer William J Crowley Fiscal Administrator Thomas M Hoffmann Publications Coordinator Jos Rafael Nieto Senior Production Specialis
139. els to variables FORMAT IDPOP ASG3F INSERT ASSIGNMENT OF FORMATS TO VARIABLES HERE Assign labels to variables LABEL VERSION FILE VERSION INSERT VARIABLE LABELS HERE RECODE MISSING VALUES Select IDSTRAT when 998 IDSTRAT when 999 IDSTRAT otherwise end INSERT RECODES FOR MISSING VALUES HERE 9 6 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES Figure 9 4 CHAPTER 9 Extract from SPSS Control Code for Creating a Student Background SPSS Data Set NOTE Assignment of variable names data list file bsg country l dat VERSION 1 2 INSERT VARIABLE NAME FORMAT AND LOCATION HERE NOTE Assignment of variable labels variable labels VERSION FILE VERSION INSERT VARIABLE LABELS HERE VALUE LABELS IDPOP population 1 population 2 1 2 3 population 3 INSERT VARIABLE LABELS HERE RECODE of sysmis values RECODE IDSTRAT 999 sysmis INSERT RECODES OF MISSING VALUES HERE RECODE of other missing values MISSING value IDSTRAT 998 DEFINE MISSING VALUES HERE SAVE OUTFILE bsg lt country gt sys TIMSS DATABASE USER GUIDE CHAPTER 9 PERFORMING ANALYSES 9 3 Computing JRR Replicate Weights and Sampling Variance Using SPSS and SAS In this section example SAS and SPSS code that may be used to compute the JRR standard errors for means and percentages is described This code is provided in the form of SAS and SPSS macros These ma
140. ent task ITPARTM 1 ITPARTMS5 ITPARTS 1 ITPARTS5 ITPARTG 1 ITPARTG2 See Harmon and Kelly 1996 for a description of the performance assessment administration procedures TIMSS DATABASE USER GUIDE Y 23 CHAPTER 7 DATABASE FILES ITPAS6 Difficulty of performance tasks ITPAS7 Students attitude towards the performance assessment ITPAS8 Attitude of staff towards the performance assessment The codebooks for School Performance Assessment files include a complete list of all of these variables as well as their response option definitions 7 2 3 Coding Reliability Files General Information The coding reliability files contain data that can be used to investigate the reliability of the TIMSS free response item scoring Four coding reliability files for each country are included in the TIMSS International Database e Student Written Assessment Coding Reliability File Population 1 Student Written Assessment Coding Reliability File Population 2 Student Performance Assessment Coding Reliability File Population 1 e Student Performance Assessment Coding Reliability File Population 2 Each of these files contains one record for each student included in the reliability sample for the written assessment or the performance assessment as described in Chapter 4 For each free response item in the written or performance assessment the following three variables are included Original Code Variable cognitive item response codes obt
141. ent variable the item response probability model is written as exp b 0 art PX EA bEI0 v Q exp b 0 art k 1 and a response vector probability model as f x amp 5 0 W 6 amp exp x b0 A amp 3 with W 0 E p exp z b ss 4 where is the set of all possible response vectors 5 3 The Multidimensional Random Coefficients Multinomial Logit Model The multidimensional form of the model is a straightforward extension of the model that assumes that a set of D traits underlie the individuals responses The D latent traits define a D dimensional latent space and the individuals positions in the D dimensional latent space are represented by the vector g 6 0 0 The scoring function of response category k in item i now corresponds to a D by 1 column vector rather than a scalar as in the unidimensional model A response in category k in dimension d of item i is scored ba The scores across D dimensions can be collected into a column vector b bi bis Pup T again be collected into the scoring sub matrix for item i B b b b and then collected into a scoring matrix B B7 B7 B for the whole test If the item parameter vector E and the design matrix A are defined as they were in the unidimensional model the probability of a response in category k of item i is modeled as b 0 a Pr X 1 A B o el y a E exp b 0 a amp
142. entical score but different diagnostic code first digits are the same second digits are different 2 Different score both the first and second digits are different In general the response code data contained in the Original Code Variables are identical to those contained in the Student Written Assessment or Student Performance Assessment files In some cases however the response codes for specific items were recoded after a review of the international item statistics revealed inconsistencies in the original coding guides or showed that the original codes were not functioning as desired see Martin and Mullis 1996 The recoded diagnostic code values were used in computing the achievement scores reflected in the international reports Table 7 11 lists the recodings made to the Population 1 and Population 2 written assessment and performance assessment items These recodes are reflected in the Written Assessment and Performance Assessment data files For the items indicated in Table 7 11 the response codes in the Student Written Assessment or Student Performance Assessment files reflect the recoded values In contrast the Original Code Variables in the coding reliability files contain the original unrecoded response codes This was done so that the coding reliability measure indicated in the Agreement Code Variables was based on the original coding guides used during the free response coding sessions conducted in each country One exception to
143. eran 15875 Iran Israel Pinchas Tamir The Hebrew University Israel Science Teaching Center Jerusalem 91904 Israel Ruth Zuzovsky Tel Aviv University School of Education Ramat Aviv PO Box 39040 Tel Aviv 69978 Israel Japan Masao Miyake Eizo Nagasaki National Institute for Educational Research 6 5 22 Shimomeguro Meguro Ku Tokyo 153 Japan Korea Jingyu Kim Hyung Im National Board of Educational Evaluation Research Division 15 1 Chungdam 2 dong Kangnam ku Seoul 135 102 Korea Kuwait Mansour Hussein Ministry of Education P O Box 7 Safat 13001 Kuwait TIMSS DATABASE USER GUIDE ACKNOWLEDGMENTS Latvia Andrejs Geske University of Latvia Faculty of Education amp Psychology Jurmalas gatve 74 76 Rm 204A Riga LV 1083 Latvia Lithuania Algirdas Zabulionis National Examination Centre Ministry of Education amp Science M Katkaus 44 2006 Vilnius Lithuania Netherlands Wilmad Kuiper Anja Knuver Klaas Bos University of Twente Faculty of Educational Science and Technology Department of Curriculum P O Box 217 7500 AE Enschede Netherlands New Zealand Hans Wagemaker Megan Chamberlain Steve May Robyn Caygill Ministry of Education Research Section Private Box 1666 45 47 Pipitea Street Wellington New Zealand Norway Svein Lie University of Oslo SLS Postboks 1099 Blindern 0316 Oslo 3 Norway Gard Brekke Alf Andersensv 13 3670 Notodden Norway Phil
144. ernational Database using the sampling weights and scores discussed in previous chapters It also provides details on some SPSS and SAS code to conduct such analyses and the results of these analyses Although the analyses presented here are simple in nature they are designed to familiarize the user with the different files and their structure as well as the relevant variables that need to be included in most analyses All the analyses presented here compute the percent of students in specified subgroups the mean achievement in mathematics for each subgroup and the corresponding standard errors for the percent and mean statistics The analyses presented in this chapter based on student and teacher data replicate analyses that are included in the TIMSS mathematics international report Two tables from the international report Mathematics Achievement in the Middle School Years shown in Figure 9 1 and Figure 9 2 are replicated in Examples 1 and 2 in this chapter The user is welcomed to compare the results from these analysis to the tables in the reports and is encouraged to practice analyzing the TIMSS data by trying to replicate some of the tables that are presented in the international reports In our examples we use macros written for SAS and SPSS that can be used to perform any of the analyses that are described below These are general subroutines that can be used for many purposes provided the user has some basic knowledge of the SAS or SPS
145. escriptive Statistics for the International Mathematics Achievement Scores for Population 1 Variable AIMATSCR Lower Grade Upper Grade Standard Deviation Australia 546 69 Austria 559 25 Canada 532 13 Cyprus 502 42 Czech Republic 567 09 England 512 70 Greece 491 90 Hong Kong 586 64 Hungary 548 36 Iceland 473 75 Iran Islamic Rep 428 50 Ireland 549 94 Israel 531 40 Japan 596 83 Korea 610 70 Kuwait 400 16 Latvia LSS 525 38 Netherlands 576 66 New Zealand 498 72 Norway 501 87 Portugal 475 42 Scotland 520 41 Singapore 624 95 Slovenia 552 41 Thailand 490 19 United States 544 57 Minimum Maximum Standard Minimum Maximum SOURCE IEA Third International Mathematics and Science Study TIMSS 1994 95 TIMSS DATABASE USER GUIDE 6 5 CHAPTER 6 ACHIEVEMENT SCORES Table 6 2 Descriptive Statistics for the International Science Achievement Scores for Population 1 Variable AISCISCR Lower Grade Upper Grade Standard x Minimum Maximum Mean Standard Minimum Maximum Deviation Australia 510 24 Austria 504 55 Canada 490 42 Cyprus 414 72 Czech Republic 493 67 England 499 23 Greece 445 87 Hong Kong 481 56 Hungary 464 42 Iceland 435 38 Iran Islamic Rep 356 16 Ireland 479 11 Israel Japan 521 78 Korea 552 92 Kuwait i Latvia LSS 465 25 Netherlands 498 84 New Zealand 473 05 Norway 450 28 Portugal 422 99 Scotland 483 85 Singapore 487 74 Slovenia 486 94 Thailand 432 56 United States 511 23 SOUR
146. et to 1 for all the students in the school Since it was expected that only one classroom would be selected per grade within each school there was no need to compute an adjustment factor for the classroom weight WGTFAC3 Student Weighting Factor This is the inverse of the probability of selection for the individual student within a classroom In cases where an intact classroom was selected the value is set to 1 for all members of the classroom For the user not familiar with the data files included in the TIMSS International Database we recommend reading Chapter 7 before proceeding with sections 3 8 3 9 and 3 10 TIMSS DATABASE USER GUIDE Sl CHAPTER 3 SAMPLING WGTADJ3 Student Weighting Adjustment This is an adjustment applied to the variable WGTFAC3 to account for non participating students in the selected school and or classroom If we were to multiply the variables WGTFAC2 WGTFAC3 and WGTADJ3 and add them up within each school we would obtain an estimate of the number of students within the sampled school The five variables listed above are all used to compute a student s overall sampling weight A two stage sampling design was used in TIMSS schools were first selected from a national list of schools in the second stage classrooms were selected within these schools Some countries used a third stage in which students were selected within classrooms We compute the probability for selecting an individual student as the produ
147. file nway compute k 1 llet seq index cvarl i 9 16 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 7 continued aggregate outfile tmpjckA break cvar2a wgt rwgtl to concat rwgt njkr wgtx rwgtxl to concat rwgtx njkr sum wgt rwgtl to concat rwgt njkr wgtx rwgtxl to concat rwgtx njkr aggregate outfile tmpjckB break cvar2b twgt twgtl to concat twgt njkr sum wgt rwgtl to concat rwgt njkr get file tmpjckA Sort cases by cvar2B save outfile tmpjckA match files file tmpjckA table tmpjckB by cvar2b compute type seq lif thead cvarl head cvar2a then save outfile jackfnl lelse add files file jackfnl file save outfile jackfnl lifend llet cvar2a llet cvar2b tail cvar2a tail cvar2b doend 0 wgtx wgt 100 wgt twgt do if wgt compute mnx compute pct uo v end if compute mnx var O0 compute pct var 0 vector rwgt rwgtl to concat rwgt njkr vector twgt twgtl to concat twgt njkr vector wgtx rwgtxl to concat rwgtx njkr loop i 1 to njkr do if rwgt i gt 0 compute mnx var mnx var mnx wgtx i rwgt i 2 compute pct var pct var pct 100 rwgt i twgt i 2 end if end loop compute mnx se compute pct se Sqrt mnx var Sqrt pct var autorecode type into level execute select
148. for the Evaluation of Educational Achievement IEA Since its inception in 1959 the IEA has sponsored more than 15 studies of cross national achievement in curricular areas such as mathematics science language civics and reading The IEA conducted its First International Mathematics Study FIMS in 1964 and the Second International Mathematics Study SIMS in 1980 82 The First and Second International Science Studies FISS and SISS were carried out in 1970 71 and 1983 84 respectively Since the subjects of mathematics and science are related in many respects and since there is broad interest in many countries in students abilities in both mathematics and science the third studies were conducted together as an integrated effort The number of participating countries the number of grades tested and the simultaneous assessment of mathematics and science has resulted in TIMSS becoming the largest most complex IEA study to date and the largest international study of educational achievement ever undertaken Traditionally IEA studies have systematically worked toward gaining more in depth understanding of how various factors contribute to the overall outcomes of schooling Particular emphasis has been given to refining our understanding of students opportunity to learn as this opportunity becomes successively defined and implemented by curricular and instructional practices In an effort to extend what had been learned from previous studies and
149. given ee R are NOT 24 cm and 18 cm but are consistent with response in part a and a difference consistent with those areas is given instead of the ratio Incorrect Response Focuses exclusively on the ratios of lengths and widths between the given rectangle and the new rectangle No areas are shown Other incorrect Nonresponse 90 Crossed out erased illegible or impossible to interpret 99 BLANK To meet the goal of reliable scoring TIMSS had an ambitious training program The training sessions were designed to assist representatives of the national centers who were then responsible for training personnel in their respective countries to apply the two digit codes reliably TIMSS recommended that scorers be organized into teams of about six headed by a team leader who monitored the progress and reliable use of the codes The team leaders were to continually check and reread the responses coded by their team systematically covering the daily work of each scorer To document information about the within country agreement among scorers TIMSS developed a procedure whereby approximately 10 of the student responses were to be scored independently by two readers As shown in Table 4 1 the within country percentages of exact agreement for both the correctness score and the full two digit diagnostic code revealed a high degree of agreement for the countries that documented the reliability of their coding Unfortunately lack of resour
150. ground File BSGCONTR ASGCONTR Control Files for Teacher Background File s BTMCONTR Mathematics ATGCONTR BTSCONTR Science Control Files for School Background File BCGCONTR ACGCONTR Control Files for Student Teacher Linkage File BLGCONTR ALGCONTR Control Files for Student Written Assessment Reliability File BSRCONTR ASRCONTR Control Files for Student Performance Assessment File BSPCONTR ASPCONTR Control Files for Student Performance Assessment Reliability File BSQCONTR ASQCONTR Control Files for School Performance Assessment File BCTCONTR ACTCONTR I Jacknife Statistics Program Files Ill Scoring Program Files Scoring Program Files for Written Assessment Cognitive Items BSASCORE ASASCORE Scoring Program for Performance Assessment Items BSPSCORE ASPSCORE Chapter 9 further describes of the SAS and SPSS program files and how they are applied through the use of specific example analyses using the TIMSS student teacher and school data files 7 38 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 7 5 Data Almanacs Data almanacs are included for all student teacher and school background variables The data almanacs are text files that display unweighted summary statistics by grade for each participating country on each variable included in the background questionnaires administered to students teachers and school administrators or principals The data almanac files corresponding to e
151. gt 1 DAT School Performance Assessment File BCT lt Country gt 1 DAT ACT lt Country gt 1 DAT The data files for each country may be identified by a three digit alphabetic country abbreviation designated as lt country gt in the general file names shown in Table 7 1 The three digit abbreviations used for each TIMSS country and the available files are listed in Table 7 2 along with the numeric code values used in the country identification variable contained in the background data files see the following section discussing identification variables 7 2 TIMSS DATABASE USER GUIDE DATABASE FILES C HAGPT E RS Z Table 7 2 Country Identification and Inclusion Status in Population 1 and Population 2 Data Files ts Inclusion in Population 1 Data Files Inclusion in Population 2 Data Files Abbreviation Numeric Code Que uoo S wn w sjou sss sonw Sia on w Australia Austria Belgium FI Belgium Fr Bulgaria Canada Colombia Cyprus Czech Republic Denmark England France Germany Greece Hong Kong Hungary Iceland Iran Islamic Rep Ireland Israel Japan Korea Kuwait Latvia Lithuania Netherlands New Zealand Norway Philippines Portugal Romania Russian Federation Scotland Singapore Slovak Republic Slovenia South Africa Spain Sweden Switzerland Thailand APARAR United States ViViV io Vivi iV S Q uw gt SN Ss ow gt Ss Sjon gt S S for gt s JQ 0 gt ao S slou Ssa w gt
152. he inclusion status of each student in the reliability file containing double coded free response items see Section 7 2 3 The tracking information available regarding students gender ITSEX or date of birth ITBIRTHM ITBIRTHY was compared to the students reports obtained from the background questionnaires If the student data were missing data in the corresponding tracking variables were substituted Also if a student s gender or date of birth reported in the background variables differed from the tracking information in Population 2 the tracking information was replaced by the background data and in Population 1 the background data were replaced by the tracking information Variables Included in the Teacher Background Files ILCLASS1 ILCLASS2 ILCLASS3 Linking variables containing class identification numbers for classes to which teachers are linked ILMATH ILSCI Sampling status of each teacher as a mathematics teacher or a science teacher ITQUEST Questionnaire status indicating whether the teacher completed the mathematics or science teacher questionnaire ITPART Participation status variable indicating whether the student participated in questionnaire session TIMSS DATABASE USER GUIDE 7 9 CHAPTER 7 DATABASE FILES Variables in the School Background Files ITREPLAC Replacement status variable indicating whether each school is an original sampled school or a replacement school ILREPLAC School identification number of origi
153. he international results and quality assurance Several important TIMSS functions including test and questionnaire development translation checking sampling consultations data processing and data analysis were conducted by centers around the world under the direction of the TIMSS International Study Center The IEA Data Processing Center DPC located in Hamburg Germany was responsible for checking and processing all TIMSS data and for constructing the international database The DPC played a major role in developing and documenting the TIMSS field operations procedures Statistics Canada located in Ottawa Canada was responsible for advising National Research Coordinators NRCs on their sampling plans for monitoring progress in all aspects of sampling and for the computation of sampling weights The Australian Council for Educational Research ACER located in Melbourne Australia has participated in the development of the achievement tests has conducted psychometric analyses of field trial data and was responsible for the development of scaling software and for scaling the achievement test data The International Coordinating Center ICC in Vancouver Canada was responsible for international project coordination prior to the establishment of the International Study Center in August 1993 Since then the ICC has provided support to the International Study Center and in particular has managed translation verification in the achievemen
154. he performance assessment sample is a sub sample of the written assessment sample 8 3 Computing the JRR Replicate Weights Having assigned the schools to zones if it is desired to use standard statistical software such as SAS or SPSS for sampling variance estimation a convenient way to proceed is to construct a set of replicate weights for each pseudo replicate sample In TIMSS the schools in the sample were assigned in pairs to one of the 75 zones indicated by the variable JKZONE and within each zone the pair members were randomly assigned an indicator u represented by the variable JKINDIC coded randomly to 1 or 0 so that one of the members of each pair had values of 1 on the variable u and the remaining one a value of 0 This indicator determined whether the weights for the elements in the school in this zone was to be doubled or zeroed The replicate weight J for the elements in a school assigned to zone h is computed as the product of k times their overall sampling weight where k could take values of zero one or two depending on if the case was not to contribute be left unchanged or have it count double for the computation of the statistic of interest In TIMSS the replicate weights are not permanent variables but are created temporarily by the sampling variance estimation program as a useful computing device An example program which makes use of replicate weights in computing JRR estimates is provided in the next chapter
155. he scale for the standardized scale scores has a mean of 500 and a standard deviation of 100 the international student mean is not exactly 500 This is because one or two countries were not included in the computation of 0 j Since the scaling of their data was not completed Since the scale scores were actually plausible values drawn randomly for individual students from the posterior distribution of achievement it was possible to obtain scores that were unusually high or low If a transformed score was below 50 the score was recoded to 50 therefore making 50 the lowest score on the transformed scale This happened in very few cases across the sample The highest transformed scores did not exceed 1 000 points so the transformed values were left untouched at the upper end of the distribution Five plausible values were drawn for each student Each of the plausible values was transformed onto the international scale The variation between results computed using each separate value reflects the error due to imputation In the TIMSS international reports plausible values were used only to compute total scores in mathematics and science The reliability of these total scores as reflected in the intercorrelations between the five plausible values was found to be sufficiently high that the imputation error component could be ignored For the purpose of reporting international achievement therefore only the first plausible value was used However all
156. he student has This weight should be used whenever the user wants to obtain estimates regarding students and their mathematics and science teachers combined The Student Teacher Linkage file also includes variables that indicate the number of mathematics teachers the number of science teachers and the number of mathematics and science teachers the student has 3 10 Weight Variables Included in the School Data Files The TIMSS samples are samples of students within countries Although they are made up of a sample of schools within the countries the samples of schools are selected so that the sampling of students rather than the sampling of schools is optimized In other words the samples are selected to make statements about the students in the country rather than about the schools in the country To this effect several weight variables are included in the school files These variables are listed below WGTFACI School Weighting Factor This variable corresponds to the inverse of the probability of selection for the school where the student is enrolled WGTADJ1 School Weighting Adjustment This is an adjustment that is applied to WGTFAC1 to account for non participating schools in the sample If we were to multiply WGTFACI by WGTADJ1 we would obtain the sampling weight for the school adjusted for non participation SCHWGT School level Weight The school sampling weight is the inverse of the probability of selection for the school multip
157. he students within each country were selected using probability sampling This means that the probability of each student being selected as part of the sample is known The inverse of this selection probability is the sampling weight In a properly selected and weighted sample the sum of the weights for the sample approximates the size of the population As is the case in TIMSS the sum of the sampling weights for a sample is an estimate of the size of the population of students within the country in the sampled grades The sampling weights must be used whenever population estimates are required The use of the appropriate sampling weights ensures that the different subgroups that constitute the sample are properly and proportionally represented in the computation of population estimates Tables 3 3 and 3 4 present the sample sizes and the estimate of the population size sum of the weights for each participating country in Populations 1 and 2 respectively Sampling weights can only be computed when the probability of selection is known for all students Baul TIMSS DATABASE USER GUIDE TIMSS SAMPLING Table 3 3 Sample Information for TIMSS Population 1 Countries COUNTRY Australia Austria Canada Cyprus Czech Republic England Greece Hong Kong Hungary Iceland Iran Islamic Rep Ireland Israel Japan Korea Kuwait Latvia LSS Netherlands New Zealand Norway Portugal Scotland Singapore Slovenia Thailand Unite
158. hen computing a statistic t from the sample for a country the formula for the JRR variance estimate of the statistic f is then given by the following equation Vas ij po s l where H is the number of pairs in the entire sample for the country The term f S corresponds to the statistic computed for the whole sample computed with any specific weights that may have been used to compensate for the unequal probability of selection of the different elements in the sample or any other post stratification weight The element t Jp denotes the same statistic using the h jackknife replicate computed for all cases except those in the h stratum of the sample removing all cases associated with one of the randomly TIMSS DATABASE USER GUIDE 8 1 CHAPTER 8 SAMPLING VARIANCE selected units of the pair within the p stratum and including twice the elements associated with the other unit in the h stratum In practice this is effectively accomplished by recoding to zero the weights for the cases of the element of the pair to be excluded from the replication and multiplying by two the weights of the remaining element within the h pair This results in a set of H replicate weights which may be used in computing the JRR variance estimate As we can see from the above formula the computation of the JRR variance estimate for any statistic from the TIMSS database requires the computation of any statistic up to 76 times for any given country
159. his documentation was used by the International Study Center in consultation with Statistics Canada the Sampling Referee and the Technical Advisory Committee to evaluate the quality of the national samples For Populations 1 and 2 the International Desired Populations for all countries were defined as follows Population 1 All students enrolled in the two adjacent grades that contain the largest proportion of 9 year olds at the time of testing Population 2 All students enrolled in the two adjacent grades that contain the largest proportion of 13 year olds at the time of testing Tables 3 1 and 3 2 show the grades tested in each country that participated in TIMSS This information is captured in the variable IDGRADE in the student data files TIMSS DATABASE USER GUIDE 9 24 CHAPTER 3 SAMPLING Table 3 1 Grades Tested in TIMSS Population 1 Lower Grade Upper Grade Years of Formal Years of Formal Count Couniry s Name for Schooling Includin Country s Name for Schooling Includin y Lower Grade g g Upper Grade 9 9 Upper Grade Lower Grade Australia 3or4 3or4 40r5 4o0r5 Austria 3 3 4 4 Canada 3 3 4 4 Cyprus 3 3 4 4 Czech Republic 3 3 4 4 England Year 4 4 Year 5 5 Greece 3 3 4 4 Hong Kong Primary 3 3 Primary 4 4 Hungary 3 3 4 4 Iceland 3 3 4 4 Iran Islamic Rep 3 3 4 4 Ireland 3rd Class 3 4th Class 4 Israel 4 4 Japan 3 3 4 4 Korea 3rd Grade 3 4th Grade 4 Kuwait 5 5 Latvia 3 3 4 4 3 Netherlands 5 3 6
160. if level 1 save outfile final lenddefine The user needs to know some basic SPSS macro language in order to use the macro It needs to be first included in the program file where it is going to be used If the user is operating in batch mode then the macro needs to be called in every batch If the user is using SPSS interactively then the macro needs to be called once at the beginning of the session and it will remain active throughout the session If the session is terminated and restarted at a later time the macro needs to be called once again Once the macro is included in a specific session the TIMSS DATABASE USER GUIDE 9 17 CHAPTER 9 PERFORMING ANALYSES word JACK should not be used within that program because doing so will invoke the macro The macro is included in the program file where it will be used by issuing the following command under SPSS include directory location jack sps where directory location points to the specific drive and directory where the macro JACK SPS can be found The macro requires that several argument be submitted when it is invoked These parameters are WGT The sampling weight to be used in the analysis Generally TOTWGT when using the student files or MATWGT SCIWGT or TCHWGT when using the teacher files JKZ The variable that captures the assignment of the student to a particular sampling zone The name of this variable in all TIMSS files is JKZONE JKI The variable
161. igned different values depending on the field width of the variable and the variable type Omitted Response Codes 9 99 999 Omitted response codes are used for questions items that a student teacher or school principal should have answered but did not answer These are indicated as missing in the codebooks For questionnaire data no differentiation has been made between no answer and invalid answers such as checking two or more response options in a categorical question or unreadable or uninterpretable responses to open ended questions In a few cases data received from a country in an invalid or inconsistent way were also recoded to missing For cognitive items an Omitted Response Code was given only in cases in which the item was left blank a special code was used for invalid answers as described below The specific Omitted Response Code value given depends on the number of valid codes available for each item For Identification Tracking or Background Questionnaire Items An Omitted Response Code value of 9 is used for categorical items with 7 or less valid response options for categorical items with more than 7 categories a code value of 99 is used For open ended background questionnaire items or other items containing non categorical values the omit code is the next available 9 code greater than the highest response in the defined valid range Background questionnaire item values outside the valid ranges were recoded to mis
162. igure 9 11 CHAPTER 9 Extract of SAS Computer Output for Performing Analyses with Student Level Variables EXAMPLE 1 COUNTRY ID Australia GRADE Eighth Grade BSBGDAY1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 1651 52718 94 538 883 5 99951 23 5254 0 91886 1 to 2 hrs 2943 92637 64 538 578 4 12722 41 3388 0 84451 3 to 5 hrs 1880 59663 57 527 545 3 80272 26 6244 0 84064 More than 5 hrs 563 19073 65 487 008 5 47335 8 5115 0 61293 COUNTRY ID Austria GRADE Eighth Grade BSBGDAY1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 673 21112 51 539 820 5 42470 25 2880 1 39738 1 to 2 hrs 1402 44420 93 545 816 4 20850 53 2063 1 06455 3 to 5 hrs 469 14158 96 539 469 5 19961 16 9592 0 97982 More than 5 hrs 118 3795 76 496 845 8 63972 4 5465 0 57168 COUNTRY ID Belgium Fl GRADE Eighth Grade BSBGDAY1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 757 18055 42 579 948 6 6674 24 3543 1 24276 1 to 2 hrs 1517 38846 19 575 118 6 1529 52 3983 1 17633 3 to 5 hrs 478 13808 20 534 656 7 1230 18 6254 1 03135 More than 5 hrs 112 3426 62 513 901 12 0735 4 6220 0 53665 COUNTRY ID Belgium Fr GRADE Eighth Grade BSBGDAY1 N TOTWGT MNX MNX_SE PCT PCT_SE Less than 1 Hr 887 19212 63 536 028 4 20418 33 3444 1 26106 1 to 2 hrs 1124 25151 10 535 669 4 87625 43 6509 1 77399 3 to 5 hrs 394 9549 30 522 292 3 98052 16 5732 1 32403 More than 5 hrs 119 3705 76 445 040 9 01330 6 4315 0 98084 COUNTRY ID United States GRADE Eighth Grade BSBGDA
163. ional Database contains a SAS macro program called JACK SAS This macro can be used to compute weighted percents and means within categories defined by variables specified by the user and the JRR standard error estimate for these statistics The control code for the macro JACK SAS is presented in Figure 9 5 9 8 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES Figure 9 5 CHAPTER SAS Macro for Computing Mean and Percents with Corresponding JRR Standard Errors JACK SAS macro jack wgt jkz jki njkr cvar dvar fname let i 1 Start counter at 1 do while length scan amp cvar amp i glet i eval amp i 1 end Slet niv eval amp i 1 Diminish by 1 at the end 1et cvar2 substr amp cvar 1 eval slength amp cvar length scan amp cvar amp niv Now create the data file with the replicate weights data a set amp fname keep amp wgt amp jkz amp jki amp cvar amp dvar array rwt rwtl rwt amp njkr Replicate Weight 7 array wtx wtxl wtx amp njkr Weighted X n k 1 computes a constant do i 1 to amp njkr if amp jkz lt gt i then rwt i amp wgt 1 if amp jkz i amp amp jki 1 then rwt i amp wgt 2 if amp jkz i amp amp jki 0 then rwt i amp wgt 0 wtx i rwt i amp dvar n end N 1 wtx0 amp wgt amp dvar for the sum of X proc summary data a missing class k amp cvar var N amp wgt rwtl rwt amp njkr wtx0 wtxl wtx a
164. ional Versions of the Background Questionnaires Population 2 This supplement contains the international versions of the student teacher and school background questionnaires for Population 2 and tables that map each question to a variable in the database Supplement 3 Documentation of National Adaptations of the International Background Questionnaire Items This supplement contains documentation of national adaptations of the international versions of the student teacher and school questionnaire items This documentation provides users with a guide to the availability of internationally comparable data for secondary analyses Supplement 4 Documentation of Derived Variables Based on Student and Teacher Background Questionnaire Items The TIMSS international reports included a number of variables derived from questions in the student and teacher questionnaires These derived variables are included in the database and are documented in this supplement to the User Guide TIMSS DATABASE USER GUIDE l 7 CHAPTER 1 IN TRODUCTION 1 6 Management and Operations of TIMSS TIMSS is managed by the International Study Center at Boston College in the United States The TIMSS International Study Center was responsible for supervising all aspects of the design and implementation of the study at the international level including development and design of the study data collection instruments and operational procedures data analysis reporting t
165. ippines Milagros Ibe University of the Philippines Institute for Science and Mathematics Education Development Diliman Quezon City Philippines Ester Ogena Science Education Institute Department of Science and Technology Bicutan Taquig Metro Manila 1604 Philippines Portugal Gertrudes Amaro Ministerio da Educacao Instituto de Inova o Educacional Rua Artilharia Um 105 1070 Lisboa Portugal Romania Gabriela Noveanu Institute for Educational Sciences Evaluation and Forecasting Division Str Stirbei Voda 37 70732 Bucharest Romania Russian Federation Galina Kovalyova The Russian Academy of Education Institute of General Secondary School UL Pogodinskaya 8 Moscow 119905 Russian Federation Scotland Brian Semple Scottish Office Education amp Industry Department Victoria Quay Edinburgh E86 6QQ Scotland Singapore Wong Cheow Cher Chan Siew Eng Research and Evaluation Branch Block A Belvedere Building Ministry of Education Kay Siang Road Singapore 248922 TIMSS DATABASE USER GUIDE Slovak Republic Maria Berova Vladimir Burjan SPU National Institute for Education Pluhova 8 P O Box 26 830 00 Bratislava Slovak Republic Slovenia Marjan Setinc Center for IEA Studies Educational Research Institute Gerbiceva 62 P O Box 2976 61111 Ljubljana Slovenia South Africa Derek Gray Human Sciences Research Council 134 Pretorius Street Private Bag X41 Pretoria 0001 South
166. ith our analysis without any modifications We then proceed to read from the SPSS system file or SAS data set that contains this variable the international mathematics achievement score BIMATSCR the sampling weight for the student TOTWGT the variables that contain the jackknife replication information JKZONE and JKINDIC the variable that will be used to select the eighth graders from the data file IDGRADER and the variable containing the country identification code IDCNTRY In this analysis we will use the data for all countries in the database although the exact same steps need to be taken if the user wants to examine these variables within a single country or for a select group of countries The SAS code used to perform this analysis is presented in Figure 9 9 and the SPSS code is presented in Figure 9 10 A selection of the results obtained from these programs are displayed in Figure 9 11 and Figure 9 12 We have included as part of the programs the corresponding value labels and format statements so that the different categories or groups are labeled appropriately Notice that one of the steps in each of these programs was to select only those students in the eighth grade who chose one of the five options presented in the question and we recoded the variable so that there would be one category for those who reported watching television or video less than one hour per day Note also in this analysis that although we have used th
167. le Because of the clustering effect in most TIMSS samples it may also be desireable to apply a correction factor such as a design effect to the HOUWGT variable 3 9 Weight Variables Included in the Student Teacher Linkage Files The individual student sampling weights generally should be used when the user wants to obtain estimates at the student level The exception is when student and teacher data are to be analyzed together In this case a separate set of weights have been computed to account for the fact that a student could have more than one mathematics or science teacher These weight variables are included in the Student Teacher Linkage file and are listed below In Israel and Kuwait only one grade was tested and the SENWGT adds to 500 TIMSS DATABASE USER GUIDE Be Z CHAPTER 3 SAMPLING MATWGT This weight is computed by dividing the sampling weight for the student by the number of mathematics teachers that the student has This weight should be used whenever the user wants to obtain estimates regarding students and their mathematics teachers SCIWGT This weight is computed by dividing the sampling weight for the student by the number of science teachers that the student has This weight should be used whenever the user wants to obtain estimates regarding students and their science teachers TCHWGT This weight is computed by dividing the sampling weight for the student by the number of mathematics and science teachers that t
168. le instead of B background variable For example BSDGEDUP is a derived variable including Population 2 students reports of parents education Additional information about the methods used to report student and teacher background data may be found in Martin and Kelly 1996 TIMSS DATABASE USER GUIDE 7 13 CHAPTER 7 DATABASE FILES Information About How the Reported Derived Variables Were Computed Descriptions of each of the derived variables based on student and teacher background data their associated international table figure references and analysis notes indicating how they were computed using the associated student or teacher source variables are provided in Supplement 4 for the following four types of derived variables Section 1 Population 1 Derived variables based on student background data Section 2 Population 1 Derived variables based on teacher background data Section 3 Population 2 Derived variables based on student background data Section 4 Population 2 Derived variables based on teacher background data Each section of Supplement 4 is presented in alphabetical order by derived variable name The documentation for derived variables reflects the rules for construction from the internationally defined background questions Due to national adaptations of some items in the background questionnaires some countries have been omitted or handled somewhat differently for certain report variables Documentation descri
169. lets At Population 2 students were given 90 minutes The organization of the clusters and booklets is summarized below first for Population 1 and then for Population 2 At Population 1 the clusters were either 9 or 10 minutes in length as shown in Table 2 2 The core cluster labeled A composed of five mathematics and five science multiple choice items was included in all booklets Focus clusters labeled B through H appeared in at least three booklets so that the items were answered by a relatively large fraction three eighths of the student sample in each country The breadth clusters largely containing multiple choice items appeared in only one booklet The breadth clusters are labeled I through M for mathematics and N through R for science The free response clusters were each assigned to two booklets so that items statistics of reasonable accuracy would be available These clusters are labeled S through V for mathematics and W through Z for science The organization of the test design is fully documented in Adams and Gonzalez 1996 2 4 TIMSS DATABASE USER GUIDE INSTRUMENTS CHAPTER 2 Table 2 2 Distribution of Item Types Across Clusters Population 1 Number Mathematics Items Number Science Items Cluster Type Cluster Label Multiple Short Extended Multiple Short Extended Choice Answer Response Choice Answer Response gt Core 10 minutes Focus 9 minutes Breadth Mathematics 9 minutes 5 4 5
170. lied by its corresponding adjustment factor It is computed as the product of WGTADJI and WGTFACI Although this weight variable can be used to estimate the number of schools with certain characteristics it is important to keep in mind that the sample selected for TIMSS is a good sample of students but not necessarily an optimal sample of schools Schools are 3 18 TIMSS DATABASE USER GUIDE SAMPLING C HAPTER 3 selected with probability proportional to their size so it is expected that there is a greater number of large schools in the sample STOTWGTL Sum of the Student Weights for the Lower Grade STOSWGTU Sum of the Student Weights for the Upper Grade STOTWGTE Sum of the Student Weights for the Extra Grade These variables are the sum of the weights of the students in the corresponding grade level within the school If there are no students at this grade level then the variable is set to zero These variables can be used to conduct analyses on questions like How many students attend schools that have a certain characteristic Although weight variables have been included in the school files there is no information in these files regarding the sampling zone in which the school is included see Chapter 8 for discussion of sampling zones in the estimation of sampling variance If the user is interested in computing proper standard errors of the population estimates then the information in the school file should be merged with the student file
171. lyses with the Population 1 data ASASCORE SAS BSASCORE SAS ASPSCORE SAS BSPSCORE SAS ASASCORE SPS BSASCORE SPS ASPSCORE SPS BSPSCORE SPS These files contain control code in SAS and SPSS that can be used to convert the response codes to the cognitive items discussed in Chapter 7 to their corresponding correctness score levels The use of these programs is described later in this chapter The files beginning with the letters ASA and BSA can be used to recode the items from the written assessment Those beginning with the letter ASP and BSP can be used to recode the items from the performance assessment 9 2 Creating SAS Data Sets and SPSS System Files The CD contains SAS and SPSS control code to read each one of the ASCII data files and create a SAS data set or an SPSS system file An extract of the main sections of the control code for reading the Student Background data file using SAS is presented in Figure 9 3 and for creating the file in SPSS in Figure 9 4 Each of these files contain information on the location for each variable in the file its format a descriptive label for each variable and their categories in the case of categorical variables and code for handling missing data The control and data files have been created to facilitate access of the data on a country by country basis The command lines in the control files should be edited to produce programs that will create SAS or SPSS files for any specified country While mos
172. mp njkr output out b sum run proc summary data b var k type output out maxtype max k maxtype data b merge b maxtype by k maxtype2 maxtype 1 Select the record which has the totals for amp cvar2 and save to a file data type2 keep k amp cvar2 t2wt t2rwtl t2rwt amp njkr set b where type maxtype2 array rwt rwtl rwt amp njkr array t2rwt t2rwtl t2rwt amp njkr t2wt amp wgt do i 1 to amp njkr t2rwt i rwt i end sees 9 TIMSS DATABASE USER GUIDE CHAPTER 9 PERFORMING ANALYSES Figure 9 5 continued proc sort data b out d by k amp cvar2 where type maxtype data e drop t2wt t2rwtl t2rwt amp njkr merge d type2 by k amp cvar2 twt t2wt array rwt trwtl trwt amp njkr array t2rwt t2rwtl t2rwt amp njkr do i 1 to amp njkr rwt i t2rwt i end data final keep N amp cvar mnx mnx se pct pct se amp wgt set e mnx wtx0 amp wgt pet 100 amp wgt twt array wtx wtxl wtx amp njkr array rwt rwtl rwt amp njkr array rmn rmnl rmn amp njkr array trwt trwtl trwt amp njkr array rp rpl rp amp njkr H mn jvar 0 p jvar 0 n jvar 0 do i 1 to amp njkr if rwt i gt 0 then rmn i wtx i rwt i Compute the mean for each replicate else rmn i A rp i 100 rwt i trwt i Compute the proportion of each replicate if rmn i lt gt then mn jvar mn jvar rmn i mnx 2 Compu
173. n and their views on mathematics and science The heads of schools responded to questions about school staffing and resources mathematics and science course offerings and support for teachers In addition a volume was compiled that presents descriptions of the educational systems of the participating countries Robitaille 1997 To measure the attained curriculum TIMSS tested more than half a million students in mathematics and science at three separate populations Population 1 Students enrolled in the two adjacent grades that contained the largest proportion of 9 year old students at the time of testing third and fourth grade students in most countries Population 2 Students enrolled in the two adjacent grades that contained the largest proportion of 13 year old students at the time of testing seventh and eighth grade students in most countries Population 3 Students in their final year of secondary education As an additional option countries could test two special subgroups of these students students taking advanced courses in mathematics and students taking courses in physics Countries participating in the study were required to administer tests to the students in the two grades at Population 2 but could choose whether or not to participate at the other levels In many countries subsamples of students in the upper grades of Populations 1 and 2 also participated in a performance assessment The data collected from the as
174. n coefficients For example Y could be constituted of student variables such as gender socio economic status or major Then the population model for student n becomes T 0 Y B E 10 n where we assume that the E are independently and identically normally distributed with mean zero and variance s so that 10 is equivalent to 6 78 6 Yb 11 f 0 Y b m 1207 expl a normal distribution with mean y7 D and variance s If 11 is used as the population model then the parameters to be estimated are b s and x The TIMSS scaling model takes the generalization one step further by applying it to the vector valued 0 rather than the scalar valued 0 resulting in the multivariate population model A O Wy 2x x evw 59 1W x 0 m 12 where y is a u x d matrix of regression coefficients 2 is a d x d variance covariance matrix and W isa u x 1 vector of fixed variables If 12 is used as the population model then the parameters to be estimated are y X and x In TIMSS we refer to the W variables as conditioning variables 5 4 TIMSS DATABASE USER GUIDE SCALING CHAPTER 5 5 5 Estimation The ConQuest software uses maximum likelihood methods to provide estimates of y X and x Combining the conditional item response model 6 and the population model 12 we obtain the unconditional or marginal response model f x amp vE ffo810 0 72 d0 13 o and it follows that
175. n to replication may be influenced by unreliability in the achievement measures and considered to be suspect In the TIMSS international reports the reliability of the achievement measures as reflected in the inter correlations between the five plausible values was found to be sufficiently high that the imputation error could be ignored For the purpose of reporting international achievement therefore only the first plausible value was used However all five values are provided in the International Database for use by other analysts The plausible values are included in the Student Background data files and in the Student Teacher Linkage files TIMSS DATABASE USER GUIDE 6 3 CHAPTER 6 ACHIEVEMENT SCORES AIMATEAP International Mathematics Score EAP Population 1 AISCIEAP International Science Score EAP Population 1 BIMATEAP International Mathematics Score EAP Population 2 BISCIEAP International Science Score EAP Population 2 International expected a posteriori EAP scores are the average of the distribution from which the corresponding plausible values are drawn Although the average of an individual s distribution of plausible values may be a better estimate of the individual s proficiency than a single plausible value in general it will not produce consistent population estimates or estimates of their error variance The EAP scores can be used if individual scores are necessary but should not be used to obtain p
176. nal school replaced by each replacement school ITPART Participation status variable indicating whether each school participated and returned a questionnaire 7 2 1 7 International Background Variables General Information International background variables obtained from the student teacher and school questionnaires are provided in the corresponding background data files In general the background variables are provided for all countries where the data are considered internationally comparable The assessment of international comparability for background variables was based on information provided by NRCs regarding any national adaptations of the background questionnaire items In a few cases some slightly modified specific country options were retained in the international variables Additional national variables not included in the international version of the questionnaire are not contained in the international background files For a description of the information obtained from the international student teacher and school background questionnaire items see Chapter 2 Copies of the international versions of the questionnaires are provided in Supplements 1 and 2 More information regarding how national adaptations of background questionnaire items were handled in creating the TIMSS background data files is provided in Supplement 3 International Background Variable Values The values for the background variables are either categorical opti
177. nd IDSCHOOL which will allow us to merge the student data to the school data We also select from this variable the international mathematics achievement score BIMATSCR the sampling weight for the student TOTWGT the variables that contain the jackknife replication information JKZONE and JKINDIC and the variable that will be used to select the eighth graders from the data file IDGRADER We then proceed to merge the school information with the student information using the variables IDCNTRY and IDSCHOOL as merge variables and then use the macro JACK to obtain the corresponding percents of students within each group and their mean achievement in mathematics The computer code used to run this analysis in SAS and SPSS can be found in Figure 9 17 and Figure 9 18 and the results are shown in Figure 9 19 and Figure 9 20 9 34 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 Figure 9 17 SAS Control Statements for Performing Analyses with School Level Variables EXAMPLE3 SAS options nocenter Read the variables from the school file and sort them by the merge variables data school set bcgalll keep idcntry idschool bcbgcomm proc sort data school by idcntry idschool Read the variables from the student file and sort them by the merge variables data student set bsgalll keep idcntry idschool idgrader jkindic jkzone totwgt bimatscr proc sort data student by idcntry idschool Merge the student and
178. nd girls reported in SCQ1 16K SCQ2 17K 7 14 TIMSS DATABASE USER GUIDE DATABASE FILES C E AUP TEE ORS X ACDGUTRT BCDGUTRT Total students repeating upper grade sum of boys and girls reported in SCQ1 16L SCQ2 17L ACDMUTER BCDMUTER Total upper grade students studying mathematics sum of boys and girls reported in SCQ1 160 SCQ2 170 ACDSUTER BCDSUTER Total lower grade students studying science sum of boys and girls reported in SCQ1 16P SCQ2 17P 7 2 1 9 Sampling Variables The following variables related to the samples selected in each country are included in the student and school background files for use in obtaining the national population estimates presented in the international reports TOTWGT Total student weight HOUWGT House weight SENWGT Senate weight WGTFACI School weighting factor WGTADII School weighting adjustment WGTFAC2 Class weighting factor WGTFAC3 Student weighting factor WGTADJ3 Student weighting adjustment JKZONE Jackknife zone JKINDIC Replicate code WGTFACI School weighting factor WGTADII School weighting adjustment SCHWGT School level weight STOTWGTL Sum of student weights lower grade STOTWGTU Sum of student weights upper grade STOTWGTE Sum of student weights extra grade Detailed descriptions of the sampling weights and jackknife variables and how they are used in computing the population estimates and standard errors presented in the international reports are described in Chapter 3 and Chapter
179. ns and proportionality Measurement estimation and number sense Data representation analysis and probability Geometry Patterns relations and functions Science Earth science Life science Physical science Environmental issues and the nature of science Population 2 Fractions and number sense Geometry Algebra Data representation analysis and probability Measurement Proportionality Earth science Life science Physics Chemistry Environmental issues and the nature of science 2 3 The TIMSS Items The task of putting together the achievement item pools for the different TIMSS student populations was immense and took more than three years to complete Developing the TIMSS achievement tests necessitated building international consensus among NRCs their national committees mathematics and science experts and measurement specialists All NRCs worked to ensure that the items used in the tests were appropriate for their students and reflected their countries curriculum TIMSS DATABASE USER GUIDE CHAPTER 2 INSTRUMENTS Different types of achievement items were included in the item pools for TIMSS The multiple choice items consisted of a stem and either four or five answer choices In the instructions at the front of the test booklets students were encouraged to choose the answer they think is best when they were unsure The instructions do not suggest or imply that students should
180. nted by assigning a weight of 1 to all students in the sample for the Philippines 3 8 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER 3 Figure 3 2 Countries Grouped for Reporting of Achievement According to Compliance with Guidelines for Sample Implementation and Participation Rates Population 1 Written Assessment Fourth Grade Third Grade Countries satisfying guidelines for sample participation rates grade selection and sampling procedures Canada Norway Canada Norway Cyprus Portugal Cyprus Portugal Czech Republic Scotland Czech Republic Scotland t England Singapore t England Singapore Greece United States Greece United States Hong Kong Hong Kong Iceland Iceland Iran Islamic Rep Iran Islamic Rep Ireland Ireland Japan Japan Korea Korea New Zealand New Zealand Countries not satisfying guidelines for sample participation Australia Australia Austria Austria Latvia LSS Latvia LSS Netherlands Netherlands Scotland Countries not meeting age grade specifications high percentage of older students Slovenia Slovenia Countries with unapproved sampling procedures at the classroom level Hungary Hungary Countries with unapproved sampling procedures at classroom level and not meeting other guidelines Israel Kuwait Thailand Thailand Met guidelines for sample participation rates only after replacement schools were included National Desired Population does not cover all of International Desired Population Becaus
181. ntroduction This chapter describes the content and format of the TIMSS International Database for Population and Population 2 This chapter is organized in five major sections corresponding to the types of files included in the database Within each section the contents of the files are described These file types are Data Files e Codebook Files e Program Files e Data Almanacs e Test Curriculum Matching Analysis Files The TIMSS international Data Files reflect the result of an extensive series of data management and quality control steps taken to insure the international comparability quality accuracy and general utility of the database in order to provide a strong foundation for secondary analyses As part of the international data files all variables derived for reporting in the international reports are included In addition analysis notes are provided for all reporting variables allowing users to replicate these computations These analysis notes are included in Supplement 4 of this User Guide Also included in the database are Codebook Files These specifically document the format of all variables in each of the data files Several Program Files are provided for use in secondary analyses including Data Access Control files for converting the raw data files provided in ASCII format into SAS or SPSS files macro programs for computing statistics using the jackknife repeated replication method discussed in Chapter 8 and mac
182. ntry to contribute proportionally to its population size with the large countries counting more than small countries Although this might be desirable in some circumstances e g when computing the 75th percentile for mathematics achievement for students around the world this is not usually the case A key property of the sampling weights is that the same population estimates for means and proportions will be obtained as long as we use a weight variable proportional to the original weights TOTWGT For example we could take the sampling weights for a large country and divide them by a constant to make them smaller We could also take the weights of a smaller country and multiply them by a constant to make them bigger Regardless of which constant is used within a country the weighted estimates obtained from each of these proportional transformations of the weights will be exactly the same To this effect two other weight variables are computed and included in the student data files Each of these is computed for a specific purpose and will yield exactly the same results within each country but will have some desirable properties when estimates across countries are computed or significance tests performed 3 2 196 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER 3 SENWGT Senate Weight This variable is computed as TOTWGT po TOTWGT within each country The transformation of the weights will be different within each country but in the end
183. nts teachers or schools and to link cases between the different data files The identification variables have prefixes of ID and are listed below Variables Included in Student Teacher and School Background Files IDCNTRY Three digit country identification code see Table 7 2 for list of country codes This variable should always be used as one of the link variables whenever files will be linked across countries IDPOP Identifies the population 1 or 2 IDSTRAT This variable identifies the sampling stratum within the country This variable was used in some but not all countries to identify specific strata within the population for that country IDSCHOOL Identification number that uniquely identifies the school within each country The codes for the school are not unique across countries Schools can be uniquely identified only by the IDCNTRY and IDSCHOOL combination Additional Variables Included in the Student Background Files IDSTUD Identification number that uniquely identifies each student in the country sampled The variable IDSTUD is a hierarchical identification number It is formed by the combination of the variables IDSCHOOL and IDCLASS followed by a two digit sequential number within each classroom Students can be uniquely identified in the database by the combination of IDSTUD and IDCNTRY TIMSS DATABASE USER GUIDE 5 CHAPTER 7 DATABASE FILES IDCLASS Identification number that uniquely identifies the sampled class
184. o a particular sampling zone The name of this variable in all TIMSS files is JKZONE JKI The variable that captures whether the case is to be dropped or have its weight doubled for the corresponding replicate weight The name of this variable in all TIMSS files is JKINDIC NJKR This indicates the number of replicate weights to be generated when computing the JRR error estimates When conducting analyses using the data from all countries the value of NJKR should be set to 75 for the student school and teacher background data and 42 for the performance assessment data The user working with the data for only one country should set the NJKR argument to as many replicates as there were in the country The maximum number of replicates by country is shown in Table 9 2 If the data from two or more countries is being used for an analysis then the larger number of jackknife zones should be used When in doubt on what number to set the NJKR parameter it should be set to 75 The error variance will always be estimated correctly if more replicate weights than necessary are computed but will be underestimated if the user specifies less replicate weights than necessary TIMSS DATABASE USER GUIDE 9 1 CHAPTER 9 PERFORMING ANALYSES Table 9 2 Number of Replicate Weights Needed for Computing the JRR Error Variance Estimate 3rd Grade 4th Grade 7th Grade 8th Grade Australia Austria Belgium FI Belgium Fr Bulgaria Canada Colombia Cy
185. o read more carefully Nonetheless a preliminary read through before actually opening up the files and trying to use them would help the user better understand the complexities of the study and the International Database When using the files the user will need to follow certain sections of this guide more carefully than others and refer to the supplements to the guide The contents of each chapter and the supplements are summarized below Chapter 2 TIMSS Instruments and Booklet Design This chapter describes the content and organization of the TIMSS tests for the lower and upper grades of Populations 1 and 2 the performance assessment administered to subsamples of the upper grade students in Populations 1 and 2 and the student teacher and school background questionnaires The TIMSS item release policy also is described Chapter 3 Sampling and Sampling Weights This chapter describes the sampling design for TIMSS the use of sampling weights to obtain proper population estimates and the weight variables included in the data files Chapter 4 Data Collection Materials Processing Scoring and Database Creation This chapter describes the data collection and field administration procedures used in TIMSS the scoring of the free response items data entry procedures and the creation of the International Database including the data verification and database restructuring Chapter 5 TIMSS Scaling Procedures This chapter provides an o
186. of categories These major categories in turn were partitioned into subcategories specifying the content performance expectations and perspectives in more detail For example for each of the content categories there are up to six more specific subcategories TIMSS DATABASE USER GUIDE 28 CHAPTER 2 Figure 2 1 INSTRUMENTS The Major Categories of the TIMSS Curriculum Frameworks MATHEMATICS Content Numbers Measurement Geometry Proportionality Functions relations equations Data probability statistics Elementary analysis Validation and structure Content Earth sciences Life sciences Physical sciences Science technology mathematics History of science and technology Environmental issues Nature of science Science and other disciplines SCIENCE Performance Expectations Knowing Using routine procedures Investigating and problem solving Mathematical reasoning Communicating Perspectives Attitudes Careers Participation Increasing interest Habits of mind Performance Expectations Understanding Theorizing analyzing solving problems Using tools routine procedures and science processes Investigating the natural world Communicating Perspectives Attitudes Careers Participation Increasing interest Safety Habits of mind TIMSS DATABASE USER GUIDE INSTRUMENTS CHAPTER 2 The two dimensions of th
187. of the data files listed in Table 7 12 Table 7 12 Population 1 and Population 2 Codebook Files Data File Population 2 Population 1 Codebook Name Codebook Name Student Written Assessment File BSACODE ASACODE Student Background File BSGCODE ASGCODE Teacher Background File s BTMCODE Mathematics ATGCODE BTSCODE Science School Background File BCGCODE ACGCODE Student Teacher Linkage File BLGCODE ALGCODE Student Written Assessment Reliability File BSRCODE ASRCODE Student Performance Assessment File BSPCODE ASPCODE Student Performance Assessment Reliability File BSQCODE ASQCODE School Performance Assessment File BCTCODE ACTCODE Codebook files are available in two different formats differentiated by the file extension e ASCII Text Format CDT e Machine Readable ASCII Format CDF 7 3 1 Accessing the Codebook Files Both codebook file types are included in the database CD in ASCII format They can be read and edited with any text editor or word processing software that can read files of their size Each is designed with a specific purpose in mind Printout Format CDT The printout format is a text file containing the information from the codebook in a printout format This format can be read with any word processing software and printed after some minor formatting We suggest using a mono spaced font and a font size and page layout combination that will accommodate 132 characters
188. on 15 for each student For details on the uses of plausible values the reader is referred to Mislevy 1991 and Mislevy Beaton Kaplan and Sheehan 1992 Unlike previously described methods for drawing plausible values Beaton 1987 Mislevy et al 1992 ConQuest does not assume normality of the marginal posterior distributions Recall from 15 that the marginal posterior is given by BE f 8 W gt 2 5 10 5 0 X d0 23 h 0 W E y EIx x M The ConQuest procedure begins drawing M vector valued random deviates Pin a from the multivariate normal distribution f 0 W y 2 for each case n These vectors are used to approximate the integral in the denominator of 23 using the Monte Carlo integration 1 M fA 16 f 8 7 2 d0 ur DAE Om S 24 0 m 1 At the same time the values Pon o f x E Pin lPi W y E 25 M are calculated so that we obtain the set of pairs Pans x that can be used as an m l approximation to the posterior density 23 and the probability that could be drawn from this density is given by p q j a 26 pn 26 m l 5 6 TIMSS DATABASE USER GUIDE SCALING CHAPTER 5 n L At this point L uniformly distributed random numbers In are generated and for each random draw the vector that satisfies the condition S a ns Ya 27 is selected as a plausible vector 5 8 Scaling Steps The item response model described above wa
189. one member of each pair of schools is randomly selected to have its weights doubled while the weights of the other member are set to zero This variable JKZONE indicates the sampling zone to which the student s school is assigned The sampling zones can have values from 1 to 75 in the Student Background and Written Assessment data files In the Performance Assessment data files there is a maximum of 42 sampling zones in Population 2 and 39 in Population 1 This variable is included in the Student Background the student Written Assessment the Student Teacher Linkage and the student Performance Assessment data files For each individual student this variable is identical in the first three files but differs in the Performance Assessment data files because the performance assessment sample is a sub sample of the written assessment sample The variable JKINDIC indicates how the student is to be used in the computation of the replicate weights This variable can have values of either 1 or 0 Those student records with a value of 0 should be excluded from the corresponding replicate weight and those with a value of 1 should have their weights doubled This variable is included in the student background the student written assessment the student teacher linkage and the student performance assessment data files For each individual student this variable is identical in the first three files but differs in the performance assessment data files because t
190. onnaires 2 10 2 4 QUuestionndiles race e op ec OH Or Per uei p V Eve a vd ede Ri pi edes 2 10 Chapter 3 Sampling and Sampling Weights eere 3 1 Bk The Target Populo S ccs ner eerte on tend eter tiii eere hg 3 1 3 2 School Sample Selecfior it eee d e ptio hentai 3 5 3 3 Classroom and Student Sampling sss 3 6 3 4 Performance Assessment Subsampling sse 3 6 3 5 Response Rates cee 3 5 1 SchoolLevel Response Rates 3 5 2 Studentlevel Response Rates t tis 3 7 3 5 3 OveralResponse Ralesc to RR RR HR DAR DREAM A AN i OA ON e ARR ate 3 8 3 6 Compliance with Sampling Guidelines sse 3 8 3 7 Sampling Weighs nee o nee EH et e Net eid RUE aun eae 3 11 3 8 Weight Variables Included in the Student Data Files sss 3 14 3 9 Weight Variables Included in the Student Teacher Linkage Files sss 3 16 3 10 Weight Variables Included in the School Data Files sss 3 17 Chapter 4 Data Collection Materials Processing Scoring and Database Creations scissasscsussssssvsvscsetasssoavsdassesesssastosnannstsvensnsesvsossnsiies 4 1 4 1 Data Collection and Field Administration sss 4 1 4 2 Free Response Scoring cioe e ete eee der e eee ba e E ii a ete teils 4 2 A3 iDatacEntry Sce te ee ee e E Rees 4 6 4 4 Database Credti n x ne ete dee RR er e be tee ten 4 7 4
191. ons or open ended numerical values in accordance with the corresponding background questionnaire item formats The codebook files for students teachers and schools contain the international background variable names descriptive labels response code definitions formats and field locations corresponding to each questionnaire item Identifying Background Variables by Questionnaire Numbers The international background variables are listed in the codebook files in order of the corresponding questions in the international version of the background questionnaires For each background variable the corresponding international questionnaire location is given The questionnaire item numbers associated with each variable are indicated by field locations according to the formats given in Table 7 3 A set of tables mapping each background variable to its location in the questionnaire is included in Supplement 1 and Supplement 2 Z 10 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 Table 7 3 Background Questionnaire Item Field Location Format Conventions Field Location Population Questionnaire Example Question Format Population 1 tudent Questionnaire s Q1 2 Are you a boy or a girl Population 1 Teacher Questionnaire TQ1 TQ1 9C Hours week planning lessons Population 1 School Questionnaire SCQ1 SCQ1 4A How many full time teachers Population 2 Student Questionnaire variables in both SQ2 and SQ2 SQ2 25C How often do you
192. ooklet Design 2 1 Introduction TIMSS used several types of instruments to collect data about students teachers and schools Each assessed student received a test booklet containing cognitive items in mathematics and science along with a separate background questionnaire Subsamples of students participating in the written assessment also participated in a performance assessment in which they completed hands on mathematics and science tasks Teacher questionnaires were given to the mathematics and science teachers of the assessed students A school questionnaire was distributed to each participating school and completed by the school principal or headmaster This chapter describes the content and organization of the assessment instruments for Populations 1 and 2 2 2 The TIMSS Mathematics and Science Content The TIMSS Curriculum Frameworks for Mathematics and Science Robitaille et al 1993 contain three dimensions subject matter content performance expectations and perspectives Subject matter content refers to the content of the mathematics or science test item under consideration Performance expectations describe in a non hierarchical way the kinds of performance or behavior that a given test item might elicit from students The perspectives aspect focuses on the development of students attitudes interests and motivations in mathematics and science As shown in Figure 2 1 each of the three aspects is partitioned into a number
193. opulation estimates of proficiency These scores are included in the Student Background the student Written Assessment and the student Performance Assessment files AIMATSCR International Mathematics Achievement Score Population 1 AISCISCR International Science Achievement Score Population 1 BIMATSCR International Mathematics Achievement Score Population 2 BISCISCR International Science Achievement Score Population 2 These are the international mathematics and science achievement scores used to report achievement at the international level They correspond to the first plausible value in each subject area It is recommended that these scores be used for both international and within country comparisons Not only do they allow for comparisons across countries but they also take into account the specific difficulty of the items attempted by each student and their relative difficulty internationally and reflect the measurement error component International proficiency scores are included in the Student Background the student Written Assessment the student Performance Assessment files and the Student Teacher Linkage files The mean of these scores within each school by grade level are also included in the School Background data files Tables 6 1 6 2 6 3 and 6 4 present weighted statistics for these scores within each participating country 6 4 TIMSS DATABASE USER GUIDE ACHTEWVEM ENT SCO RES CHAPTER 6 Table 6 1 D
194. orts and too late to correct the database The errors mostly affect the lower grade students and are expected to account for less than 1046 of students Philippines The teacher and school data submitted by the Philippines were not deemed internationally comparable and thus are not included in the International Database Due to the use of unapproved school sampling procedures the results presented in the international reports for the Philippines reflect unweighted data Consequently the sampling weights have been set to 1 for all cases in the files for the Philippines South Africa The teacher and school data submitted by South Africa were not deemed internationally comparable and thus are not included in the International Database Thailand Information made available after publication of the Population 2 international reports required that the sampling weights for Thailand be recomputed The adjusted sampling weights are included in the International Database As a consequence any computations using these new weights may be slightly different from those shown in the international report tables for Population 2 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 7 3 Codebook Files All information related to the structure of the data files as well as the source format descriptive labels and response option codes for all variables discussed in Section 7 2 is contained in codebook files One codebook file is provided for each
195. ountries thereby increasing the likelihood of producing useful age based comparisons in addition to the grade based analyses The stated objective in TIMSS was that the effective population the population actually sampled by TIMSS be as close as possible to the International Desired Population Figure 3 1 illustrates the relationship between the desired populations and the excluded populations at the country school and student levels Figure 3 1 Relationship Between the Desired Populations and Exclusions International Desired Target Population National Desired Target Population Exclusions from National Coverage National Defined Target Population School Level Exclusions Effective Target Population Within Sample Exclusions Using the International Desired Populations as a basis participating countries had to operationally define their populations for sampling purposes Some NRCs had to restrict coverage at the country level for example by excluding remote regions or a segment of the educational system Thus the Nationally Desired Population sometimes differed from the International Desired Population The basic sample design used by TIMSS is generally referred to as a two stage stratified cluster sample design The first stage consisted of a sample of schools which may be stratified the second stage consisted of samples of classrooms from each eligible target grade in sampled schools In some countries a third stage consi
196. per line The information for each variable is presented in several lines of text The lines for each variable are properly labeled TIMSS DATABASE USER GUIDE 33 CHAPTER 7 DATABASE FILES Machine Readable Format CDF A second formatted version of the codebooks is also included in the database In this version each variable occupies one line and the following fields are included variable name question location starting column ending column number of digits number of decimals variable label and value labels These files can be used by those who want to use programming languages other than SAS or SPSS to conduct analysis with the TIMSS data The value labels in these files are separated by semicolons Table 7 13 describes the structure of the machine readable codebook files Table 7 13 File Structure of Machine Readable Codebook Files Field Name Location Variable name Question location Starting location in data file Ending location in data file Number of digits Number of decimals Variable label Value labels 7 3 2 Using the Codebooks The variables in the codebooks appear in order by variable name within the section for each codebook type The major sections of each codebook type are as follows Student Teacher and School Background File Codebooks e Identification Variables e Tracking Linking Variables e International Background Variables in order of questionnaire item location e Derived Variables in o
197. phical error in coding guide Typographical error in coding guide Typographical error in coding guide Typographical error in code 74 28 instead of 280 leaves gap in 7 diagnostic codes Typographical error in coding guide Typographical error in coding guide Typographical error in coding guide Typographical error in coding guide Typographical error in coding guide Typographical error in coding guide Population 2 Written Assessment Items gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt gt BSEMSO1A 19 BSEMS02A 19 BSEMTO1A 29 BSEMT02A 19 BSEMUO1A 19 BSEMUO2A 19 29 BSEMUO2B 19 29 Typographical error in coding guide Vivi vv VyVy Vx 26 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 Table 7 11 Continued Variable Recodes Comment 20 29 10 11 ASEMT04B 20 29 10 11 ASEMV04A 30 20 21 ASESY01 20 29 10 11 19 ASESZ02 30 31 20 29 10 11 12 13 Only 20s have positive point biserial correlation y Only 20s have positive point biserial correlation Differentiation between 30s 20s and 10s not clear Only 20s have positive point biserial correlation Only 30s have positive point biserial correlation n E oO c o E n N N n lt c o i m c 9 3 2 o o a VES VV VV V VV VV Vili Vi
198. prus Czech Republic Denmark England France Germany Greece Hong Kong Hungary Iceland Iran Islamic Rep Ireland Israel Japan Korea Kuwait Latvia LSS Lithuania Netherlands New Zealand Norway Portugal Romania Russian Federation Scotland Singapore Slovak Republic Slovenia South Africa Spain Sweden Switzerland Thailand United States 9 12 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 CVAR This lists the variables that are to be used to classify the students in the data file This can be a single variable or a list of variables The maximum number of variables will depend mostly on the computer resources available to the user at the time It is recommended to always include the variable that identifies the country At least one variable has to be specified usually IDCNTRY DVAR This is the variable for which means are to be computed Only one variable has to be listed here If the user wants to find out for example the results in mathematics and science then the macro needs to be invoked separately to generate each table Although in most cases the continuous variable of interest will be an achievement variable this can actually be any other continuous variable in the data file FNAME The name of the data set that contains the variables necessary for the analysis The name of this file has to be in either the single or multilevel SAS format The multilevel SAS file format contains the lib
199. question or perform and report on a specific task rather than choosing an answer from a list of options The answers to these questions were coded by coders trained to use the two digit scoring rubrics described in Chapter 4 The first digit of the two digit code indicates the score given to the question and the second digit in conjunction with the first provides diagnostic information on the specific answer given by the student These types of response codes were used for the free response items administered as part of the written assessment and for the items in the performance assessment tasks The codes used to represent the responses to these items are shown in Table 9 4 below Table 9 4 Definition of Response Codes for the Open Ended Items in the Written Assessment and Performance Assessment Data Files Code Description 301039 Three point answer Second digit provides diagnostic information 201029 Two point answer Second digit provides diagnostic information 10to 19 4 One point answer Second digit provides diagnostic information 701079 X Zero point answer Second digit provides diagnostic information 90 Gave a response that could not be scored 96 Did not reach the item 98 The item was not administered 99 Did not respond to the item although the item was administered and was reached The Performance Assessment and Written Assessment data files contained in the CD include information about the answer given to each item
200. r The file resulting from using this macro can then be printed using a SAS procedure of choice An example call to this macro and a subset of the resulting file is presented in Figure 9 8 In this example the macro will compute the percent of boys and girls by grade and by country and their mean achievement in mathematics The listing presented in Figure 9 8 is interpreted in the following way The first line shows the results for the students with IDCNTRY 36 Australia in IDGRADER 1 Seventh grade and who had ITSEX 1 Girls It is estimated that there are 123649 seventh grade girls in Australia their mean mathematics score is 500 with a standard error of 4 3 We can also tell from this line of data that it 1s estimated that 52 percent of the seventh graders in Australia are girls and the standard error of this percent is 2 3 The second line shows the same information but for the seventh grade boys ITSEX 2 It is estimated that there are 114646 seventh grade boys in Australia their mean mathematics score is 495 with a standard error of 5 2 We can also tell from this line of data that it is estimated that 48 percent of the seventh graders in Australia are boys and the standard error of this percent is 2 3 9 20 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES Figure 9 8 SPSS Control Code and Extract of Output File for Using the Macro JACK SPS get file bsgalll sys keep idcntry idstud idgrader jkindic jkzone totwgt itsex
201. r as b 5 Digsotss Dix F bs Dyzio by Siue Dix By definition the score for a response in the zero category is zero but other responses may also be scored zero In the majority of Rasch model formulations there has been a one to one match between the category to which a response belongs and the score that is allocated to the response In the simple logistic model for example it has been standard practice to use the labels O and 1 to indicate both the categories of performance and the scores A similar practice has been followed with the rating scale and partial credit models where each different possible response is seen as indicating a different level of performance so that the category indicators 0 1 2 that are used serve as both scores and labels The use of b as a scoring function allows a more flexible relationship between the qualitative aspects of a response and the level of performance that it reflects Examples of where this is applicable are given in Kelderman and Rijkes 1994 and Wilson 1992 A primary reason for implementing this feature in the model was to facilitate the analysis of the two digit coding scheme that was used in the TIMSS short answer and extended response items In the final analyses however only the first digit 5 2 TIMSS DATABASE USER GUIDE SCALING CHAPTER 5 of the coding was used in the scaling so this facility in the model and scaling software was not used in TIMSS Letting 0 be the lat
202. r want to preserve the original answers and codes assigned to the questions then the file with the recoded item variables needs to be saved under a different file name A copy of the macro that scores the items in SAS and SPSS and an example of how it is invoked in presented in Figure 9 21 and Figure 9 22 TIMSS DATABASE USER GUIDE 9 41 CHAPTER 9 Figure 9 21 Extracted Sections of SAS Control Code Used to Convert Cognitive Item Response Codes to Correctness Score Levels MACRO SCOREIT ITEM TYPE RIGHT NR NA OM OTHER r PERFORMING ANALYSES SIF amp TYPE MC THEN DO SCORE 0 IF amp ITEM amp RIGHT THEN SCORE 1 IF amp ITEM amp NR THEN SCORE 0 IF amp ITEM amp NA THEN SCORE IF amp ITEM 60M THEN SCORE 0 IF amp ITEM amp OTHER THEN SCORE 0 amp ITEM SCORE SEND SIF amp TYPE SA OR amp TYPE EX THEN DO SCORE 0 IF amp ITEM gt 30 AND amp ITEM 40 THEN SCORE 3 IF amp ITEM 20 AND amp ITEM 30 THEN SCORE 2 IF amp ITEM 10 AND amp ITEM 20 THEN SCORE 1 IF amp ITEM gt 70 AND amp ITEM 80 THEN SCORE 0 IF amp ITEM amp NR THEN SCORE 0 IF amp ITEM amp NA THEN SCORE IF amp ITEM 60M THEN SCORE 0 IF amp ITEM amp OTHER THEN SCORE 0 amp ITEM SCORE SEND MEND SCOREIT LET ARIGHT BSMMA01 BSMSA10 BSMSB05 BSMSB06 BSMMBO8 BSMMB09 BSMMB12 BSMMCO1 BSM
203. rary as well as the data set It is important to emphasize that this data set must include only those cases that are of interest in the analysis If the user wants to have specific cases excluded from the analysis as for example students with missing variables or select students from a specific grade this should be done prior to invoking the macro The simplest and most straightforward way is to invoke the macro using the conventional SAS notation for invoking macros followed by the list of arguments for the analysis enclosed in parenthesis separated by commas and in the same order as they are listed above For example if the macro is invoked as jack TOTWGT JKZONE JKINDIC 75 IDCNTRY IDGRADER ITSEX BIMATSCR BSGALL1 it will compute the mean mathematics achievement and its corresponding standard error for boys and girls by grade within each country using the variable TOTWGT as the sampling weight It will also compute the percent of boys and girls by grade within the country and its corresponding standard error The data will be read from the data set BSGALL1 and the standard error of the statistics will be computed based on 75 replicate weights The file that contains these results is called FINAL and can be found in the WORK library The variables that are contained in this file are Classification Variables Each of the classification variables are kept in the resulting file In the above example there would be three variables
204. rder of international table figure reference Sampling Variables Student and School Files only e Score Variables Student Files only 7 34 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 Student Written and Performance Assessment File Codebooks Identification Variables Tracking Linking Variables Cognitive Item Variables in order by item within clusters or performance assessment tasks Sampling Variables Score Variables School Performance Assessment File Codebooks Identification Variables Tracking Linking Variables Student Teacher Linkage File Codebooks Identification Variables Sampling Variables Score Variables Teacher Linking Weighting Variables Reliability File Codebooks Identification Variables Tracking Variables Reliability Variables organized into sets of three variables described previously in order by item within cluster or performance assessment task TIMSS DATABASE USER GUIDE 7 35 LEES F DATABASE 7 CHAPTER An example printout of a page from the codebook for the student background data BSGCODE is shown in Figure 7 1 The files are as follows butsstu lt uoseez peurgep euorjeu obenbue peexr zo yeeds o4 e qeun quepnais pepaej3ei2 a3ueu e qeonpe sr 4uepnjs petqestp T I euora3oung sr 4uepnjs xSLNSGAQLS GHGd IOXH AOT HOLVOIGNI G lIOXHGI Gn IoXH uTupe jou butsstu qi unjez3s uor3erndod z uort3erndod I uotjzetndod 3eui
205. recode btbgtaug 0 thru 5 1 6 thru 10 2 10 thru 20 3 20 thru 90 4 98 99 sysmis sort cases by idcntry idteach idlink save outfile teacher get file blgalll sys keep idcntry idteach idlink idgrader jkindic jkzone matwgt bimatscr select if matwgt 0 and idgrader 2 sort cases by idcntry idteach idlink Save outfile studteac Now merge the two files match files file studteac table teacher by idcntry idteach idlink Select if not missing btbgtaug save outfile merged Define the format for the variables used value labels idgrader 1 Seventh Grade 2 Eighth Grade btbgtaug 1 0 to 5 years 2 6 to 10 years 3 11 to 20 years 4 Over 20 years identry 036 Australia 040 Austria 056 057 Belgium Fr 100 Bulgaria 124 170 Colombia 196 Cyprus 200 208 Denmark 826 England 250 280 Germany 300 Greece 344 348 Hungary 352 Iceland 364 372 Ireland 376 Israel 380 392 Japan 410 Korea 414 428 Latvia LSS 440 Lithuania 528 554 New Zealand 578 Norway 608 620 Portugal 642 Romania 643 827 Scotland 702 Singapore 201 890 Slovenia 717 South Africa 724 752 Sweden 756 Switzerland 764 840 United States Now use the macro to get the results include jack sps jack cvar dvar bimatscr njkr 75 jkz jkzone
206. representing students proficiency scores in mathematics and science are discussed in Chapter 6 These variables were used in the reporting of international achievement results in the international reports and are provided in the Population 1 and Population 2 Student Background files under the following variable names AIMATSCR AISCISCR BIMATSCR BISCISCR Population 1 International Mathematics Score Population 1 International Science Score Population 2 International Mathematics Score Population 2 International Science Score The following additional achievement variables described in Chapter 6 are also provided in the Student Background files ASMPVO0I ASMPV05 ASSPV01 ASSPV05 BSMPVOI BSMPV05 BSSPVO01 BSSPV05 AIMATEAP AISCIEAP BIMATEAP BISCIEAP ASMNRSC ASSNRSC BSMNRSC BSSNRSC ASMSCPT ASSSCPT Population 1 Mathematics Plausible Values 1 5 Population 1 Science Plausible Values 1 5 Population 2 Mathematics Plausible Values 1 5 Population 2 Science Plausible Values 1 5 Population 1 International Mathematics Score EAP Population 1 International Science Score EAP Population 2 International Mathematics Score EAP Population 2 International Science Score EAP Population 1 National Rasch Mathematics Score ML Population 1 National Rasch Science Score ML Population 2 National Rasch Mathematics Score ML Population 2 National Rasch Science Score ML Population 1 Number of Raw
207. rformance Assessment Items Value s Character Vs Definition Written Assessment Performance Assessment Position Population A Population 1 A Population 1 B Population 2 B Population 2 Student Variable S S Item Format M Multiple Choice P Performance Assessment S Short Answer E Extended Response Subject Area M Mathematics M Mathematics S Science S Science G Math Science Combined Cluster or Task Location AB Z 1 6 number of task within each subject category Item Number 01 99 1 2 3A 3B etc item number within task For example ASMMIOI Population 1 written assessment mathematics multiple choice item number I01 BSPS11A Population 2 performance assessment science task 1 S1 item 1A 7 2 2 4 Performance Assessment Task Identifications The performance assessment tasks are listed in Chapter 2 Table 2 7 which indicates the task identification number and task titles Complete descriptions of the tasks administered to students in each population and the items contained in each may be found in the international report describing the performance assessment results Harmon et al 1997 The task identification information is also indicated in the variable labels in the performance assessment codebooks TIMSS DATABASE USER GUIDE 7 7 CHAPTER 7 DATABASE FILES 7 2 2 5 Cognitive Item Response Code Values The values assigned to each of the cognitive item variables depend on the item format
208. ro programs for converting cognitive item response codes to score values used in the computation of international scores Data Almanac Files contain unweighted summary statistics for each participating country on each variable in the student teacher and school background questionnaires To investigate the match between the TIMSS achievement tests and the curriculum in participating countries country representatives indicated whether or not each test item addressed a topic in their curriculum The Test Curriculum Matching Analysis Files contain this information for each country The following sections describe each of the file types and how they can be used to access and analyze the TIMSS international data for students teachers and schools For more detailed information about data entry data processing dota cleaning data management database structure as well as analysis and reporting of the TIMSS data see the TIMSS Technical Reports Volumes and II Martin and Kelly 1996 1997 and TIMSS Quality Assurance in Data Collection Martin and Mullis 1996 TIMSS DATABASE USER GUIDE ze CHAPTER 7 DATABASE FILES 7 2 Data Files There are four basic types of data files in the TIMSS International Database e Background Files e Assessment Files e Coding Reliability Files e Student Teacher Linkage Files These files and the variables contained in each are described in Sections 7 2 1 through 7 2 4 Data files are provided for each country
209. rs 1859 55882 22 533 099 8 4781 27 6038 2 60588 COUNTRY ID Austria GRADE Eighth Grade BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 159 4573 10 515 596 19 7319 7 1599 2 31322 6 to 10 years 298 8429 89 545 782 9 4934 13 1983 2 47861 11 to 20 years 1182 32826 23 553 934 6 6800 51 3945 4 00150 Over 20 years 500 18041 87 549 280 8 7719 28 2473 3 64019 COUNTRY ID Belgium Fl GRADE Eighth Grade BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 281 6915 80 555 760 17 9455 9 5878 2 75864 6 to 10 years 306 6351 73 589 843 14 5214 8 8058 2 21509 11 to 20 years 864 23216 69 553 862 13 4302 32 1866 4 75228 Over 20 years 1357 35647 25 574 548 10 5636 49 4198 4 90806 COUNTRY ID Belgium Fr GRADE Eighth Grade BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 159 3051 78 536 164 12 2866 8 1705 3 15008 6 to 10 years 168 2844 35 528 197 13 7981 7 6151 2 31007 11 to 20 years 565 11466 42 558 394 6 9786 30 6989 5 20517 Over 20 years 976 19988 70 543 097 6 3910 53 5155 4 84059 COUNTRY ID United States GRADE Eighth Grade BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 1642 684804 90 483 554 6 34204 24 6560 3 42971 6 to 10 years 950 383442 47 488 370 9 75550 13 8056 2 69418 11 to 20 years 1408 706610 59 500 842 7 29526 25 4411 3 24706 Over 20 years 2107 1002577 37 513 453 7 51101 36 0972 3 31137 COUNTRY ID Slovenia GRADE Eighth Grade BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 83 839 82 536 528 23 2197 3 8502 1 89562
210. rs for a representative sample of students within a country Therefore it is appropriate that statements about the teachers be made only in terms of how many students are taught by teachers of one kind or another and not in terms of how many teachers in the country do one thing or another TIMSS DATABASE USER GUIDE 9 27 CHAPTER 9 PERFORMING ANALYSES Figure 9 13 SAS Control Statements for Performing Analyses with Teacher Level Variables EXAMPLE2 SAS options nocenter Read the variables from the Mathematics Teacher file and sort by merge variables data teacher set btmalll keep idcntry idteach idlink btbgtaug if btbgtaug gt 0 amp btbgtaug lt 5 then btbgtaug 1 if btbgtaug gt 5 amp btbgtaug lt 10 then btbgtaug 2 if btbgtaug gt 10 amp btbgtaug lt 20 then btbgtaug 3 if btbgtaug gt 20 amp btbgtaug lt 90 then btbgtaug 4 proc sort data teacher by idcntry idteach idlink Read the variables from the student teacher link file and sort by merge variables data studteac set blgalll keep idcntry idteach idlink idgrader jkindic jkzone matwgt bimatscr proc sort data studteac by idcntry idteach idlink Now merge the two files data merged merge studteac teacher by idcntry idteach idlink if matwgt gt 0 and nmiss btbgtaug 0 Define the format for the variables used proc format library work value grade 1 Seventh Grade 2 Eighth Grade value exper 1
211. s corresponding standard error The data would be read from the system file BSGALLI The file that contains these results is then called FINAL and is saved to the default directory being used by SPSS The variables that are contained in this file are Classification Variables Each of the classification variables is kept in the resulting file In our example above there would be three variables in the resulting system file These would be IDCNTRY IDGRADER and ITSEX There is one unique occurrence for each combination of the categories for these variables Weight Variable Contains the estimate in the population that belongs to the group defined by the specific combination of the classification variable categories MNX Contains the weighted mean of the variable DVAR for the group defined by the corresponding combination of classification variable categories TIMSS DATABASE USER GUIDE 9 19 CHAPTER 9 PERFORMING ANALYSES MNX SE Contains the standard error of MNX computed using the JRR method for computing the standard error PCT Contains the weighted percent of people in the group for the classification variable listed last within the specific combination of the categories defined by the groups defined initially In our example we would obtain the percent of boys and girls within each combination of country and grade PCT SE Contains the standard error of PCT computed using the JRR method for computing the standard erro
212. s achievement score BIMATSCR and the information necessary to compute the replicate weights for estimating the JRR standard error We select the variable that will give us the correct weight for mathematics teacher variable MATWGT If the user is interested in looking at the science teachers then the weight variable that should be selected is SCIWGT and TCHWGT if the user is interested in analyzing both mathematics and science teachers combined The two files are then merged or matched into one file that will then be used with the JACK macro These two files will be merged using the variables IDCNTRY IDTEACH and IDLINK The combination of values for these three variables is unique within the teacher data but is repeated in the Student Teacher Linkage file as many times as the specific teacher teaches students in a class After the files are merged the macro JACK is used and the results can be printed Selected results from this analysis are shown in Figure 9 15 and Figure 9 16 9 30 TIMSS DATABASE USER GUIDE PERFORMING ANALYSE Figure 9 15 CHAPTER Extract of SAS Computer Output for Performing Analyses with Teacher Level Variables EXAMPLE 2 COUNTRY ID Australia GRADE Eighth Grade 9 BTBGTAUG N MATWGT MNX MNX SE PCT PCT SE 0 to 5 years 1174 35747 90 517 042 8 5001 17 6582 2 31566 6 to 10 years 1164 39265 42 528 252 11 5941 19 3957 2 55903 11 to 20 years 2205 71548 18 541 140 8 3675 35 3423 2 74924 Over 20 yea
213. s fit to the data in two steps In the first step a calibration of the items was undertaken using a subsample of students drawn from the samples of the participating countries These samples were called the international calibration samples In a second step the model was fitted separately to the data for each country within the item parameters fixed at values estimated in the first step There were three principal reasons for using an international calibration sample for estimating international item parameters First it seemed unnecessary to estimate parameters using the complete data set second drawing equal sized subsamples from each country for inclusion in the international calibration sample ensured that each country was given equal weight in the estimation of the international parameters and third the drawing of appropriately weighted samples meant that weighting would not be necessary in the international scaling runs 5 8 1 Drawing The International Calibration Sample For each target population samples of 600 tested students were selected from the database for each participating country This generally led to roughly equal samples from each target grade For Israel where only the upper grade was tested the sample size was reduced to 300 tested students The sampled students were selected using a probability proportionality to size systematic selection method The overall sampling weights were used as measures of size for this purpose
214. s provided by the National Center for Education Statistics of the U S Department of Education the U S National Science Foundation and the International Association for the Evaluation for Educational Achievement Eugene Owen and Lois Peak of the National Center for Education Statistics and Larry Suter of the National Science Foundation each played a crucial role in making TIMSS possible and for ensuring the quality of the study Funding for the International Coordinating Center was provided by the Applied Research Branch of the Strategic Policy Group of the Canadian Ministry of Human Resources Development This initial source of funding was vital to initiate the TIMSS project Tjeerd Plomp Chair of the IEA and of the TIMSS International Steering Committee has been a constant source of support throughout TIMSS It should be noted that each country provided its own funding for the implementation of the study at the national level NATIONAL RESEARCH COORDINATORS The TIMSS National Research Coordinators and their staff had the enormous task of implementing the TIMSS design in their countries This required obtaining funding for the project participating in the development of the instruments and procedures conducting field tests participating in and conducting training sessions translating the instruments and procedural manuals into the local language selecting the sample of schools and students working with the schools to arrange for the testing
215. se options in order to include the appropriate wording or options most consistent with their own national systems In the international versions of the questionnaires Supplements 1 and 2 such questions contain instructions to NRCs to substitute the appropriate wording for their country or to modify or delete any inappropriate questions or options These instructions were indicated in two ways in the questionnaires 1 NRC NOTE 2 International Option indicating that the NRC was to substitute if necessary an appropriate national option that would retain the same basic interpretation as the international version In addition to some of the questions designed to contain some nationally modified portions some countries also included revised versions of other questions as well as some inadvertent deviations from the international version e g option reversals Although no formal translation verification step was used for the background questionnaires countries were to provide a National Deviations Report and a Data Management Form that included information about how any revised or deleted items were to be handled during data entry and data processing The information contained in these reports was used together with the international univariate summary statistics for background questionnaire items and direct comparisons of the international and national versions of the questionnaires in order to evaluate the international comparability of
216. secondary analyses based on these content areas the classification of TIMSS written assessment items into each of the Population 1 and Population 2 mathematics and science content area reporting categories is presented in Tables 7 7 through 7 10 Achievement scale scores were not computed for each of these content area reporting categories See Chapter 2 for a discussion of the design of the TIMSS written assessment and performance assessment 18 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 Table 7 7 Classification of Population 1 Items into Mathematics Content Area Reporting Categories Items in Content Area Reporting Category Measurment Data Patterns Whole Fractions and OA Representation Estimation and Geometry Relations and Numbers Proportionality Analysis and Number Sense SN Functions Probability 25 items 21 items 20 items 12 items 14 items 10 items TIMSS DATABASE USER GUIDE 7 19 CHAPTER 7 DATABASE FILES Table 7 8 Classification of Population 1 Items into Science Content Area Reporting Categories Items in Content Area Reporting Category Environmental Earth Science Life Science Physical Science Issues and the Nature of Science 17 items 41 items 30 items 9 items 7 20 TIMSS DATABASE USER GUIDE DATABASE FI LES CHAPTER 7 Table 7 9 Classification of Population 2 Items into Mathematics Content Area Reporting Categories Items in Content Area Reporting C
217. sessment of students in their final year of secondary school Population 3 will be released in a separate database In each country a National Research Coordinator was responsible for conducting TIMSS in accordance with the international procedures established by the TIMSS International Study Center at Boston College This included selecting a representative sample of schools and students for each population translating the data collection instruments into the language s of testing assembling the data collection instruments sending them to the sampled schools and arranging for data collection in the schools In each school sampled for TIMSS a School Coordinator and a Test Administrator administered the assessment instruments and followed security procedures After the testing session the School Coordinator returned the testing materials to the national research center At that time the National Research Coordinator arranged for scoring the open ended responses and following that arranged to have the test and questionnaire responses entered into data files These data files were then submitted to the 4 TIMSS DATABASE USER GUIDE IN TRODUCTION C HAPPE RI IEA Data Processing Center for international processing For each task manuals documenting the international procedures were provided together with various forms used to document the implementation of the tasks In addition international training sessions were held several times a y
218. sette Le Coq Centre International d Etudes P dagogiques CIEP Avenue L on Journault 93211 S vres France TIMSS DATABASE USER GUIDE Germany Rainer Lehmann Humboldt Universitaet zu Berlin Institut fuer Allgemeine Erziehungswissenschaft Geschwister Scholl Str 6 10099 Berlin Germany Juergen Baumert Max Planck Institute for Human Development and Education Lentzeallee 94 D 14195 Berlin Germany Manfred Lehrke Universitat Kiel IPN Olshausen Str 62 24098 Kiel Germany Greece Georgia Kontogiannopoulou Polydorides Department of Education Tmima Nipiagogon University of Athens Navarinou 13 Neochimio Athens 106 80 Greece Joseph Solomon Department of Education University of Patras Patras 26500 Greece Hong Kong Frederick Leung Nancy Law The University of Hong Kong Department of Curriculum Studies Pokfulam Road Hong Kong Hungary P ter Vari National Institute of Public Education Centre for Evaluation Studies Dorottya u 8 P F 701 420 1051 Budapest Hungary Iceland Einar Gudmundsson Institute for Educational Research Department of Educational Testing and Measurement Surdgata 39 101 Reykjavik Iceland ACKNOWLEDGMENTS Ireland Deirdre Stuart Michael Martin Educational Research Centre St Patrick s College Drumcondra Dublin 9 Ireland Iran Islamic Republic Ali Reza Kiamanesh Ministry of Education Center for Educational Research Iranshahr Shomali Avenue Teh
219. sing For Cognitive Items An Omitted Response Code value of 9 is used for multiple choice cognitive items For free response written assessment items and performance assessment items the two digit 99 code is used for omitted blank responses Uninterpretable Response Codes 7 90 For the cognitive items separate codes were established to distinguish between totally blank responses omitted missing and uninterpretable or invalid responses For multiple choice items cases where more than one response option was checked were classified as uninterpretable and given a code 7 For the free response items in the written assessment or the performance assessment uninterpretable student responses were given a code 90 which is distinguished from the code 99 given to items that were left blank The SAS and SPSS control statement files will recode these missing categories to special numeric missing codes in SAS and explicit missing codes in SPSS see Section 7 4 TIMSS DATABASE USER GUIDE Z 29 CHAPTER 7 DATABASE FILES Not Administered Codes 8 98 998 Special codes were given for items that were not administered to distinguish these cases from data that are missing due to non response The specific Not Administered Code value given depends on the number of valid codes available for each item as described above for the Omitted Response Codes There are two general cases when the Not Administered Codes are used 1 Data were no
220. spond to a questionnaire for each class taught that contained sampled students The Teacher Background files contain one record for each of the classes taught by either a mathematics or a science teacher In some cases although the teacher was to respond to more than one questionnaire responses to only one were obtained In these cases there were as many records entered in the teacher file as classes were taught by the teacher and the background information from the complete questionnaire was entered into these teacher records In Population 1 a single questionnaire was administered since both mathematics and science are usually taught by the same teacher at this age level the responses to this questionnaire can be found in one file There were two questionnaires administered in Population 2 one for the mathematics teachers and one for the science teachers The data from these questionnaires are found in separate files Variable names for questions asked in both questionnaires are the same In the Teacher Background data files each teacher was assigned a unique identification number IDTEACH and a Teacher Link Number IDLINK that is specific to the class taught by the teacher and to which the information in the data record corresponds The IDTEACH and IDLINK combination uniquely identifies a teacher teaching one specific class So for example students linked to teachers identified by the same IDTEACH but different IDLINK are taught by the same
221. sponse 9 cm and 2 cm Correct drawing shown Partial Response 9 cm and 2 cm Drawing is incorrect or missing The length and or width is not given or is incorrect Correct drawing is shown 70 n 72 cm width and a length equal to any other numbers except those given above Explicitly written or implicit from the drawing 74 9 cm length and a width equal to any other numbers than those given above Explicitly written or implicit from the drawing 79 Other incorrect Nonresponse 90 Crossed out erased illegible or impossible to interpret 98 BLANK 4 4 TIMSS DATABASE USER GUIDE DATA COLLECTION PROCEDURES CHAPTERS 4 Figure 4 2 Continued B Codes for ratio and areas C orrect Response 3 4 3 4 or equivalent The areas are 18 cm and 24 cm No work needs to be shown Also areas do not need to be mentioned if the ratio is consistent with the areas of the given rectangle and the rectangle the student has drawn in U 2a The ratio is NOT 3 4 but areas and ratio of part b are consistent with response in part a Partial Response 10 433 or equivalent Ratio is reversed The areas are 18 cm and 24 cm d An incorrect ratio or no ratio is given The areas are 18 cme and 24 cme o 02 The difference between the areas 6 is given instead of the ratio The areas are 18 cm and 24 cm Areas are NOT 18 cm and 24 cm but are consistent with response in part a and an incorrect ratio or no ratio is
222. st and stopped as soon as they did not know the answer to a question Logit scores for the students generally ranged between 4 and 4 Since it is not possible to obtain finite logit scores for those students who correctly answered all or none of the items scores for these students were set to 5 and 5 logits respectively These logit scores were then standardized to have a weighted mean of 150 and a standard deviation of 10 6 2 TIMSS DATABASE USER GUIDE ACHIEVEMENT SCORES CHAPTER 6 The national Rasch scores should not be used for international comparisons for two reasons they were computed with a different set of item difficulties for each country and the weighted mean score within each country is always equal to 150 National Rasch scores can be found in the Student Background data files and in the Written Assessment data files ASMPV01 ASMPV05 Mathematics Plausible Value 1 to Plausible Value 5 Population 1 ASSPV01 ASSPV05 Science Plausible Value 1 to Plausible Value 5 Population 1 BSMPV01 BSMPV05 Mathematics Plausible Value 1 to Plausible Value 5 Population 2 BSSPV01 BSSPV05 Science Plausible Value 1 to Plausible Value 5 Population 2 As described in chapter 5 TIMSS made use of multiple imputation or plausible values methodology to provide estimates of student proficiency in mathematics and science Because of the error involved in the imputation process TIMSS produced not one but five imputed values for e
223. sted of sampling students within classrooms Exclusions could occur at the school level student level or both TIMSS participants were expected to keep such exclusions to no more than 1046 of the national desired populations Nevertheless the National Defined Populations generally were subsets of the desired populations Participants could exclude schools from the sampling frame if they were in geographically remote regions were extremely small offered curriculum or structure different from the mainstream or provided instruction only to students in the within school exclusion categories The general TIMSS rules for defining within school exclusions follow Educable mentally disabled students These are students who are considered in the professional opinion of the school principal or other qualified staff members to be educable mentally disabled students or who have been so diagnosed in psychological 3 4 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER 3 tests This includes students who are emotionally or mentally unable to follow even the general instructions of the TIMSS test It does not include students who merely exhibit poor academic performance or discipline problems e Functionally disabled students These are students who are permanently physically disabled in such a way that they could not perform in the TIMSS tests Functionally disabled students who could perform in the TIMSS test were included in the testing e Non nativ
224. t Ann G A Tan Conference Coordinator TIMSS DATABASE USER GUIDE ACKNOWLEDGMENTS International Study Center continued Mary C Howard Office Supervisor Diane Joyce Secretary Joanne E McCourt Secretary Kathleen A Haley Graduate Assistant Craig D Hoyle Graduate Assistant International Coordinating Center 1991 93 David F Robitaille International Coordinator Robert A Garden Deputy International Coordinator Barry Anderson Director of Operations Beverley Maxwell Director of Data Management Statistics Canada Pierre Foy Senior Methodologist Suzelle Giroux Senior Methodologist Jean Dumais Senior Methodologist Nancy Darcovich Senior Methodologist Marc Joncas Senior Methodologist Laurie Reedman Junior Methodologist Claudio Perez Junior Methodologist IEA Data Processing Center Jens Brockmann Senior Researcher Michael Bruneforth Senior Researcher former Jedidiah Harris Research Assistant Dirk Hastedt Senior Researcher Heiko Jungclaus Senior Researcher Svenja Moeller Research Assistant Knut Schwippert Senior Researcher Jockel Wolff Research Assistant Australian Council for Educational Research Raymond J Adams Principal Research Fellow Margaret Wu Research Fellow Nikolai Volodin Research Fellow David Roberts Research Officer Greg Macaskill Research Officer TIMSS DATABASE USER GUIDE ACKNOWLEDGMENTS FUNDING AGENCIES Funding for the International Study Center wa
225. t cases in selecting the sample the value of 7r was set proportional to MOS within each explicit stratum it is generally the case that weighted and unweighted rates are similar 3 5 2 Student Level Response Rates Like the school level response rate the minimum acceptable student level response rate was set at 85 This criterion was applied to the unweighted student level response rate Student level response rates were computed and reported by grade weighted and unweighted The general formula for computing student level response rates is shown in the following equation Up R stu bar EUR elig where p denotes the probability of selection of the student incorporating all stages of selection Thus the weighted student level response rate is the sum of the inverse of the selection probabilities for all participating students divided by the sum of the inverse of the selection probabilities for all eligible students The unweighted student response rates were computed in a similar way but with each student contributing equal weight TIMSS DATABASE USER GUIDE BF CHAPTER 3 SAMPLING 3 5 3 Overall Response Rates The minimum acceptable overall response rate combined school and student response rates was set at 7596 This overall response rate for each grade was calculated as the product of the weighted school level response rate at the grade without replacement schools and the weighted student level response rate at th
226. t collected for a variable for specific individuals Reasons for this include Booklet not assigned to the student Only one of the eight rotated booklets used in the TIMSS study was assigned to each student All variables corresponding to items which were not given to a student have been coded to Not administered Booklet not received booklet lost If a respondent did not receive the instruments assigned to him her or the instruments were lost after administration all items have been coded to Not administered Student absent from session If a student was not present for a particular testing session then all variables referring to that session have been coded to Not administered However if a student participated in a session and did not answer any of the items these questions have been coded to Omit Item left out or misprinted If a particular question or item or a whole page was misprinted or not available to the student teacher or school the corresponding variables have been coded to Not administered 2 An item was omitted for all cases in a country All cases are coded to not administered Cognitive items omitted or mistranslated in student test booklets Any items identified during the translation verification or item analysis processes that were mistranslated such that the nature of the question was altered were removed for a country Background questionnaire items were omitted Questions in the student t
227. t of the program code is functional users will need to edit input and output file names Performing analyses that require the data from more than one country will necessitate merging the respective data files into a larger one Alternatively the user can access the data and compute the necessary statistics on a country by country basis by reading one file at a time computing the necessary statistics and then moving on to the next country s data The method chosen by the user will depend greatly on the storage and processing capacity of the computer system that is used For the examples that we present in this User Guide we have combined the data files of individual countries into one larger data file that contains the data for all participating countries TIMSS DATABASE USER GUIDE 9 5 CHAPTER 9 PERFORMING ANALYSES Figure 9 3 Extract from SAS Control Code for Creating a Student Background SAS Data Set LIBNAME libdat dataset library LIBNAME library format library FILENAME rawinp bsg lt country gt dat r PROC FORMAT LIBRARY library DATA libdat bsg lt country gt INSERT VALUE FORMATS HERE INFILE rawinp LRECL 1024 END eof LENGTH lnline LINE 1n MISSOVER LENGTH STATEMENTS FOR ALL VARIABLES LENGTH DEFAULT 5 LENGTH VERSION 3 5 r INSERT VARIABLE LENGTH STATEMENTS HERE MISSING CATEGORIES MISSING ABRN S INPUT VERSION 1 2 INSERT VARIABLE NAME LOCATION AND FORMATS HERE Assign value lab
228. t test development process and has published several monographs in the TIMSS monograph series As Sampling Referee Keith Rust of Westat Inc United States worked with Statistics Canada and the NRCs to ensure that sampling plans met the TIMSS standards and advised the International Study Director on all matters relating to sampling TIMSS was conducted in each country by the TIMSS National Research Coordinator NRC and the national research center NRCs and their staff members were responsible for carrying out the TIMSS data collection scoring and data entry and contributing to the study design and development and the analysis plans The Acknowledgments section contains information about the management and operations of TIMSS the National Research Coordinators and the TIMSS advisory committees 1 7 Additional Resources Although this User Guide is intended to provide secondary analysts with sufficient information to conduct analyses on the TIMSS data some users may want additional information about TIMSS Further documentation on the study design implementation and analysis can be found in these publications TIMSS Quality Assurance in Data Collection Martin and Mullis 1996 e TIMSS Technical Report Volume I Design and Development Martin and Kelly 1996 e TIMSS Technical Report Volume II Implementation and Analysis Martin and Kelly 1997 1 28 TIMSS DATABASE USER GUIDE Chapter 2 TIMSS Instruments and B
229. te JK VAR for Mean p jvar p jvar rp i pct 2 Compute JK VAR for P if rwt i gt 0 then n jvar n jvar amp wgt rwt i 2 Compute JK VAR for N end mnx se sqgrt mn jvar pct se sqrt p jvar n se sgrt n jvar run mend jack 9 10 TIMSS DATABASE USER GUIDE PERFORMING ANALYSES CHAPTER 9 The user needs to know some basic SAS macro language in order to use JACK SAS The macro needs to be first included in the program file where it will be used If the user is operating in batch mode the macro needs to be called in every batch If the user is using SAS interactively the macro needs to be called once at the beginning of the session and it will remain active throughout the session If the session is terminated and restarted at a later time the macro needs to be called once again The macro is included in the program file or session where it will be used by issuing the following command as part of the SAS syntax include directory location jack sas where directory location points to the specific drive and directory where the macro JACK SAS can be found The macro requires that several parameter arguments be submitted when it is invoked These parameters are WGT The sampling weight to be used in the analysis Generally TOTWGT when using the student files or MATWGT SCIWGT or TCHWGT when using the teacher files JKZ The variable that captures the assignment of the student t
230. te coding sheets and similar procedures were used for the questionnaires Entry of the achievement and background data was facilitated by the International Codebooks and the DATAENTRYMANAGER software program 4 6 TIMSS DATABASE USER GUIDE DATA COLLECTION PROCEDURES CHAPTER 4 The background questionnaires were stored with the various tracking forms so that the data entry staff could control the number of records to enter and transcribe the necessary information during data entry NRCs were asked to arrange for double entry of a random sample of at least 5 of the test instruments and questionnaires An error rate of 1 was considered acceptable After entering data files in accordance with the international procedures countries submitted their data files to the IEA Data Processing Center 4 4 Database Creation Even though extreme care was taken in developing manuals and software for use by the more than 40 participating countries the national centers inadvertently introduced various types of inconsistencies in the data which needed to be thoroughly investigated by the IEA Data Processing Center and the International Study Center at Boston College The TIMSS data underwent an exhaustive cleaning process designed to identify document and correct deviations from the international instruments file structures and coding schemes The process also emphasized consistency of information with national data sets and appropriate linking among
231. te region The exclusion rate for the performance assessment sample was not to exceed 25 of the national desired population 3 5 Response Rates Weighted and unweighted response rates were computed for each participating country by grade at the school level and at the student level Overall response rates combined school and student response rates also were computed 3 2 6 TIMSS DATABASE USER GUIDE SAMPLING CHAPTER 3 3 5 1 School Level Response Rates The minimum acceptable school level response rate before the use of replacement schools was set at 85 This criterion was applied to the unweighted school level response rate School level response rates were computed and reported by grade weighted and unweighted with and without replacement schools The general formula for computing weighted school level response rates is shown in the following equation M098 n R sch E wa Sch Y MOS x elig For each sampled school the ratio of its measure of size MOS to its selection probability 77 was computed The weighted school level response rate is the sum of the ratios for all participating schools divided by the sum of the ratios for all eligible schools The unweighted school level response rates are computed in a similar way where all school ratios are set to one This becomes simply the number of participating schools in the sample divided by the number of eligible schools in the sample Since in mos
232. teacher but in different classes The Teacher Background files cannot be merged directly onto the 4 TIMSS DATABASE USER GUIDE DATABASE FILES C RASP TIRE Re Z student data files and they do not contain sampling information or achievement scores It is important to note that the Teacher Background data files do not constitute a representative sample of teachers in a country but consist rather of the teachers who teach a representative sample of students The teacher data should therefore be analyzed only in conjunction with the Student Teacher Linkage file The Teacher Background data files contain a series of other identification variables link variables and the derived variables that were used for the creation of the international reports 7 2 1 3 School Background File The principals or administrators of the schools in the TIMSS sample were administered a school background questionnaire with questions about school policy and school environment The School Background data file contains the responses given to the questions in this questionnaire That file also contains a series of identification variables link variables sampling variables and achievement variables The school data files can be merged with the student data files by using the country and school identification variables 7 2 1 4 Identification Variables In all background files several identification variables are included that provide information used to identify stude
233. ten assessment data files also contain a series of identification variables sampling variables and achievement variables The data contained in this file can be linked to the student background data files by using the variables IDCNTRY and IDSTUD 7 2 2 2 Performance Assessment Files A subset of the students who participated in the TIMSS written assessment were also sampled to participate in a performance assessment These students were presented with three to four performance assessment tasks each of which asked them conduct some activities and respond to a set of questions The responses to the performance assessment tasks were also coded by trained coders using a two digit system The performance assessment files contain the codes assigned by the coders to the student responses They also contain identification variables sampling variables and achievement variables The achievement variables included in the performance assessment data files are those from the written assessment The data contained in these files can be linked to the Student Background data files and the Written Assessment data files by using the variables IDCNTRY and IDSTUD 7 2 2 3 Cognitive Item Variable Names The cognitive item variable names are based on 7 digit alphanumeric codes according to the general definitions given in Table 7 6 Z 6 TIMSS DATABASE USER GUIDE DATABASE FILES CHAPTER 7 Table 7 6 Variable Name Definitions for the Written Assessment and Pe
234. the Royal Statistical Society Series B 39 1 38 Foy P 1997a Implementation of the TIMSS sample design In M O Martin and D L Kelly Eds TIMSS technical report volume II Implementation and analysis Chestnut Hill MA Boston College Foy P 1997b Calculation of sampling weights In M O Martin and D L Kelly Eds TIMSS technical report volume II Implementation and analysis Chestnut Hill MA Boston College Foy P Rust K and Schleicher A 1996 Sample design In M O Martin and D L Kelly Eds TIMSS technical report volume I Design and development Chestnut Hill MA Boston College TIMSS DATABASE USER GUIDE RETE REN CES Gonzalez E J and Foy P 1997 Estimation of sampling variability design effects and effective sample sizes In M O Martin and D L Kelly Eds TIMSS technical report volume II Implementation and analysis Chestnut Hill MA Boston College Harmon M and Kelly D L 1996 Performance assessment In M O Martin and D L Kelly Eds TIMSS technical report volume I Design and development Chestnut Hill MA Boston College Harmon M Smith T A Martin M O Kelly D L Beaton A E Mullis I V S Gonzalez E J and Orpwood G 1997 Performance assessment in IEA s Third International Mathematics and Science Study Chestnut Hill MA Boston College Kelderman H and Rijkes C P M 1994 Loglinear multidimensional IRT models for polytomously scored items
235. the likelihood is N 4 5 z 14 n l where N is the total number of sampled students Differentiating with respect to each of the parameters and defining the marginal posterior as f x 8 0 f 0 W y E h 9 W S y Elx 15 i f x W E y E E provides the following system of likelihood equations N A S x fE z 10 A 0 Y S y E 1x a0 0 16 n l 0 i 58w ww 17 1 N and D fl YW 6 vw 0 Y Sv 1x d0 18 n lg where E z16 0 6 Y zexp z bo A amp 19 zco aid 9 f6 0 Y amp v Z 1x a0 20 The system of equations defined by 16 17 and 18 is solved using an EM algorithm Dempster Laird and Rubin 1977 following the approach of Bock and Aitken 1981 TIMSS DATABASE USER GUIDE SiS CHAPTER 5 SCALING 5 6 Latent Estimation and Prediction The marginal item response 13 does not include parameters for the latent values 0 and hence the estimation algorithm does not result in estimates of the latent values For TIMSS expected a posteriori EAP estimates of each student s latent achievement were produced The EAP prediction of the latent achievement for case n is Pp A QA yendo w 8 3 x Q1 r Variance estimates for these predictions were estimated using va ot Y o e yo e n o W 89 51 Q2 5 7 Drawing Plausible Values Plausible values are random draws from the marginal posterior of the latent distributi
236. the many data files The national centers were contacted regularly throughout the cleaning process and were given multiple opportunities to review the data for their countries 4 5 Instrument Deviations and National Adaptations Ensuring the international comparability of both the cognitive and contextual variables was an important aspect of TIMSS A number of data management steps were focused on evaluating the international comparability of the TIMSS items and any deviations for specific items were handled on an individual basis An overview of the procedures and policies applied to ensuring international comparability is provided in the following sections relating to the test items and the background questionnaire items 4 5 1 Cognitive Items All TIMSS written assessment test items and performance assessment tasks were originally developed in English and then translated by the participating TIMSS countries into more than 30 languages In addition to the translation verification steps used for all TIMSS test items Maxwell 1996 a thorough item review process was also used to further evaluate any items that were functioning differently in different countries according to the international item statistics Mullis and Martin 1997 As a result of this review process a few items were identified as not being internationally comparable in certain countries and were deleted from the international data files and from the analyses for the international
237. though that meant not testing the two grades with the most age eligible students This led to the students in these countries being somewhat older than those in the other countries and they are shown in a separate panel For a variety of reasons some countries did not comply with the guidelines for sampling classrooms They also are shown in a separate section as are the countries that had unapproved classroom sampling procedures as well as other departures from the guidelines Finally at Population 2 the Philippines had unapproved sampling procedures at the school level and so sampling weights could not be computed Therefore data for the Philippines are unweighted The performance assessment subsamples also were reviewed on the basis of their quality and adherence to the international standards Figure 3 4 indicates the degree to which countries performance assessment samples met the standards The sample of schools and students for the performance assessment was a subsample of schools and students that participated in the main written assessment Consequently the characteristics of each country s performance assessment sample reflect the quality of the sampling for the written assessment and compliance with the guidelines for the performance assessment sampling Due to unapproved sampling procedures at the school level the performance assessment data for Israel at both Population 1 and Population 2 are unweighted This is effectively impleme
238. to not applicable The specific Not Applicable Code value given depends on the number of valid codes available for each item in the same fashion as was described above for the Omitted Response Codes Not Reached Item Codes 6 96 The Not Reached Item Codes are used only for cognitive items Test items at the end of each test booklet in each testing session which were left blank were considered not reached due to the fact that the student did not complete the test These responses are distinguished from the missing responses as they are handled differently during the item calibration process see Chapter 5 They are treated as incorrect responses however in computing achievement scores For the multiple choice items a Not Reached Item Code value of 6 is used For the free response written or performance assessment items a Not Reached Item Code value of 96 is used 7 2 6 National Data Issues Affecting the Use of International Data Files In some cases resources were not available to resolve database issues for specific countries in time for either the release of the international reports or the production of the international data files As a result some international data are modified or not available for some countries These general database issues are documented below Australia Information made available after publication of the Population 1 international reports required that the Population 1 sampling weights for Australia be re
239. to the structure of the data files as well as the source format descriptive labels and response option codes for all variables Program Files These files include programs that allow the user to convert the raw data files into SAS data sets or SPSS system files estimate sampling variance using the jackknife repeated replication method and convert item response codes to score values Data Almanacs The data almanacs are text files that display unweighted summary statistics for each participating country for each variable in the background questionnaires Test Curriculum Matching Analysis Files These files contain data collected for the TIMSS test curriculum matching analysis TIMSS DATABASE USER GUIDE 5 CHAPTER 1 IN TRODUCTION These files are further described in Chapter 7 Each variable in the TIMSS database is designated by an alphanumeric variable name Throughout this guide these variables and the appropriate use of them in conducting analyses are described 1 5 Contents of the User Guide Given the size and complexity of the TIMSS International Database a description of its contents is also complicated It is recommended that the user read through this guide to understand the study and get a sense of the structure and contents of the database prior to trying to use the files contained on the CDs During this first reading there may be particular sections that the user can skim and other sections that the user may want t
240. tribute the student questionnaires In many countries the student questionnaires were distributed in a packet with the test booklets not separately Nevertheless the quality assurance monitoring revealed no indication of any problems with the distribution of the student questionnaires Martin and Mullis 1996 The test administration script prescribed 20 minutes to administer the student questionnaire but more time was required in about 60 of the sessions about 20 minutes extra The student questionnaires asked about students demographics and home environment including academic activities outside of school people living in the home parental education only at Population 2 books in the home possessions in the home and the importance students mothers peers and friends placed on different aspects of education Students also were queried about their attitudes towards mathematics and science The final sections of the questionnaires asked about classroom experiences in mathematics and science In general the structure and content of the student questionnaires were similar across Populations 1 and 2 However for many questions the response options were reduced in the Population 1 version from four to three categories At Population 2 there were two versions of the student questionnaires One version was for use in countries teaching general or integrated science non specialized version and the other was for use in systems where stu
241. ts JRR replication information and any other variables used in the selection of cases e Retrieve the relevant classification variable or variables from the school database Merge the variables from the school database onto the student database using the variables IDCNTRY and IDSCHOOL e Use the macro JACK with the corresponding arguments and parameters Print out the result file 9 7 Scoring the Items There were several types of items administered as part of the TIMSS tests There were multiple choice items in which the student was asked to select one of four or five options as the correct response These were administered as part of the written assessment The responses to these items are coded with one digit The codes used to represent the responses to these items are shown in Table 9 3 below Table 9 3 Definitions of Response Codes for the Multiple Choice Items in the Written Assessment Data Files Code Description Chose the first option a Chose the second option b Chose the third option c Chose the fourth option d Chose the fifth option e Did not reach the item Gave an invalid response chose more than one of the options available The item was not administered Did not respond to the item although the item was administered and was reached TIMSS DATABASE USER GUIDE 9 39 CHAPTER 9 PERFORMING ANALYSES There were also open ended items where the students were asked to construct a response to a
242. uI SSWIL eua TOF oeueu e ejep pequbre un Uuorae ndod seTqetzeA punozbyoeg quepnis Apna3s eouetos pue soTra3 ueuaew euora eude4qul PITUL S L661 zz aequeides 9 qpupDA snonulijuo 10 p dsiq punpuiv DJbD e duinx3 Z e1nBiu USER GUIDE DATABASE IMSS T CHAPTER 7 DATABASE FILES 7 6 Test Curriculum Matching Analysis Data Files To investigate the match of the TIMSS tests to the curricula of each of the participating countries TIMSS carried out a special analysis called the Test Curriculum Matching Analysis TCMA Each country was to identify for each item whether the topic of the item was intended in the curriculum for the majority of the students Results based on items considered appropriate are presented in Appendix B of Beaton et al 19962 Beaton et al 1996b Mullis et al 1997 and Martin et al 1997 The selection of the items by each country by grade is included as part of the International Database There are four files that contain the item selection by each country at each grade level These files are located in the subdirectory called TCMA in the corresponding CD The four files are TCMAMPI CSV Test Curriculum Matching Analysis Population 1 mathematics item selection TCMASPI CSV Test Curriculum Matching Analysis Population 1 science item selection TCMAMP2 CSV Test Curriculum Matching Analysis Population 2 mathematics item selection TCMASP2 CSV Test Curriculum Matching Analysis Pop
243. udent Background SPSS Data Set 9 7 SAS Macro for Computing Mean and Percents with Corresponding JRR Standard Errors JACK S AS te er e ertet miei d pn ite 9 9 Number of Replicate Weights Needed for Computing the JRR Error Variance Estimate 9 12 SAS Control Code and Extract of Output File for Using the Macro JACK SAS 9 15 SPSS Macro for Computing Mean and Percents with Corresponding JRR Standard Errors AT ACKSPS 1575 emet ri E DR toD rep Unete 9 16 SPSS Control Code and Extract of Output File for Using the Macro JACK SPS 9 21 SAS Control Statements for Performing Analyses with Student Level Variables IXXAMAPIET GAG cece Ne ntn cue ede M e M nk tna Ean atten 9 23 Figure 9 10 SPSS Control Statements for Performing Analyses with StudentLevel Variables EXAMPEE LSPS rne E eee OR E RU A UIROS E 9 24 Figure 9 11 Extract of SAS Computer Output for Performing Analyses with Student Level Vdriables EXAMPLE si 6 sese epa oar ee afe eder OT DOE 9 25 Figure 9 12 Extract of SPSS Computer Output for Performing Analyses with Student Level Variables EXAMPLE T 4 ierit rti ederent derriere entera 9 26 Figure 9 13 SAS Control Statements for Performing Analyses with Teacher Level Variables AAMT EZ SAS o tri ca ete s reri aet oeeeob lea ir setae tert Rea e 9 28 SPSS Control Statements for Performing Analyses with Teacher Level Variables EXAMPLE Z SPS ttis rere rere e erbe bee pese He gre li em redes
244. ulation 2 science item selection These files are in text format with their fields separated by commas The first record for each file contains the labels for each field in the file Each row in the file contains a country s selection for inclusion indicated with a 1 or exclusion indicated with a 0 of the items for a specific grade level For more information on the TCMA see Beaton and Gonzalez 1997 7 42 TIMSS DATABASE USER GUIDE Chapter 8 Estimating Sampling Variance With complex sampling designs that involve more than simple random sampling as in the case of TIMSS where a multi stage cluster design was used there are several methods for estimating the sampling error of a statistic that avoid the assumption of simple random sampling One such method is the jackknife repeated replication JRR technique Walter 1985 The particular application of the JRR technique used in TIMSS is termed a paired selection model because it assumes that the sampled population can be partitioned into strata with the sampling in each stratum consisting of two primary sampling units PSU selected independently Following this first stage sampling there may be any number of subsequent stages of selection that may involve equal or unequal probability selection of the corresponding elements The TIMSS design called for a total of 150 schools for the target population These schools constituted the PSUs in most countries and were paired sequenti
245. verview of the scaling methodology used by TIMSS including a description of the scaling model plausible values technology the international calibration sample and standardization of the international scale scores Chapter 6 Student Achievement Scores This chapter describes the student level achievement scores that are available in the International Database including how they were derived and used by TIMSS and how they can be used by secondary analysts 1 6 TIMSS DATABASE USER GUIDE IN TRODUCTION CHAPTER 1I Chapter 7 Content and Format of Database Files This chapter provides detailed descriptions of the TIMSS data files codebook files data access programs and data almanacs provided in the TIMSS database Chapter 8 Estimating Sampling Variance This chapter describes the jackknife repeated replication procedure for estimating sampling variance Chapter 9 Performing Analyses with the TIMSS Data Some Examples This chapter provides example programs in SPSS and SAS for conducting analyses on the TIMSS data including merging data files and using the jackknife repeated replication procedure to estimate standard errors Supplement 1 International Versions of the Background Questionnaires Population 1 This supplement contains the international versions of the student teacher and school background questionnaires for Population 1 and tables that map each question to a variable in the database Supplement 2 Internat
246. wer grade GEN STUDENT S SEX NOT ADM OMIT 1 GIRL 2 BOY Country Cases Australia 4741 2335 2280 Austria 2526 1261 1243 Canada 7594 3660 3708 Cyprus 3308 1640 1632 Czech Republic 3256 1646 1600 England 3056 1507 1478 Greece 2955 1441 1499 Hong Kong 4396 1967 2406 Hungary 3038 1492 1456 Iceland 1698 836 819 Iran Islamic Rep 3361 1722 1595 Ireland 2889 1348 1506 Japan 4306 2105 2189 Korea 2777 1323 1451 Latvia LSS 2054 1027 991 Netherlands 2790 1379 1391 New Zealand 2504 1287 1210 Norway 2219 1063 1101 Portugal 2650 1286 1359 Scotland 3132 1523 1488 Singapore 7030 3378 3639 Slovenia 2521 1228 1282 Thailand 2870 1433 1415 United States 3819 1857 1962 DATABASE September 22 8 15 54 36 a 0n Bmnpbm FLA SESS 1997 A second type of data display is used for continuous variables This includes the sample size the count of students who were not administered the question the count of those who did not respond the count of those to whom the question did not apply the mean mode minimum maximum and the 5th 10th 25th 50th 75th 90th and 95th percentiles An example of a continuous variable almanac display is shown in Figure 7 3 7 40 TIMSS DATABASE USER GUIDE 7 CHAPTER 7 41 LES F DATABASE 0 0T 0 L o s o e T OT 0 T L E vce OTT Te 618 Se3e3s pearun i J i i i ZI8Z 9E ZZ 0L8Z pue reui 0 0I 0 L 0 9 0 v 0 z o I 0 0 6 E 66EZ cc ET IZSZ eIUSAOTS OET 0 6 0 8 0 9 ore o z
247. within the school IDBOOK Identifies the specific test booklet 1 8 that was administered to the student IDGRADER Indicates whether the student was selected from the upper or the lower grade of the target population IDGRADE Indicates the actual grade denomination within the country Additional Variables Included in the Teacher Background Files IDTEACH Identification number that uniquely identifies the selected teacher within the school It is a hierarchical identification number formed by the combination IDSCHOOL and a two digit sequential number within each school This variable is unique to each teacher within each country but is not unique in the teacher file IDLINK This variable uniquely identifies the class for which the teacher answered the questionnaire The combination of variables IDCNTRY IDTEACH and IDLINK uniquely identifies a teacher class combination in the database IDGRADE The target grades for which the teacher answered the questionnaire IDGRADER Indicates the TIMSS grade level associated with each teacher 1 lower grade 2 upper grade IDSUBJCT The subject s taught by the teacher mathematics science or both Additional Variable Included in the School Background Files IDGRADER Indicates the TIMSS grade levels contained in each school 1 lower grade school only 2 upper grade school only 3 both lower and upper grades In the Student Background file the IDSTUD variable provides a unique identific
248. xojJ uor4eooT oureN Teqel eTqetzea das epoohse TTA L6 60 PZ uorasen ON ILA 93eq xooqepoo eBpg ooqepo n jo 1nojuug e dubx3 L Z 94081 7 36 USER GUIDE DATABASE IMSS T DATABASE FILES CHAPTER Z Variable Number The first column Var No contains a sequential number for each variable in each codebook file Question The second column contains an abbreviated variable identifier providing descriptive information needed to identify the content of the question and or the source for each type of variable Variable Name The third column Variable Name contains the variable name associated with each variable included in the international data files The naming system used for each variable type is described in the previous sections on the contents of data files Variable Label The fourth column Label contains an extended textual variable label of up to 40 characters associated with each variable providing more descriptive information about the content of each variable For multiple choice cognitive items the variable label includes the correct response option enclosed in brackets During data analysis the variable labels can be printed out to enhance understanding of the results Code The fifth column Code contains the codes used for variable responses For variables where numeric data are supplied in response to open ended questions the keyword VALUE is entered in the Code column
249. ychometric innovations employed the TIMSS database is enormous and extremely complex There are more than 500 files on the two compact disks containing data and documentation Every effort has been made to organize the database and provide adequate documentation so that researchers can access the database for secondary analysis Reading this User Guide is the first step in using the TIMSS database This guide describes TIMSS including the data collection instruments sample design and data collection procedures documents the content and format of the data files in the international database and provides example analyses Appropriate use of the various files and variables as well as special considerations arising from the complex design are described There are four supplements to the User Guide containing copies of the TIMSS international background questionnaires documentation of national adaptations of the international background questionnaire items and documentation of derived variables reported in the international reports This chapter of the User Guide provides an overview of TIMSS briefly describes the contents of the database and describes the contents of this User Guide TIMSS DATABASE USER GUIDE TEN CHA PATES RS 1 Table 1 1 Countries Participating in TIMSS at Population 1 and 2 Data Included in Database Population 1 Written Assessment Australia Austria Canada Cyprus Czech Republic England Greece Hong Kong
250. zalez and Foy 1997 The sample selection method used for first stage sampling was based on a systematic probability proportional to size PPS technique The schools in each explicit stratum e g geographical region public private etc were listed in order of the implicit stratification variables and then further sorted according to their measure of size MOS Of course the stratification variables differed from country to country Small schools were handled either through explicit stratification or through the use of pseudo schools In some very large countries there was a preliminary sampling stage before schools were sampled in which the country was divided into primary sampling units TIMSS DATABASE USER GUIDE 9 25 CHAPTER 3 SAMPLING It was sometimes the case that a sampled school was unable to participate in the assessment In such cases this originally sampled school needed to be replaced by replacement schools The mechanism for selecting replacement schools established a priori identified the next school on the ordered school sampling list as the replacement for each particular sampled school The school after that was a second replacement should it be necessary Using either explicit or implicit stratification variables and ordering of the school sampling frame by size ensured that any original sampled school s replacement would have similar characteristics 3 3 Classroom and Student Sampling In the second sampling stag

User Guide for the TIMSS International Database

Contents

Download Pdf Manuals

Related Search

Related Contents