Quick Guide to EduG 6
Contents
1. Facet Label Level Universe Observation design reduction Pesos PP B IN rp occassions Jo R Nr items 5 JN E Browse Edit data nsen data mport sums of square Export data Delete data Measurement design POL z Reports C Textformat Number of decimals 5 Decimal separator Period CEA File C Usersihathcoa Documents EDUG Datalaahlett MF ANOVA Iv Coef G M saate of Phi lambda _ Means _ Optimizatign 6 analysis Compute Edit report Save Save as Close After checking the circled box you will be presented with the following screen We can modify the options in several ways in order to examine the effect of increasing the number of items or occasions on the estimated generalizability coefficient In this situation since a large portion of the error variance resides within PI we will examine the effect of changing items It should be noted however that we can try different combinations of items and occasions in order to optimize our measurement procedure Optimization Observ Coef_G rel rounded Coef_G abs rounded Rel Err Var Rel Std Err of M Abs Err Var Abs Std Err of M Ro 0 71910 0 72 0 16133 0 40166 0 20833 0 45644 0 80 0 75073 0 75 0 13625 0 36912 0 17708 0 42081 84 41 0 818 0 0 77509 0 78 0 11833 0 34400 0 15476 0 39340 96 65 52 0 835 U2. ere gt un wo oc O r oO a z O o 2 oO P 2 jo Q delimited file Item Occasion Person D D OO OO OO OO OO OO OO OO U1 U1 U1 UT UT MN NNNNNRPRPRRPRPRRFPNNNN ON BF U R NN FR 01 amp NN FR 01 R NN RQ UI N BR BR BR NN BR NN N NN UW ANN Once the data is imported then the file is automatically saved After saving we can no longer change the observation design i e FACET LABELS OR LEVELS We can however still change whether a facet is declared as random or finite Measurement Design This section indicates the specification of a measurement design GI EduG 6 0 e C Users hathcoa Documents EDUG Data aahlel gen 60 scores Tite OEM Number of facets 3 Observation and estimation designs Facet Label Level Universe Observation design reduction Persons Occassions Import a file with raw date Browse Edit data Insert data Import sums of squares Export data Delete data E C Textformat Number of decimals 5 x Decimal separator Period v e RTE DEEE On File C Users hathcoa Documents EDUG Data aahle1 rt Parameters Compute A is used to indicate differentiation facets object of measurement from sources of error Differentiation facets are placed to the left of while sources of error are placed to the right Some points that are worth noting 1 Donotindicate nesting in the measurement design We have already in
3. 0 79441 0 79 0 10490 0 32388 0 13802 0 37151 0 85 0 81013 0 81 0 09444 0 30732 0 12500 0 35355 Nb of levels Opt1 Opt2 Opt3 Opt4 Opt5 Facet Obs Univ Obs Univ Obs Univ Obs Univ Obs Univ Obs Univ P Vs en o a GE ee M eee HE EE Copy OK Cancel Quit Below have kept everything constant except for the number of items Optimization G study Option 1 Option 2 Option 3 Option 4 Option 5 Lev Univ Lev Univ Lev Univ Lev Univ Lev Univ Lev Univ P 6 INF 6 INF 6 INF 6 INF 6 INF 6 INF O 2 INF 2 INF 2 INF 2 INF 2 INF 2 INF l 5 INF 6 INF 7 INF 8 INF 9 INF 10 0 82315 0 82 0 08608 0 29340 0 11458 0 33850 Note how the estimated G_coefficient changes as we increase the number of items This information allows one to conclude that increasing the number of items to 7 would provide a generalizability coefficient approximately 82 Itis up to you however as a researcher to decide what optimization procedure is appropriate given practical constraints In this situation we may be satisfied with the original G_coefficient and decide that increasing the number of items for this improvement is not worth additional resources Variance Attribution Diagrams Variance attribution diagrams are extremely beneficial before conducting a G study These diagrams allow you to determine which sources of error may be confounded These diagrams are also beneficial in that they allow you to determi
4. are three columns Random Mixed and Corrected Since all of our facets are random we are using a random model However if some facets were fixed we would use the mixed model Only use the corrected model if the differentiation facet is fixed G Study Table Source Differ Source Relative Absolute error relative absolute of entiation of error variance variance variance variance On 0 01000 4 8 me a a 0 03700 7 8 me PO 0 01083 6 7 0 01083 5 2 E 0 09467 58 7 0 09467 4 TE SN DUT 0 00000 0 05583 0 05583 Sum of 0 16133 100 0 20833 varlances Standard ee Relative SE 0 40166 Absolute SE 0 45644 Coef_G relative Coef_G absolute This is important if we are interested in absolute decisions i e placing students ona scale Notice that all sources of error are This is the differentiation variance universe score variance If this is very small then a measurement procedure may have difficulty detecting differences i e there are little differences to detect included in this decision whereas only some sources of error affect relative rankings This is important when making relative decisions which students are higher or lower than others This is only affected by interactions of the differentiation facet with other facets PO PI etc Source Differ Source of entiation of variance variance variance P 0 53333 une
5. last facet to be declared would be the facet whose levels change most rapidly when scanning from left to right Let s assume that we have 6 persons measured on 2 occasions At each occasion each person responded to the same five items This may have a format that is similar to the data given below For this data persons are changing least rapidly followed by occasion and then item Consequently in the EduG 6 0 program it is important to first label persons then occasions and then items This file was pulled from excel To actually import the file however I ve noticed that you must delete the first row indicating variable names and the person occasion and item column The program will then only read the score column of course the name score is deleted So before saving you should delete everything except the actual scores If you have appropriately labeled the facets in the EduG 6 0 program putting in person first then occasion then item then the program will accurately read a single column of data You can also insert the data manually after defining your facets have often found it easier to simply click insert data and then paste the values from an excel file into the EduG 6 0 program This is also a good way to examine whether you are thinking about your design appropriately v O s U a n c Ee 2 8 v gf v Ss v 2 N Oo own 0O U ao S gt U a 8 Q pa n
6. within children If in a different study we have 2 groups of raters and let s assume that there are three raters within each group One group of raters is assigned to classroom 1 and the second group is assigned to classroom 2 In this case raters are nested within classrooms Raters would be crossed with classrooms if every rater went to each classroom So let s assume that we wish to design a study that is fully crossed For this study we have 6 students observed by a group of raters across 2 occasions In this situation each rater i e 2 total raters is assigned to each occasion Consequently this is fully crossed given that each rater observes each student for every occasion The data for this situation is actually presented below under importing data However for now we will examine how this design is specified in the EduG 6 0 program z or oa Documents EDUG Data aahle1 gen 60 scores X Title Crossed Example Number offacets 3 Observation and estimation designs Facet Label Universe Observation design reduction Persons P Occassions QS items I Importa fil hra j Insert data Ir r f sgu Delete data Measurement design P OI Reports C Texti format Number of decimals Decimal separator Period v RTF format Word File C Users hathtoa Documents EDUG Datal aahle1 rtf Parameters Iv ANOVA Iv Coef_G M Compute Estimate of Phi lambda
7. O es l a PO dees PI m Ol Dr POI Sum f 0 53333 variances Standard deviation 0 73030 This indicates the two reliability like coefficients associated with making both relative and absolute decisions In most cases the absolute will be lower than the relative coefficient Relative Absolute error error variance relative variance absolute me 0 01000 48 De 0 03700 17 8 0 01083 6 7 0 01083 5 2 0 09467 58 7 0 09467 45 4 p 0 00000 0 0 0 05583 34 6 0 05583 26 8 0 16133 0 20833 100 Absolute SE 0 45644 This is an important piece of information It indicates how much scores are expected to vary if the study were replicated by taking a random sample of 5 items and 2 occasions In other words on average each person s score tends to deviate 40 points from their universe score what would be expected across all items and occasions We can also use this to create confidence intervals around individual scores For 95 CI simply Score SE 1 96 D Study Optimization Study A Decision study i e D study allows us to use the information from a generalizability study in order to examine how changes in sampling may influence the results Below is a brief illustration depicting how to use the optimization feature of the EduG 6 0 program F sl EduG 6 0 e C Users hathcoa Documents EDUG Data aahlel gen 60 scores Title Crossed Example Observation and estimation designs
8. Quick Guide to EduG 6 0 Download from website http www irdp ch edumetrie englishprogram htm After downloading complete the following instructions to be compatible with security software This bug is a side effect of an excessive protection of Windows EduG uses an external module APLgrid dil which must be Registered by Windows The registration is automatically done by the install procedure and should be unchanged Unfortunately believe that some Cure System Cleaners remove Registry of APLgrid dil because it is potentially dangerous Since Windows Vista this Registration is protected and can only be done by the Administrator of the computer If you are running Windows Vista try the following actions Attention You must be running your computer as Administrator open Windows Control Panel select User Accounts click Turn User Account Control on or off clear the check box labeled Use User Account Control UAC to help protect your computer click OK restart computer for this change to take affect Num A UNBE Then EduG will automatically register the module APLgrid dil If you are running Windows 7 the actions should be the same you have to deactivate the User Account Control UAC The Windows7 panels are different of those of Windows Vista have only the french version of Windows 7 give you an approximative translation in English 1 open Windows Control Panel 2 select Protection o
9. _ Means Optimization G Facets analysis Edit report Save l nn N Simply use a capital letter to signify each Facet There are 6 persons observed on two occasions On each occasion each person responded to the same 5 items luG 6 0 e C Users hathcoa Documents EDUG Data aahle_manual gen 0 score Title Crossed Example Number offacets 3 Observation and estimation designs Facet Label Level Universe Observation design reduction Persons P Occassions ems Insert data Export data Delete data Measurement design Reports Textformat Number of decimals 5 7 Decimal separator Period gt RTF format Word File B CAUsers hathcoa Documents EDUG Data aahle_manual txt Parameters Compute If we had a different design on each occasion individuals responded to different items then items would be nested in occasions 1 0 indicates that items are nested within occasions This is shown only for illustration purposes duG 6 0 e C Users hathcoa Documents EDUG Data aahle_manual gen 0 score Title Crossed Example Number offacets 3 Observation and estimation designs Facet Persons Occassions Items FA Import a file with raw data dit data Insert data a Import sums of squares Delete data Measurement design Reports amp Textformat Number ofdecimals 5 aa Decimal separator Period aa RTF f
10. ation G Facets analysis Click this option if you wish to have output reports in word Once you have imported the data and have specified a measurement design you may then click compute to generate a report You will actually see more output than what is displayed below will however provide you with a quick description of the output that is most relevant Analysis of Variance Components EON firn wes comae fa E 34 73333 5 6 94667 0 53333 0 53333 0 53333 0 37615 1 06667 1 1 06667 0 02000 0 02000 0 02000 0 03260 14 10000 4 3 52500 0 18500 0 18500 0 18500 0 17518 3 33333 5 0 66667 0 02167 0 02167 0 02167 0 07882 30 10000 20 1 50500 0 47333 0 47333 0 47333 0 24200 1 43333 4 0 35833 0 03333 0 03333 0 04445 11 16667 0 55833 0 55833 0 55833 0 16834 95 93333 59 This indicates the relative proportion of variation attributed to each facet or combination of facets Notice that PI and POI are relatively large This lists the variance components for our estimation design It is important to understand what these components reflect for later interpretation So for example P is differentiation facet and reflects mean differences of each person across each occasion and item PO indicates the extent to which person scores change across each occasion A high PO would suggest that which persons were ranked higher changed across occassions Notice that there
11. dicated what facets are nested so there is no need to replicate that information here 2 Ifthe object of differentiation is nested within a second facet then you must include both of these facets on the left hand side of So for example if persons were nested in occassions then our measurement design would be PO I 3 The inverse of this rule is not true So in other words if we have a facet that is nested in the differentiation facet object of measurement then there is no need to include this information to the right of the So for example if items are nested in persons then items error are nested within a differentiation facet persons Consequently our measurement design would remain P OI Interpretation of Output The following section will briefly illustrate how to compute and interpret output for the person x occasion x item design There are numerous other examples within the EduG 6 0 manual and that can be found in Cardinet Johnson amp Pini 2010 GI EduG 6 0 e C Users hathcoa Documents EDUG Data aahlel gen 60 scores Number of facets 3 m Observation and estimation designs Facet Label Level Universe Observation design reduction Persons Occassions Browse Edit data Insert data Export data Delete data Number of decimals 5 x Decimal separator Period v File B C Users hathcoa Documents EDUG Data aahle1 rtf Jv ANOVA v Coef G Estimate of Phi lambda Optimiz
12. e object of differentiation at which an intersection exists with other facets Error for absolute decisions includes both the hatched lines and the horiziontal lines i e R OR and O Variance Attribution Diagram Nested Design For this design we will modify the previous example so that we may compare it with the fully crossed design Let s assume that we assigned 2 raters to each occasion There are different raters however for each occasion In this case we have 6 individuals that are observed on 2 occasions The main distinction is that now we have 4 raters 2 assigned to occasion 1 and 2 assigned to occasion 2 This indicates that raters are nested within occassions Step 1 Let s start with the occasion circle Step 2 Now we have raters nested within occasions In order to indicate nesting we place the nested facet inside whatever it happens to be nested within Since raters are nested within occasions we will therefore place the circle for raters inside the circle for occasions Step 3 We now have to indicate a circle for person variance Since all people are examined across each occasion then this circle should intersect occasions It is also important to note that all people are rated by the same raters Consequently this circle should intersect with both the O and the R 0 circles Step 4 Let us now label each intersection It is important to note that we must still label and intersection among O and R O In other words thi
13. f User Accounts 3 select Modify your User Accounts 4 select Modify Control Parameters of User Account 5 move cursor down 6 Click OK then Close Control Panel 7 Restart computer for this change to take effect Then EduG will automatically register the module APLgrid dll USER MANUAL http www irdp ch edumetrie documents EduGUserGuide pdf Data Constraints 1 If measures are made using several items then all measurements should be on the same scale e g 1 5 e Ifthis is not the case then you may force them to be on the same scale or you may choose to report proportions 1 5 2 5 3 5 etc 2 EduG 6 0 can only handle balanced data In other words if students are nested within classes then we must have equal students in each class If raters are nested within occasions then the number of raters assigned to each occasion must be equal e Consider data imputation methods e Force design to be balanced randomly select to obtain balanced design e Estimate sums of squares using SAS or SPSS and then use sums of squares to estimate variance components in EduG 6 0 3 EduG 6 0 can handle up to 8 facets which includes a differentiation facet what we wish to distinguish in a measurement procedure Determining Observation and Estimation Design This is perhaps the most important aspect of using the EduG 6 0 program The following steps should be taken when conducting a G study 1 The first thing to consider is identifying the
14. ne which sources of error contribute to both relative and absolute decisions Unfortunately was unable to find a reference that described how to construct these diagrams in a way that is easy to understand The book provided by Cardinet and colleagues 2010 as well as the Shavelson and Webb 1991 introduction to G theory are good places to start They do discuss these in slightly different terms however though believe that the presentation by Cardinet et al 2010 is easier to follow than the Shavelson and Webb 1991 text These diagrams can become increasingly complicated particularly for designs with numerous facets It is also important to recognize that specifying a facet to be fixed can drastically alter the sources of error that can be estimated in a G study As a general rule of thumb if a facet is fixed then it will not contribute to error This makes sense given that this specification indicates that we have observed every possible level the universe of admissible observations is contained in our measurement procedure thus it will not contribute to error when we attempt to make generalized inferences would suggest that you consult both of these texts to get a better understanding of fixed facets and the construction of Venn diagrams in this situation will however provide an overview of two designs one of which is completely crossed and the other has a facet that is nested within a second facet hope that the presentation pr
15. object of differentiation or the differentiation facet In other words what do we wish to differentiate Are we concerned about examining differences among students items methods etc 2 Other facets will contribute to error in our efforts to differentiate our object of measurement differentiation 3 Now we must first identify each facet including the differentiation facet and determine the number of levels for each facet If there are two raters in each occasion then raters will have two levels If 30 people are tested on two occasions then a person facet will have 30 levels It is important in most situations to first name the facet which changes least rapidly in your data file Then identify the face that changes the next least rapidly and so on This is further explained below under Importing Data 4 After we identify the facet we must then provide a label The label is also where we identify whether a facet is nested within a second facet Remember if facet A has two or more levels associated with facet B then A is nested within B If 20 people are in a class and there are 5 classes then persons are nested within classes only if we have different people in each classroom It is conceivable for classrooms to be nested within students though this study is typically rare For example let s assume that we have 8 children that are observed within 3 classrooms If these children are attending different courses then classrooms are nested
16. ormat Word a Parameters F ANOVA v Coef G Compute Estimate of Phi lambda Optimization G Facets analysis This indicates that the universe is infinite for these facets In other words we are willing to treat both persons and occasions as In order to illustrate a ixed facet interchangeable Other raters and I have inserted a 5 here This occasions of the same sample size indicates that we are only would work equally well interested in these 5 items This limits our generalization claims to these specific items but this choice would eliminate items as a source of error Though in the previous example stated that items are fixed we are actually interested in inferences pertaining to any items of the same characteristics In other words for this example we will treat items as interchangeable or random The next step is to import data This can be done in several ways import raw data file insert data manually or insert sums of squares will briefly review how to import data using a raw data file though generally find the insert data command to be more useful Importing Data 1 Important to convert file to ASCII format before importing 2 To be successful care must be taken that facets declared in EduG 6 0 conforms to structure of the file e The first facet declared in EduG 6 0 should be one whose levels change least rapidly when scanned from left to right e The
17. ovided below when coupled with the information provided by these other authors will allow you to construct Variance Attribution Diagrams when thinking about the best design for your study Variance Attribution Diagram Crossed Design To remain consistent will present a variance attribution diagram that is structured around the previous example In this example we have 6 students that are observed on 2 occasions All raters observe each student on every occasion Thus we have a person x occasion x rater design Step 1 Make a circle to signify raters the choice in this situation is rather arbitrary Step 2 Draw a circle to represent occasions Since occasions are crossed with raters the circles must intersect Step 3 Now do the same thing for persons Since persons are crossed with both raters and occasion then these circles should also intersect Step 4 Now we should label each intersection Step 5 Identify the differentiation facet object of measurement In this case we wish to differentiate people though note that we could easily use this design to differentiate raters or occasions Create vertical lines through the differentiation facet Step 6 Now create horizontal lines through the facets of differentiation facets that potentially contribute to error in our measurement procedure KEY Hatched areas signify error pertaining to relative decisions Error for relative decisions is therefore any point within th
18. s intersection will have both O and R O and consequently R O effect of raters within each occasion is confounded with an occasion effect In other words we cannot estimate the effect of raters within each occasion separately from an occasion effect It is also important to indicate that there is a 3 way interaction PRO that potentially exists at the intersection of PRO Though we can estimate an effect of R O it cannot be estimated separately from a three way interaction Consequently we will have output for R O but this is confounded with a three way interaction R O PRO Step 5 Now we will create vertical lines within the circle that represents our facet of differentiation or the object of measurement In this case we will again place vertical lines within the P circle since we wish to differentiate people Step 6 Now create horizontal lines across the sources of error that we wish to make generalizations In this case we will draw horizontal lines across both R O and the O circles KEY Once again the hatched areas indicate sources of error for relative decisions Hatched areas areas with a horizontal line indicate sources of error for absolute decisions Cross or Nested Designs 1 As you can tell the fully crossed design is in many ways optimal to a nested design Nesting leads to confounding sources of error i e O and R O Fully crossed designs therefore allow us to examine more sources of error than nested de
19. signs There are practical advantages to nesting however Some facets may be naturally nested within others i e students nested within classrooms whereas at other times we can force nesting assign different raters to each occasion There may be other advantages to nesting such as better control over carry over effects For example what would be the impact of subjecting the same students to the same items multiple times Might a practice effect be at work Carry over effects can partially be controlled through nesting General advice use fully crossed designs when feasible
Download Pdf Manuals
Related Search